No really. Should I use that library function that I shouldn't use?

The world would make more sense had it happened this way:

Shoot in grainy black and white.

Scene: a sprawling cellar lit by torches and candelabras. Brick archways lead in multiple direction. All the spaces we see are filled with abandoned experiments, strange equipment, and untidy storage. Some of the bits spark. Others bubble. Here and there tables are covered with books, notes, and abandoned comestibles.

Richiestein: "Igor, the world degenerates. Many programmers lose their edge, hiding every day behind strong guarantees from safe environments.¹ We must bring LIFE back to the demons and dragons of yesteryear!"

Igor: "Yes Master"

Richiestein: "I need a way, Igor; a way to trap programmers. To make it easy for them to err; easy to use memory they do not intend."

Igor: "Oh, master. Igor knows. Igor is sure master. You must use sentinel terminated strings, master!"

Richiestein: "Will that help?"

Igor: "Oh yes, master!"

Richiestein: "Will they not see that it is a bad design?"

Igor: "Master must tell them it is for simplicity. And show them correct code saying that it is elegant and basic. Maybe write a book?"

Richiestein: "I suppose."

Richiestein: "But can I get them to overrun their buffers that way?

Igor: "Oh yes master, let Igor show you!"

Igor hurries to a crowded corner of the room and digs frantically through trunks and shelves before returning triumphantly to show the good doctor what he has found.

Igor: "See here, master, I have gets!"

The doctor squints at the jagged and rusty artifact before looking sharply at his assistant.

Richiestein: "Would anyone accept something so obviously cursed as this? Surely not?"

Igor: "They will, Master! They will! You need only put it in the Standard Library master!"²

Richiestein: "Hmmm. We'll try it. But we also need something more subtle. Something that looks like it would work. Maybe something that only brings disaster occasionally. Do we have that, too?"

Igor: "We do, Master!"

Igor roots through another corner of the cellar before bringing a many-geared contraption of only slightly tarnished brass.

Igor: "See master: snprintf. A powerful tool, but if it runs out of space no sentinel is placed, making their string a trap!"

Richiestein: "Oh, that one is better. It almost works."

Exeunt. Laughing.

Sadly it's much more likely there were making only what they really, really needed (and would fit in the very small machine they had on-hand) and intended to come back and fix it later. That's the way these things usually happen.

I do know the danger

The standard library function snprintf³ guarantees to not overwrite the buffer (assuming of course that the programmer passed the right length). Which looks so promising, but they don't guarantee to write a terminal '\0'. They only add that if there is room left in the buffer.⁴

So, if you use snprintf to fill in a buffer and run out of room without noticing (always get the young man's name and address check the return value!), and are later so incautious as to try to measure the length (with strlen) there'll be no terminator and you read off into random memory until you encounter a zero byte. Or a seg-fault. Or nasal demons, of course.

And it can get even worse.⁵ Say you think you want to perform an in-place tokenization with strtok.⁶ Now you can be writing past the end of the buffer.

And, yet, I still use the function

When I exhibited my nifty "safe" string builder for c, I used vsnprintf. Twice. Even though it's a trap. What gives?

grug quite satisfied when complexity demon trapped properly in crystal, is best feeling to trap mortal enemy!

—The Grug Brained Developer

The answer, of course, is that by using it very carefully in just one place I intend to relieve future programmers (including my future self) from the need of using it repeatedly elsewhere.

I also put prominent comments in, not so much to convince future readers as to make sure that I was paying attention, though they should serve as a alert for future readers too. They'll have reason to read the docs with "how long" and null termination in mind. I hope.

When is it a good idea, again?

So ... it's sometimes okay to mess with the magic lamp? Really?

Honestly, that function is dangerous, and the comments don't fix it. If I were developing a non-trivial code in c as a green field project, I'd give real consideration to not using big parts of the c standard library starting with string.h. But the use case that drove me to thinking about it in the first place was maintenance on a statically-linked launcher for a C++ project.⁷

---

¹ Okay, so strong guarantees were few and far between in the early 1970s. but, this is Hollywood! Work with me here.

² And don't call me Shirley!

³ Or its variadac sibling vsnprintf.

⁴ Cue Admiral Ackbar.

⁵ As far as the standard is concerned this is exactly as bad as before—undefined behavior—but from a practical point of view it is more immediately destructive.

⁶ You probably don't, really. For several reasons, but take it arguendo.

⁷ It is harder to write a C++ program you know will "just work" on a target machine, then to do the same with a C program. Especially on windows. So we have a little program written in c that should "just work". It is suppose to detect the environment, set up the prerequisites for the big C++ code, produce a clear error message if it can't, and then launch the main app. It does a lot of string manipulation. But it's been there a long time and it works; there is no way I could justify switching string libraries wholesale.

Elaborate solution to a very minor pain-point

Over the last few years I've repeatedly encountered an annoyance in my work projects. It's not a practical problem, you understand, merely a stylistic one. I mean, when the issue rears it's ugly head I pause, curse, write the code that works just fine but offends my sense of modern C++ idiom, and move on.

It's just that the issue lives rent-free in my head.

So here's the set up: I've spent the last six years upgrading my C++ skills to the "modern" era which means, amongst other things, preferring ranged for loops and routines from the algorithms header to explicit loops using indices, pointers or even iterators.¹ Only my code has to interact with libraries and plugins written in C or exposing C-centric APIs, and those interactions often involve linked-lists. Which you can't put into ranged for loops or algorithms.

So, in my copious spare time, I set out to write a set of iterator templates that you can bolt-on to an arbitrary C linked list to support using modern C++ idioms with the list. That project is now approaching minimum-viable-product levels of completness, though there is a lot left to do.

---

¹ That kind of preference is, of course, not absolute. Sometimes the old, explicit loops have some over-arching advantage and you use them without guilt.

Vectors in C: an all-you-can-eat buffet of so-so choices

Let's say you want to model a physical system. Often you'll want to represent spacial vectors (members of the space $\mathbb{R}^N$) somehow. In fact there are many use case for Cartesian 2-, 3-, or 4-vectors, and for Lorentz 4-vectors as well as less common uses for other dimensionalities, but let's stick with Cartesian 3-vectos for the sake of definiteness because I want to focus on the programming tradeoffs you face if you want to use C (or a C interface¹) for this purpose. Nor do we want to worry about manually optimising for the SIMD module of our chips (compilers are smart these days) or worse still laying things out for the benefit of the GPU.²

Aside: I mostly program in C++ where there are some better options, but I get to mess with a lot of legacy code, so the consequences of someone else making this choice for a code-base I work on are still with me. I might get around to a followup post on ways to adapt an existing legacy library for cleaner inter-operation with new C++ code, but that's for another day: first we must understand the root problem.

C offers us an obvious choice with two painful drawbacks (or perhaps it's one underlying drawback that rear's it's ugly head in two contexts) and a clever way to avoid that issue at the cost of having your soul slowly nibbled to death by syntactic ducks. Nice, huh?

Arrays

The obvious choice is the built-in array type: double vec[3];, right?

The underlying problem is that array are only sort-of first-class types. Consider this code:

#include <stdio.h>

void pass_ptr(double* p)
{
    printf("Passed pointer: %lu\n", sizeof(p)/sizeof(double));
}

void pass_ary(double a[3])
{
    printf("Passed 'array': %lu\n", sizeof(a)/sizeof(double));
}

void pass_c99(double a[static 3]) // Syntax added in c99
{
    printf("Staticly sized: %lu\n", sizeof(a)/sizeof(double));
}

int main()
{
    double vec[3];
    
    printf("In local scope: %lu\n", sizeof(vec)/sizeof(double));

    pass_ptr(vec);
    pass_ary(vec);
    
    return 0;
}

Each of the printf statements is executed once on the same variable, but they generate two results: one says 3 and the others say 1.

This is a classic trap for C newbies. Arrays are not, as is sometimes said, "just pointers" because the symbol table knows how big they are when declared at static or automatic scope. But most things that you can do with them drop that knowledge at which point all that is left is a pointer to the start.³

The other manifestation of the limitation is that you can't assign or perform operations on arrays. That is, this is not legal code:

double v1[3] = {1, 2, 3}
      double v2[3];
      v2 = v3;       // Error! Even when the compiler *does* know the sizes!

and as a result you end up writing your library functions with signatures like cross(double *result, const double *v1, const double *v2) which makes you write particualrly clunky code to use the library:

double v1[3] = {1, 2, 3}
      double v2[3] = {4, 5, 6};
      double cp[3];
      crossProduct(cp, v1, v2);

Ugh. And you get to do it over and over again.

Array-in-struct

Oddly it is amazingly easy to solve both these problems: you just wrap the array declaration in a structure declaration (and typedef it for convenience):

typedef struct {
    double a[3];  // a for "array"
} vector;

This takes up exactly as much memory as before, still knows how many elements are involved when you pass it to a function, and can be assigned! Yeah! It's like magic.

Of course, to access an elements you now write vec.a[2] instead of vec[2], but that's a small price to pay. Right? It's not like your soul will die a little bit each time or anything.

Seasoning with unions

Once you're drunk that CoolAid there is no reason not to go a little further: maybe sometimes you'll want to talk about the coordinates of these things, right? So you make that possible, too:⁴

typedef struct {
    union {
        double a[3];               // a for "array"
        struct {double x, y, z} c; // c for "coordinate"
    }
} vector;

You still need to write a full set of library routines, but now they can have signatures like⁵ vector cross(const vector v1, const vector v2) and you can call them like vector v1 = {1, 2, 3} vector v2 = {4, 5, 6}; vector cp = crossProduct(v1, v2); which is much better.

---

¹ Even if you're confident that you won't be writing in C, you may find it necessary to deal with the limits of the language while planning a binary interface.

² Those are great tools and worthy of your attention if you have a computationaly demanding task, but beyond the scope of this post.

³ Note that as far as the compiler is concered the declaration of pass_ary is identical to that of pass_ptr: the array-like notation is accepted as syntactic sugar only and the compiler pays no attention to the 3. The _c99 variant is a little more subtle but it still doesn't really know the size of the array, it just assumes a minimum for the sake of optimization (and compilers can complain if static analysis show the assumption is violated). Some folks like to use the array form because it makes the declarion express intent to the human reader (though, like comments, it can lie). Others are not so enamored of it, with Linus famously coming out strongly against it.

⁴ Type punning with unions this way is strictly forbidden in C++ (because lifetime-model and invariant-enforcement, that's why; and don't give me any guff about POD types either, sonny, the divine gave you memcpy and std::bit_cast for a reason!), but C programmers are rugged, self-reliant individualists who carry Colt-45 six pointers (colt45****** shootin_iron;) on their hips and ain't afraid of no Endian no how.

⁵ Of course, you might pass pointers for the in-parameters to save a little copying at the cost of writing & all over the place. C sure seems have that same issue come up a lot, eh?.

Bringing C Structs into the C++ Lifetime Model

In addition to legacy code in our own projects, I sometimes "get" to work against libraries (legacy or modern written in plain C. Which is OK. I learned C a long time ago and I'm not intimidated by it, though it can take a while to get back into the right mindset. Of course, there are things I miss. Static polymorphism and namespaces, for instance, are pretty small conceptual changes with significant convenience factor for the programmer.¹

Now, C++ has a reputation as being a dangerous language where it is easy write really broken code. That impression is not wrong, but it is incomplete: the lagnuage also offers features that support writing code that has enforced safety in some aspects. It's not trivial and it takes both discipline and some understanding of how the features work, but in my opinion it takes less discipline to write memory-safe C++ code than memory-safe C code.²

This article covers one way to bring a C struct into the C++ lifetime model to leverage the better (or at least more automatic) memory safety of C++ library primitives.

We start with a highly artificial example struct designed to be a pain memory wise:³

 struct thing {
    int i;
    double d;
    char *s
    int *ary;
};

Each of the pointers pose us some (interrelated) questions:

• Where do the objects that will be pointed to live? Heap? Stack? Data segment? Global memory? Memory mapped file? Something really exotic?
• How do we ensure that the pointer is not used after the objects go away (if they go away)?
• If they exist on the free-store, how do control deallocation?

The questions aren't unique to C, they are the same ones that must always be dealt with. But C code deals with them all every time, while other languages may have built-in answers to some of them.⁴

Nor can you necessarily answer the questions by static examination of the code, but in the case I faced at work, both pointers were consistently pointing at dynamically allocated objects. Moreover the number we needed could not be determined at compile time, so we were storing the thing *'s in a vector.

We had an existing C function thing *newThing(size_t array_size, const char *label) which would create a new struct thing on the heap (with a alloc family function), set default values of i and d, set the string and allocate (but not populate) the array and return the pointer to the thing. This is analogous to a C++ constructor, but for some reason (history, no doubt) we were handling the three calls to free manually each time we needed to reap one of these things.

Then we did something roughly like this:

{
    std::vector<const thing*> thing_list;
    for (const auto &input : inputs)
        list.push_back(newThing(input.name, input.size()));
    process_list(thing_list);
}

Which, of course, loses three heap allocated objects for every item in the inputs container.

Replicating a proper, but C-like, approach to memory management here would mean writing a destructor-analog (perhaps void reapThing(thing *p)) as a free function and inserting std::for_each(thing_list.begin(), thing_list.end(), reapThing); before the closing brace. That works and I wouldn't be displeased to see it in a legacy project like the one I'm working on, but I think we can do a little better.

The "doing better" interface is actually quite simple:⁵

#include "thing.h"

struct thing_wrapper : public thing
{
    thing_wrapper();
    thing_wrapper(size_t array_size, std:string_view label);
    thing_wrapper(const thing_wrapper &);
    ~thing_wrapper();
    
    thing_wrapper &operator=(const string_wrapper &);
}

The wrapper has the same data, but manages the sub-allocations for you. What complexity there is lies in ensuring that the constructors, assignment operators and destructor all agree on memory management of the sub-allocations.⁶ You might also want to add a constructor and assignment operator taking a const thing &, but this is a leap of faith insofar as nothing will enforce a consistent allocation strategy on those inputs. Similarly you can consider supporting move operations if you have a particular use for them.

With the wrapper in place we can change the original code to something like:

{
    std::vector<std::unique_ptr<const thing>> thing_list;
    for (const auto &input : inputs)
        list.emplace_back(std::make_unique<thing_wrapper>(input.name, input.size()));
    process_list(thing_list);
}

With no need, now, for explicit clean-up code.

---

¹ Oddly, neither of these is trivial to add because they imperil the universal linkability of C (which depends on not needing a vendor-dependent name-mangling scheme).

² At the foundational level, it is the object lifetime model that supports this, and at the practical level it is exploited in the standard library which offers a more powerful set of primitives than the C standard library. Step one for writing a robust C program at scale is to get a more robust library (which you might be able to get off the shelf or might want to write yourself).

³ It is, however, analogous to the problem I faced at work today.

⁴ Many "managed" languages have everything on the free-store, and use a garbage collector to resolve the lifetime question.

⁵ I've chosen to make this a struct rather than a class for two reasons. First because the whole interface we want to derive is public: we're not going to extend thing in any way beyond supporting the C++ lifetime model. Second because of Core Guideline C2: the C code enforces no invariant so we don't add one.

⁶ The safe thing to do, is use the facilities used by the code that provides the underlying structure, which in the case of pure c libraries usually means *alloc/free or some wrapper around the same. You may be able to defer to any pre-existing C functions that perform the set-up and tear-down.

The limits of "Fix it when you touch it."

My main projects at work have been on minimal-spend for a few months which means I've been shifted to some feature adds for our biggest product. This thing goes back to the mid-eighties and is coded in C (updated to ANSI syntax, at least), Fortran (updated to f90, at least), C++ (with the standard containers, at least, but lots of it predates the "modern" era), and python (recently ported to python3, at least). So, yeah, it has all the issues you'd expect in a legacy codebase. Some of them in spades.¹

We're basically a contract shop, so we don't do "Let's fix this entire module because it's grotty enough to be a pain", because who would pay for that? On the other hand, we really would like to have nice code, so we have a "You can fix issues with the bits you touch." policy.

Not complaining about that. My last feature add actually removed net lines of code because I replaced some really wordy, low-level stuff with calls to newer library features and factored some shared behavior into utility code to reduce repetition. So the policy makes me a happy (and perhaps even productive) programmer.

But it has it's limits. The short version is some legacy issues span a lot of code and you can't fix them locally.

Case Study

The bit of code I'm working on right now has a peculiar feature: in several places I find a std::vector<SomeStruct> paired with a count variable.² They appear in an effectively-global³ state object and they're passed to multiple different routines as a pairs. This is very much not what you'd expect in code originally written in C++.

Sometime in the past this was almost certainly a dynamic array coded in plain 'ol C. And not even a struct darray {unsighned count; SomeStruct * data}; one paired with some management functions, but a bare, manage-it-yourself-you-wimp pairing of a count and a pointer.⁴ But why wasn't the count discarded when transitioning to std::vector?

Finding out requires a lot of tedious, close reading of code where the pairs are used. And, of course, seeing the old C code behind the current C++.

There are several places where the vector is resized (meaning multiple extra entries are added to the end in one fell swoop) to some "bigger than we need" value. Those extra entries are filled with default data, which the code then overwrites one at a time with freshly calculated "actual" data. This is (a) a performance optimization insofar as it prevents the possibility of multiple re-sizes and copies that exists if you added entries incrementally, (b) a fairly faithful transliteration of what would have done with the dynamic arrays in C, and (c) the wrong way to perform the trick with std::vector.⁵

When this was originally done in C, the "count" variable would have tracked how many entries had "good" data while the allocated size would have been known because you knew the maximum expected size was. But the container doesn't know that you want to do that and it's size() method will always return the number of objects it has (including the default valued one). So the manual count was still needed in places, and they kept the (effectively) global version because it was easier.

Result: getting rid of the extraneous count variable means fixing half-a dozen routines elsewhere in the project and I end up touching scores of lines in a dozen files. That's not "fix what you are touching anyway".

Takeaway

Some legacy maintenance is too big for purely local fixes.

Today was actually the second one of these I've looked at in the last couple of months. I was able to fix the first one mostly with global search-and-replace and only touched five files; I felt that was "local" enough for the payoff in terms of making the code more comprehensible. So I was optimistic on this one, too, but it quickly grew out of hand. None the less, I may be finished and if the regression tests are clean I'm going to commit it.

---

¹ But, honestly, I worked with worse in my physicist days.

² For those who don't do C++, the standard vector container is a extensible array-like data-structure. It maintains its own count.

³ Possibly a subject for another day, and another example of something you can't re-factor in the small.

⁴ In the Bad 'Ol Days, a significant number of programmers would begrudge the cycles lost to function calls for that kind of things when it was "easy" to do it inline at each site. Of course, they could have used a macro DSL for the purpose, but those are tricky and (even then) had a mixed reputation.

⁵ It's wrong for two reasons in general. The less important one here is that it default constructs the new values which can take cycles (in a C dynamic array using realloc means you just get whatever garbage was in the memory occupied by the new spaces so you don't pay for that). The more important issue here is that vector has reserve which unlike resize just makes sure you'll have room for the new stuff if you want to use it, meaning you can then emplace_back for the best of both worlds.

"Safe" string building in c

The deficiencies of the c standard library where strings are concerned have been discussed at length in many places, but I'd like to talk about one issue in particular: building up strings from pieces.

This week I wanted to build up some strings from pieces. Lots of little pieces. In a language with "real" strings this would be easy, you'd just do something like string result = substring1 + substring2 + substring3;, but c does not support that in any general way.¹ There are really only two classes of tools available: strcat/strcpy functions from string.h and sprintf functions from stdio.h. Essentially all the "plain" functions will buffer overrun, and the "n" variants only protect you if you pass the right length (where "right" means paying attention to which functions count the terminating '\0' and which don't). Let's take a closer look at some of them .

strcat, strncat (and strcpy, strncpy)

On the face of it this is the "obvious" thing to do. After all it concatenates strings, right? The main issue is that the first string needs to occupy a sufficiently large buffer or you run into trouble. Which means knowing the desired length when you create the buffer.

If you have a separate buffer you use a "copy" function to move the first string into it and then concatenate onto the end.

As a side note, if you are going to repeatedly apply strcat, do capture the return pointers from each call to avoid a Shlemiel the Painter algorythm.

sprintf, snprintf

Same basic problem: the target buffer needs to be big enough. On the up-side, you don't need extra elements for any little connector text you want to stick in between the substrings that you are recieving from the caller, the environment, the database, or whatever.

And that is really it.

Variable length arrays or malloc

So, you need to wait until you have all the parts, find the desired length, and then create a buffer. Fine. You have two choices: a dynamic allocation or variable length arrays.

The issue with variable length arrays is that they weren't stanadrdized until 1999 and then were made optional in 2011 (and at least one major compile does not support them). Code that depends on VLAs is going to have limited poratability.

So you're going to have to put it on the heap with all the hassle and risks that entails. Great. Be sure to check the return value.

Recipe

All this is old hat and there is a well known hack to accomplish it. You use the "returns the length that would have been written" feature of snprintf functions like this:

nonnullcount = snprintf(NULL,0,...);
buf = malloc(nonnullcount+1); /* watch out for the with/without '\0' issue! */
/* check for errors */
snprintf(buf,nonnullcount+1,...); /* watch out for the with/without '\0' issue! */

It's not hugely time efficient; the two passes through the formatting engine are inellegant, but it's very flexible and does the job.

The three lines of code I exhibited there are idomatic and easy to recognise, so sprinkling them through your code wouldn't be too bad. Except that the error checking is pretty improtant and it will add to the length and break up the visual block. Not nice. So I'd like to encapsulate all that in a function. Which is where it gets technical; how are you with c's variadac function support?

Wrapped up and made pretty I get something like (written to be compatible with c89 compilers even if they would require a more up-to-date libc):

/* A "safe" auto-allocating sprintf.
 *
 * Returns a pointer to a malloc'ed buffer containing the resulting string or
 * NULL if an error occurs. Leaves errno set following the error.
 *
 * It is the caller's responsibility to free the buffer.
 */
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>

char * smallocprintf(const char * format, ...)
{
    char * buf = NULL;
    size_t nonnullcount = 0;
    va_list args;
    va_start(args,format);

    nonnullcount = vsnprintf(NULL,0,format,args);
    va_end(args);
    buf = malloc(nonnullcount+1); /* +1 for the '\0' */

    if (buf == NULL)
        return NULL; /* Leave errno intact for the caller to deal with */

    va_start(args,format);
    vsnprintf(buf,nonnullcount+1,format,args); /* +1 for the '\0' */
    va_end(args);

    return buf;
}

Frankly I imagine that code like this exists in private libraries all over the place, but I hadn't seen it myself, and it does what I need. Late edition: In particular this seems to be a close analog to the GNU c library function asprintf though there are some interface differences.

---

¹ String litterals may be simply concatenated in code, but not null terminated character arrays and buffers.