The limits of "Fix it when you touch it."

My main projects at work have been on minimal-spend for a few months which means I've been shifted to some feature adds for our biggest product. This thing goes back to the mid-eighties and is coded in C (updated to ANSI syntax, at least), Fortran (updated to f90, at least), C++ (with the standard containers, at least, but lots of it predates the "modern" era), and python (recently ported to python3, at least). So, yeah, it has all the issues you'd expect in a legacy codebase. Some of them in spades.¹

We're basically a contract shop, so we don't do "Let's fix this entire module because it's grotty enough to be a pain", because who would pay for that? On the other hand, we really would like to have nice code, so we have a "You can fix issues with the bits you touch." policy.

Not complaining about that. My last feature add actually removed net lines of code because I replaced some really wordy, low-level stuff with calls to newer library features and factored some shared behavior into utility code to reduce repetition. So the policy makes me a happy (and perhaps even productive) programmer.

But it has it's limits. The short version is some legacy issues span a lot of code and you can't fix them locally.

Case Study

The bit of code I'm working on right now has a peculiar feature: in several places I find a std::vector<SomeStruct> paired with a count variable.² They appear in an effectively-global³ state object and they're passed to multiple different routines as a pairs. This is very much not what you'd expect in code originally written in C++.

Sometime in the past this was almost certainly a dynamic array coded in plain 'ol C. And not even a struct darray {unsighned count; SomeStruct * data}; one paired with some management functions, but a bare, manage-it-yourself-you-wimp pairing of a count and a pointer.⁴ But why wasn't the count discarded when transitioning to std::vector?

Finding out requires a lot of tedious, close reading of code where the pairs are used. And, of course, seeing the old C code behind the current C++.

There are several places where the vector is resized (meaning multiple extra entries are added to the end in one fell swoop) to some "bigger than we need" value. Those extra entries are filled with default data, which the code then overwrites one at a time with freshly calculated "actual" data. This is (a) a performance optimization insofar as it prevents the possibility of multiple re-sizes and copies that exists if you added entries incrementally, (b) a fairly faithful transliteration of what would have done with the dynamic arrays in C, and (c) the wrong way to perform the trick with std::vector.⁵

When this was originally done in C, the "count" variable would have tracked how many entries had "good" data while the allocated size would have been known because you knew the maximum expected size was. But the container doesn't know that you want to do that and it's size() method will always return the number of objects it has (including the default valued one). So the manual count was still needed in places, and they kept the (effectively) global version because it was easier.

Result: getting rid of the extraneous count variable means fixing half-a dozen routines elsewhere in the project and I end up touching scores of lines in a dozen files. That's not "fix what you are touching anyway".

Takeaway

Some legacy maintenance is too big for purely local fixes.

Today was actually the second one of these I've looked at in the last couple of months. I was able to fix the first one mostly with global search-and-replace and only touched five files; I felt that was "local" enough for the payoff in terms of making the code more comprehensible. So I was optimistic on this one, too, but it quickly grew out of hand. None the less, I may be finished and if the regression tests are clean I'm going to commit it.

---

¹ But, honestly, I worked with worse in my physicist days.

² For those who don't do C++, the standard vector container is a extensible array-like data-structure. It maintains its own count.

³ Possibly a subject for another day, and another example of something you can't re-factor in the small.

⁴ In the Bad 'Ol Days, a significant number of programmers would begrudge the cycles lost to function calls for that kind of things when it was "easy" to do it inline at each site. Of course, they could have used a macro DSL for the purpose, but those are tricky and (even then) had a mixed reputation.

⁵ It's wrong for two reasons in general. The less important one here is that it default constructs the new values which can take cycles (in a C dynamic array using realloc means you just get whatever garbage was in the memory occupied by the new spaces so you don't pay for that). The more important issue here is that vector has reserve which unlike resize just makes sure you'll have room for the new stuff if you want to use it, meaning you can then emplace_back for the best of both worlds.

You. Have. Got. To. Be. F*<#!^&. Kidding. Me.

There must be things in the communication world that make me angrier that a computerized voice prompt system that makes "typing" noises at you while it works, but right now I can't think of one.

Didn't expect it to be quite that hard

I'm getting my first non-trivial experience with JavaScript. It seemed like a good idea at the time.

Backing up to explain, I wanted to write a talk for my colleagues for some time. On C++ templates. Another thing that seemed like a good idea at the time.

Anyway, there is some interest around the office in a brown-bag lunch series, so I've actually gotten serious about writing it. And I want to own it, so I"m doing on my own time and resources, but that means I don't have a copy of PowerPoint. Not that I'd want to use PowerPoint, anyway, of course. Now in my academic days using $\LaTeX$¹ would have been a no-brainer, but I've been watching conference talks given in various HTML/CSS/JavaScript engines recently and it seemed like an ideal time to try something like that out. Yep, seemed good idea again.

Enter reveal.js, but also a headache brought on by perfectionism.

You see, it's to be a talk about a feature of a programming language and that means code samples sprinkled throughout. No problem, reveal has support for that. Only it would be nice if the code samples were syntactically correct.² So, I'd like to...

1. include the text of external files in my presentation
2. wrap that text in whatever html is needed to get reveal to mark it up as syntax highlighted code blocks
3. be able to compile the code in a separate tool to check it's correctness

Well, number three is entirely on me and it's easy. But both of the other gave me some trouble though it turns out I didn't need to know how to do the second item at all.³

It was the inclusion problem (item 1) where I got the surprise. You see, html doesn't have an inclusion primitive. You have to use JavaScript if you want to do that on the client side.⁴ Fine. Apparently that was messy in the past, but modern JavaScript has fetch to get the contents of an external asset and lots of DOM nifftiness to stick it where you want it with various kinds of preprocessing (as html, as script, as text, and maybe some others).

So, in the first place I want a code sample I write a little script that runs code = fetch("myfile.h") and then inserts it in a useful place. Reload. Nothing appears. Force reload. Still nothing. Open the inspector and find the tag. The tag is empty. What? Check paths. Force reload. Still empty. Try with "./myfile.h". And with "/home/noswampcoolers/projects/template_thing/myfile.h".

At this point I took some time off to curse many things. Including my own ignorance, but mostly other people and tools. Debugging by rant. Or something.

Finally notice the tabs in the developer tools. Particularly the one named "Console" and the red decoration thingy. Oh, there's an error; something about a "same origin policy" not letting me do what I want. Even though the data file resides not only on the same filesystem but in the same directory.

More ranting, then hit the web for some education.

Turns out the same origin policy says what external data can be loaded without a security negotiation. But it's not terribly well named because while it does put limits on the origin it also imposes limits on the protocol.⁵ Short version: I'm going to need a web server running on my machine to fetch stuff from the local filesystem.

The first solution I found for just serving a named directory was python3 -m http.server --directory "$PWD" 8080 > server.log 2>&1 &, but I think I could probably do the same thing with some npm invocation which would be more elegant in the sense of using the same tool for both halves of the task. It's also not worth my time right now. Python it is.

So at this point I could load some external code into my presentation.⁶ But it wasn't getting highlighted the way it did if I wrote it right in the html file. Argh!

Maybe it has to do with the order various scripts run in. Check the web and try moving my bit around. Lots of places. No dice.

More reading suggested that I needed to integrate my code with the framework by writing plugin. And just for once in this saga I was smart enough to find at least three existing plugins before starting down that rabbit hole. No idea if the one I picked is the best, but it seems to work.

All-in-all a enlightening but rather disheartening epic to achieve what sounded like a little thing. I'm sure there is a deep story in how it got to be that way, starting with "first to market is more important that getting it right" and moving on through various pitfalls of the standards process. I'd love to read about it if someone else would research it.

---

¹ For a long time I had a custom set of extensions to slides, but eventually I graduated to beamer.

² Semantically as well, of course. For those fragments that have semantics. But that's a lot less amenable to automated validation.

³ The issue I had with the second item is that I wanted to show C++ template code, and that means things surrounded by angle brackets: < and > which is, of course, the tag syntax of html. Before I solved item one, I learned a trick for that. You can include the text as the contents of a script tag marked with attribute class="text/template".

⁴ Lots of choices on the server side, but lets not go there.

⁵ I was too focused on solving the problem in front of me to read into the details, but I suppose this has to do with preventing externally supplied code from reading your locally stored data and sending the it back to the mother-ship. Which would make sense but it still leads to a minor absurdity.

⁶ Somewhere along the way I'd also taken the time to run down a DRY solution to including many snippets and adapted it to produce the same "pre-code-script" nesting I had learned for including code directly in the file. Shout out to Mustafa even though I didn't end up using the code. Plus, I enjoyed recognizing the immediately-invoked lambda from it's use as an initializing idiom in C++.

Really, Qt?

Qt Creator is poluting my nice clear build results window with a complaint that my build directory is not on the same level of the filesystem as my source directory. It's a new complaint even though I've been using Qt Creator for ages, because I recently had to re-oganize my work a little. I have, for a long time, used a layout where by source, build, and distribution directories for each project are sub-directories of $HOME/source, $HOME/build, and $HOME/dist which actually meets Qt's strange requirement.¹ Alas IT issued a new edict concerning the centalized backup system and on windows I've been forced to move my source hierarchy to $HOME/Documents/source on that platform. Which broke QT's little brain.

The policy is, presumably, intended to enfore a prefernce for out-of-source build, which is a good idea (every build should be out of source). But it is also way, way too restrictive a way to enforece that behavior.

---

¹ This lets be share the source directory across the Windows host, several WSL linux installations, and the few VMs I still run while each platform maintains it's own build and distribution locations.

Why isn't there a slider version of QInputDialog?

I need a quick-n-dirty UI element in my QT-based application for selecting a number. A number with a well defined lower limit (certainly not less than one, but honestly even 5 is kinda silly) and a fuzzier but still very finite upper limit (somewhere between 60 and 100, I'd say).

Now, there is QInputDialog::getInt which is what I'm going to end up using, but ... it's a natural use-case for a slider and there seems to be no similar convenience class supplying that particular functionality. Grrrrrrr!

The invention of fanfic

I've been reading The Hobbit to the child for bedtime story recently. Bit of a slog at times because (a) she falls asleep in the most surprising places in the text, (b) frequent stops to discuss new vocabulary and previously unknown customs, and (c) lengthy interruptions to hear how she things the story should proceed. That latter bit came to a head tonight as we neared the end of Chapter 8. She believes that the story would be better if we dispensed with the spiders and added some unicorns and has decides to make up her own, improved version.

What if it's untestable?

I've been working on evolving some code down in my project's foundation to enable to user visible improvements. My stategy involves the C++ standard library facility variant which allows the user to stores a single value at any given time but from multiple different types. For those of you with the proper background this is an improved union.¹ It also provides some basic infrastructure for the situation when it does not hold a value at all, and I've been diligently covering my a all the bases as I go along.

Now I want to test so aspects of the work I've done so far.

Testing the "variant is in a bad, bad place" code means making a variant in a bad, bad place. Only it turns out that is not at all easy.² There isn't a facility to make one. Not brand new to made-to-order, nor from an existing one by any usual means. cpp reference says

A variant may become valueless in the following situations:

• (guaranteed) an exception is thrown during the initialization of the contained value during move assignment
• (optionally) an exception is thrown during the initialization of the contained value during copy assignment
• (optionally) an exception is thrown when initializing the contained value during a type-changing assignment
• (optionally) an exception is thrown when initializing the contained value during a type-changing emplace

and it includs a sample of code to do the job, and a link to an online platform to compiler and run and see that it does the job. Now that code uses a custom type so I can't use it directly, but where I try to ape it using one of the types in my variant ... it doesn't work. I'm able write code that will hit all the last three conditions, but apparently my compiler is too smart for that.

Well, given how hard it is to access that state maybe testing the code that handles is houldn't be a high priority, but I"m annoyed.

---

¹ Nor is it anything like a new idea, but it wasn't made part of the C++ standard library until the 2017 standard. Thankfully, by project has moved that far forward in time.

² Of course this is a good thing. Except that it isn't quite impossible, so there is all that infrastructure there for handling the special case. Argh!