Showing posts with label VersionControl. Show all posts
Showing posts with label VersionControl. Show all posts

2025-12-19

Little surprises #10: iterators, half-open intervals, and git range selections

Let's talk about for loops. That's not actually the subject of the post, but we have to start there, hit a couple of stops along the way, and only then complain about git.1

Back in the misty depths of time, there were programming languages that indexed their arrays starting with one. If you write that in a c-like syntax you end up with something like

int array[ARRAY_SIZE];
// fill it somehow
for (size_t i=1; i<= ARRAY_SIZE; i++)
	print(array[i]);
where the end condition is controlled by <=. That has never really gone away—despite the efforts of some influential people—but I want to direct your attention briefly to the same for-loop in a language that uses zero-indexed arrays. Now idomatic code uses a strict less-than comparison:
int array[ARRAY_SIZE];
// fill it somehow
for (size_t i=0; i<ARRAY_SIZE; i++)
	print(array[i]);
but you could also use a not-euqal-to (!=) because the upper limit of the loop no longer represents a cell of the array that is accessed. Instead it represents where a cell would be if it were one-past-the-current-end.

Pointer and Iterator loops

Of course, you don't have to use an index for a loop counter. K&R is full of examples where they use a pointer for processing a (null terminated) string:

const char *s = "Yellow whirled!";
for (char *p=s; *p != '\0'; p++)
	print(*p);
or a linked list:
LIST_NODE *list;
// fill it somehow
for (LIST_NODE *p=list; p != NULL; p=p->next)
	print(p->payload);
In those cases the termination condition is a not-equal comparison to something that is (again) not processed.

And that is the pattern than many languages (especially the ones I use regularly) use for iterating a (possible subset of a) collection. The beginning iterator designates something that is to be processed and the ending iterator either signals the lack of further data or designates data to not process. In C++ this is the pattern for the algorithms library (where you hand the routines explicit iterator pairs), and by ranged-for loops where the syntactic sugar calls std::begin and std::end on the object to be iterated. The end-marker does not represent data to be processed.

On to git

Some git operation let you interact with groups of prior commits. To inspect a sub-set of the history, run a partially automated bisection process for finding where an issue was introduced, or (as I was doing when I discovered2 this behavior) to select a set of commits to "cherry-pick" to another branch. Unsurprisingly the command line syntax needs you to say what commits you want, and listing more than a few hashes explicitly is a pain. Luckily there is a "range" notation: [Early commit reference]..[later commit reference].

Now, if you use this notation with cherry-pick it will apply your selected commits to the new branch in time order: starting with the earliest selected commit and working it's way steadily toward the latest. Which is exactly what you would expect.

What you probably wouldn't expect (unless you, ya'know, actually read the documentation), is that it will exclude the earliest commit you entered and included the latest. This was so surprising to me that I actually did it wrong a second time before I started scouring the manual for an explanation.

That is a half-open range like we find in the discussion of iterator for-loop above. But instead of being [start, end) it is (start, end]. What. The. Absolute. Heck?!?

That said...

Interestingly, if you look a little deeper, there is a reason. Or at least a way in which this behavior arguably conforms to my description above.

Start by understanding how a git repository is structured. Each commit contains a record (effectively a pointer-to) all the commits that act as its parent(s) (there might be zero, one, two...). But commits are immutable once created, so they can't be amended with pointers to their descendants. The effect is that (assuming your range selection represents a branchless part of the graph3) you're looking at a linked list with the last commit as the head and the earliest commit at the end (of the range you care about, anyway).

So, even though commits are applied in time-order, they are found starting from the last event and working back in time. The docs make this clear: the selection is all the nodes reachable from the final commit, but not reachable from the first commit.4

And in that view, this is exactly like the linked list example I wrote above.


1 A little like eating your vegetables before dessert.

2 The mere fact that it is clearly documented if you bother to read doesn't affect my right to use that word. Does it?

3 I haven't actually tried it yet, but I imagine that selecting a span with this syntax that has a merge in it would include a large historical sub-tree you don't want.

4 And a commit is reachable from itself, naturally.

2025-06-17

An orange ring for git submodules

At work we have a common library, call it libThing, that underlies several of the products we produce for one of our big customers. We bring it into the projects that use it with git submodules. And that's a bit of a problem.

Let me explain.

You see, the design of the submodules facility is clearly predicated on an understanding that the submodule is a separate thing, and is not edited in situ by a programmer working on the super-project. If the sub-project needs updating, it is assumed, you will send the maintainer a well defined change request, wait for it to happen, point your super-project at the updated version, and go about your business.

Which is what you would expect if the sub-project belongs to someone else.

And some of our changes look like that: "Folks, we have a note from the customer. There's a new file format for specifying widgets. Someone needs to update the WidgetLoader in libThing to handle it. Joe, you've worked in that module recently, can you get to it this week?". Fine. Joe updates libThing, pushes to the reference repository and the next release of each of our project can manage the new Widget files. Nice. And exactly as Linus envisioned it.

On the other hand a lot of times we find out that libThing needs updates because we're in the course of making changes to one of the project that use it. By working both side together we can work through the trade offs dynamically. Its more natural, and probably faster, to just work on them together. Even though submodules doesn't encourage it.

Buuuut ... if you're not careful, you'll make one or more commits to the sub-project in a detached head state.1

The rest of this post is a recipe getting safely back to a happy state after you make this mistake.

Advancing the branch you aren't on

If you had done this right, you'd have put the subproject on the branch before you started editing and the branch would have advanced as you made commits. We want to get the repository into the state it would have had.

  1. Determine the name of the branch you're suppose to be on by looking in the .gitmodules file of the super-project. Remember that for later.
  2. Give the current state of your work in the submodule a temporary name with git checkout -b life-preserver2
  3. Get on the right branch with a git checkout to the branch you found in step 1.
  4. Fast-forward with git merge life-preserver (and it should be a fast-forward merge; if not you should probably take stock at this point.)
  5. Check that nothing is missing by examining the commit tree. I like gitk --all.
  6. Assuming all is well, dispose of the evidence with git branch --delete life-preserver.
  7. Detach the head again (at least if you're done) git checkout --detach HEAD.
  8. Pretend you're the kind of coder who never misses the little details.

1 If you don't work with git, then this post probably isn't much use to you, but short-short version:

  • In git a "branch" is a name that refers to some remembered state of the project. A project can have a lot of branches, and you can start with one state and make different changes to the project and have both of the remembered as different branches. That is, they can split off like the, well, branches of a tree.* They can also join up which is called a "merge".
  • You can tell git to remember a new state of the project. That's called a making a "commit" and each commit knows about the one(s) it came from, so the software can navigate back in time.
  • A commit can be the target of zero, one, or more branches.
  • You can be "on a branch" meaning that (a) you are working on the state of the project remembered by that branch name and (b) git has a record of which branch you're "on".
  • When you're on a branch and you make a commit, git changes the association of branch name to the new commit. Remember that commits know what came before, so you can still go back, but the name now refers to the new state of the project.
  • If you're not on a branch you are in a "detached head state" which means git knows which version of the project you start with but doesn't know a branch. Any commits you make in this state are nameless because there is no branch name to move forward.
  • * In computerese it's actually a directed acyclic graph (DAG), but that's neither here nor there.

2 Yes, I have a name I use for this. Not that it happens often or anything.

2024-04-04

Yeah, well, you know, that’s just, like, your workflow, man.

I caught some flack at work this week: I circulated an early draft of a document that I was struggling with in plain text1 and my boss was very clear that he wanted me to use Word in the future so there would be change tracking and out-of-band comments. On the plus side those remarks came packaged up with some useful suggestions for the piece.

Once I tamped down my reflexive defensiveness and the basic anxiety that comes with screwing up at work, I pulled up my big kid underwear and moved on. Then, having decided to be an adult about this, I ran smack dab into a counter example for $BOSS's point. I received a second set of highly useful changes in the same document. Conflicting changes. I'm not aware of any good tooling to handle conflicting changes in Word, but it was no problem for me to handle the conflicts in my text document: I just opened the files in my favorite visual merge tool.2 and got on with it.

Caveat time. To take the "plain text means we can use good tools" thing seriously we'd want to put all our draft work in VC repositories, and when that occurred to me my first reaction was "Who'd want to do that?" I mean, yeah that makes sense for major pieces of writing, but it's not obvious that you want to maintain a full history on every minor document you bang out day in and day out.

But then I had another thought...

Caveat on the caveat. Which was "Hey, how do people who are really committed to Word deal with the possibility of conflicting changes, anyway?" A little poking around the web suggest that my employer's answer is completely mainstream. At the management tier we put everything in SharePoint and let it enforce serialized editing, so they're already putting all their work in a repository. Maybe the whole idea isn't so silly after all.


1 Now, I would never send plain text to the clients, but I often do my initial composition in text because the sense of informality helps me feel safe trying out different formulations in search of a natural arc through complex subjects.

2 Meld as it happens. But not because I've tried all the options: it was just the first one I spent any time with and it's been consistently available.

2023-02-03

Now what?

Challenge of the day:

Search the web for a git workflow that is suitable for cases where you need to take a sub-module through a non-trivial evolution (that is something you'll be working on for a while, making multiple commits on and want to share with your colleagues as you go).

Go ahead. I'll wait. Probably for quite a long time.

You see, the web is teaming with tutorials and workflow articles on submodules, but they rarely get farther than setting up a repository and/or checking one out. At that point the article is already too long for a blog post: you readers are bored and have deleted the browser tab displaying your hard-won expertise.

I thinking of trying something like

  1. Create a pair of custom branches with the same name in the parent and child repositories.
  2. In the parent, edit .gitmodules to point at the branch im the child.
  3. In the child do the first step of your evolution.
  4. Push both and inform your colleagues
  5. Continue working in the child; push and inform your colleagues when appropriate.
  6. Once your colleagues have approved the work, merge your branch in the child to whatever the main development branch is.
  7. Edit .gitmodules in the parent to point to the merged code.
  8. Push both and inform your colleagues.

Idle curiosity

What was the ratio of torches to pitchforks when the mob came to make Linus pay for git submodule?

2022-11-20

Not actually that bad (AKA git submodules part 2)

In my last post, I vented some frustration related to a work project. At this point, enough progress has been made to walk back the most wild speculations. Unsurprisingly, part of the problem was me, though that leaves the tool to take some of the blame.

At the end of the last episode we had explored the technical reason you can't simply chain a series of clones of a repository using submodules. Depending on the way submodules are identified, you may be able to work around the limitation (what I've done)1 and you may be able have all clones use the same (possibly thrid-party) master repository for some submodules. Depending on your use case these two options may be sufficient, and in the case of third party modules the latter may the Right Thing (tm).

At that point I actually had my local server copies in place but it wasn't working right. I'd started my investigation and gotten far enough to write

Only now there is the matter of branches and tags.
without having quite solved it. The symptom I had noticed is that there was a branch, call it develop, on the central server that I couldn't checkout from the local server. If I drilled down on the hosting website to find hashes for the version of the files I wanted to look at I found that those hashes were present on the local server. But the branch wasn't.

So, what is a branch, where do they come from, and how do I make sure that the local server has the ones that are on the central server?

A branch (and indeed a tag) is just the associate of a textual name with a particular commit. This class of objects are called "refs".2 And while cloning copies all the contents (commits, trees, and blobs) of the cloned repository it performs some bookkeeping on refs. Moreover exactly what bookkeeping is performed depends on how you run your clone. The gory details are available thanks to stackoverflow user Cascabel and editors, but the long and short of it is I had created the repositories on the local server using git clone --bare when I should have used git clone --mirror.

Sigh.

Thankfully, stackoverflow user onionjake knows the incantation to fix it up in place.


1 In the work-around the non-terminal repositories are in no way unified. You check out the top-level without recursing into the sub-modules and then check out each submodule separately and carefully locate them on your file-system relative the top-level in the way that the terminal repositories are going to expect. The downside of this is that there is no tooling for keeping them in-sync. I suppose I'll write a script. In python, perhaps, because I'm trying to get away from using unix-specific tools for things that could be cross-platform.

Verbing weirds language.
Calvin
2 The main difference between branches and tags is how the associated behaves when you git commit. Tags simply don't care, once you set them up they are fixed and always point to the same commit, but branches can move. Git keeps track of what branch you are "on" and when you perform a commit action, it moves that branch to point to the newly created commit object. Of course, using the same word for the noun and verb is not in the least confusing.

2022-11-16

Is git submodules really this bad? (part 1)

As I mentioned, I'm "getting" to use git submodules, and my frustration level is making a bid for a new personal best. My sense is it's a little clunky even when you use it exactly as envisioned and breaks completely as soon as you stress it. I hope I'm exagerating or outright wrong because I have no choice but to work wih it.

Here's the problem: we're working on one small piece of a larger project (a plugin, as it happens), and the project management is security conscious enough to:1

  • Use submodules as part of a system to selectively limit access to the repository: I can see all main API headers and only those implementation details I will be working directly with. I don't "need" the rest because I can test my plugin against a binary disribuion of the core program.
  • Make it quite a gaunlet to get individual credentials for direct access to the central repository.

To avoid sending each member of my team through the gaunlet as they join I thought "Oh, git is a distributed system,2 right? I'll just create a local working repository for my team and we can push back upstream when we're happy."3 Which is, evidently, not something the designers of submodules anticipated.

The core issue is submoules are found by follwing a either a path or a url which has implications for how a clone of a clone works in projects that use submodules. Look at the contents of a .gitmodules file: for each module there will be a url tag. That tag may be formatted as a filesystem path telling git where to look for the sub-repository on the filesystem where it found the super-repository, or as a url telling git where to find the repository on the wider network.

Image that there exists a project on a central.server: central.server:/repos$ ls compoundproject.git includedproject.git utilityproject.git central.server:/repos$ cat compoundproject.git/.gitmodules [submodule "IncludeProject"] path = IncludeProject url = ../includeproject.git [submodule "UtilityProject"] path = UtilityProject url = http://central.server/repos/utilityproject.git and note that I've rigged the two submoules to use different logic about finding their related repos, but that they will both find the one on the central server.

Now I create the repository for my team (there are slight differences if we make this a bare repository): local.server:/home/git$ git clone --recurse-submoules http://central.server/repos/compoudproject.git [...various git output that looks good...] local.server:/home/git$ ls compoundproject local.server:/home/git$ ls -A compoundproject .git .gitmodules IncludeProject UtilityProject [...some top-level contents...] and if we peak in the sub-project directories we'll see the expected contents.

Next a member of my team tries to set up a working repository developerworkstation:/home/developer/Projects$ git clone --recurse-submoules http://local.server/home/git/repos/compoudproject but this is going to fail when it tries to get IncludeProejct because it is going to look for it at http://local.server/home/git/includeproject.git instead of at http://local.server/home/git/compoundproject/IncludeProject, and if we assume that the developer does not have credentials for central.server then it would also fail when trying to get UtilityProject because it gets that from the central source.

Now, I can solve the first problem by (bare) cloning includeproject.git to local.server beside compoundproject.get. The second problem can only be overcome by getting the developers credentials for central.server.

Okay, backup and replace the subjunctive above with my actuall situation. In effect utilityproject.git is actaully reached by relative reference just like includeproject.git. Consequently I have made bare clones of all three projects on local.server and my developers can do a git clone --recurse-submoules http://local.server/home/git/repos/compoudproject.git and get all three. Yeah! Go me!

Only now there is the matter of branches and tags. I'm not sure I understand this, so the saga will have to continue another day...


1 Coming in heavily on the side of security in the security-versus-getting-things-done trade-off is par for the course in my industry. I've been sighing a lot about this but I'm not at all surprised.

2 Big selling point, right? Every repository is equivalent and you can move updates from any repository to any other repositorye. Of course, the way people actually use DVCS there is a (or are a few) repositories that are central to the workflow even if they are not special to the underlying software. For that matter in git those ones are usually configured as bare repositories so there's feature support in the tool for the distinction. But it is still better than using SVN.

3 Bear in mind that this plan would work just fine for a plain boring project that used git without any sub-whatevers.

2022-11-08

Mystery of the Month Club

Git.

The git parable does a pretty good job of explaining why git makes sense as an abstract tool, and of preparing you to understand other articles on why you should or should not git in various ways.1

But git will still surprise you. I think I sussed out the bit that confused me today. Maybe. It's all to do with submodules. We've opted not to use them locally because we were convinced we didn't understand all the implications. I think we were right but I've gotten involved in a project to write a plugin for a third party tool which does use submodules, so I'm getting to learn.

Of course, submodules is just one option in the mix-n-match-repositories marketplace (along with subrepos and subtrees as well as various wrappers). Probably because the use case was not part of the original design and all the patch-it-up-afer-the-fact schemes have realy drawbacks.


1 Did you notice that the cherry picking article comes in 10 (!) parts. That's because, elegant though it is, git's underlying graph theoretical model requires you to keep track, not just of a DAG, but of an evolving DAG. And also because the series actually explored multiple issues. The first four or five articles (if I recall correcly) treat the main subject and the rest explore extentions to the basic idea.

2022-11-01

Merge tool supporting on-the-fly revision history?

There is this internal library where we've been working on four branches. Call them devel, feature-a, feature-b, and feature-c. All three feature branches are significant and long running, but they have at least had the incremental improvements and bug fixes from devel regularly merged back in.

Recently feature-a and feature-b were merged to devel (by someone else, yeah!). Which means that devel now has large changes relative feature-c. Now, I'm one of the folks working on feature-c and I spent a little while today trying a "what-if" merge of devel into that branch. It wasn't as bad as I feared (because much of this branch is working on different parts of the code than the others), but it was bad.

While I was staring at the merge tool showing another set of conflicting changes without (a) recognising either change as one I had done or (b) understanding the intent behind either occured to me that it would be really nice to be able to ...

Frob1 a highlighted (i.e. changed) section of text to obtain a quick peak at the relevant change logs.

In the simplest form you would see the change summaries from the current commit back to the common anscenstor for each branch; better would be filtering only those commits that touched the current file, and best would be filtering those that involve the lines in question. Being able to drill down on interesting summaries is a bonus feature.

At that point—assuming your team writes reasonable commit messages—you have a fighting chance of sussing out the intent of the changes and thereby a better chance of choosing the right merge action. Of course, I can (and do) check those logs in a couple of terminals kept handy, but UI sugar can improve the experience.

This seemed like something someone might already have implemented, but my Google-fu wasn't up to finding it (or reliably eleminating the possibility). Anyone know?


1 Hover over, center click on, or something. Ask a UI specialist to help.

2021-06-28

Oh, no! It's worse than that!

Me
[Rants for a while about a desired mechanism for solving a technical project-management issue at work]
Better-half
But there's no way to do that?
Me
Oh, no! It's worse than that! There are at least six ways!

Seriously, I'm spending too much time reading about `git-subwhatever`, build-system support for external dependencies, and even dependency management systems (though this really shouldn't require going to those lengths). And I'm really thinking about suggesting we do this the old fashioned way.1


1 Take the bits we want to factor out and share between projects, make each it's own project that builds a library, and link the libraries in the overlying projects. Done and dusted. Except for the hassle of getting a new hire up and running; and versioning issues; and remembering where each bit actually exists when you need to work on it; and deciding on, setting up, and enforcing commit privileges for each library; and extra deployment complexity for software we ship. But at least those are all known problems.