A New One By Me

Our daughter is intensely interested in upcomming changes in her life. In particular she's aware that she is currently attending the most senior program as her pre-school (a transitional class the state will accept in lieu of kintergarten) and wants to know about where she will go to "homework school".¹

The phrase is descriptive and its meaning is clear, so it's a logical neologism, but I'd never heard it before. Is that a term of art among the pre-school set?

---

¹ A very good question about which we are currently having a slow motion panic. Much research is being conducted and many experts are being consulted.

API history

So I made some progress today on the Emacs configuration improvements I mentioned in my last post.¹ That work was performed using the partially completed configuration on my Raspberry Pi 400 (i.e using Emacs 27.1).

Then I pushed and opened a machine at work to try it out in a legacy environment (using Emacs 25.2.2). No go: one of the elisp functions I used in solving a problem apparently got more robust between 25 and 27.² Check the documentation to no avail. Not surprising when I think about it: no reason the on-line³ documentation for Emacs 27 should contain any history.

But having a central source with the history is hugely useful. I regularly make heavy use of cppreference.com where APIs are exhibited annotated with the version(s) to which they apply and historical versions are listed along side their more recent counterparts. The availability of the history makes it easy to see why you can't use that StackExchange solution as is, and why the other one seems clunkier than need be.

So far, I haven't found anything equivalent for Emacs.

---

¹ I also ran into that "supporting multiple versions is hard" thing even programming on a single machine. I used a function defined in the subr-x package meaning that until I re-implement that I'm limited to version 24.4 and newer. Sigh. We'll see about that re-implementation thing. I don't think I currently need to go that far back in time. If I still had to support that 2010 legacy platform it would be another issue.

² So now I'm building versions 24.4, 25.3, and 26.3 on my Pi so I can test a variety of version in one place. Might want to do a 28 and a 29, too, but let's start with what we've got, OK? Anyway, I remain impressed with the Pi, but trying to build Emacs in the background while doing other work does expose its limitations.

³ That is live-in-the-program rather than available over the network. How the language has changed, eh?

Compatability history

Have you ever noticed how hard it can be to find out which versions of software (a) were compatible with which versions of software (b)?

Documentation for software that is meant to interoperate with other software generally says something about which version(s) of the counterpart it works with, but no one has any particular reason to keep a list of the history of that relationship.

What brought this on, you ask?

I'm trying to modernize my Emacs configuration file (for which there are many luxurious options) any have reasonably uniform behavior across the several systems I work on and home and work. Alas, customer requirements at work mean I need to deal with systems dating from 2014 (and we're happy they've finally gotten serious about migrating off that platform from 2010).

All those, aforementioned, luxurious ways to configure a modern Emacs system (Spacemacs, Doom Emacs, and other "distributions") require you to be using, well, a modern version of Emacs. By which they mean v25 is definitely out and v26,fi supported at all, will soon by unsupported.

So I'm trying to do it for myself. My strategy depend on several things:

• add-package to centralize the configuration
• straight as a way to get packages
• the identification semantic categories of behavior I'd like my emacs to have and the understanding that those behaviors could be provided by different packages for different Emacs versions
• a system for selecting implementations of those categories by Emacs version number

I expect to make the project publicly available pretty soon.

But let's say my category is language server services. Popular suppliers seem to be eglot and lsp-mode. If I want to use lsp-mode on a up-to-date Emacs I'll just let straight grab the newest release of the package. But the newest version relies on variable that are not in Emacs 26, so I need an older version, but how do I learn which tag straight should get for those system?

And the problem repeats for every behavior which isn't handled internally.

Which, I suppose, is why the big distributions tell you "Use a modern Emacs, already!", albeit usually in more diplomatic terms.

But ... I knew that. I just didn't understand the importance of what I knew.

Still more Randall Monroe greatness.

What with one thing and another I learned about the TEA project recently. Ambitious stuff, and I'm completely unprepared to talk about the project as a whole, but being a ex-academic, I wasted no time downloading the white-paper, and brought along a dead-trees version for flight as we took our spawn off to visit one set of Grandparents.¹ The paper concerns itself with a distributed crytographic ledger to be used for archiving and authenticating the open-source ecosystem but also to provide a framework for distributing donations across that ecosystem.

The point is that much of the open-source ecosystem is un-sexy tooling and infrastructure that only excites the people who use it to write other open-source software, but donation come from either users (who tend to direct them to sexy top-of-the-stack end-user stuff) and companies (who direct them to things they are specifically trying to improve). Very little money flows to underlying stuff that "just works" even if it requires on-going maintenance to preserve that state of operability. There is a reason Randall uses the word "thanklessly" in the comic. Anyway, the paper observes that to perform such a distribution you would need to know how much each project contributes to the ecosystem, and then that a package manager (such as apt, pip, npm, or homebrew) is exactly an encoded version of that knowledge.

I'm pretty sure my jaw hung slack for the several seconds it took for my mind to re-orient on my freshly remade understanding of the open-source ecosystem. Of course, the package managers already know that stuff. At least collectively.² And maybe missing the dependency links implicit in rarely used build options. But for the most part it's there.

---

¹ Between the pandemic and our commitments taking care of honorary Grandma we'd been able to avoid this for some time. Now we will, no doubt, have to do the other Grandparents soon as well.

² It is worth noticing that there are multiple classes of package managers and build dependency systems out there, handling independent (well, largely independent) and distinct (again, largely) ecosystems. To perform as envisioned the proposed tea protocol will need to be flexible enough to express relationships within and between all such groupings. Not easy stuff.

Not actually that bad (AKA git submodules part 2)

In my last post, I vented some frustration related to a work project. At this point, enough progress has been made to walk back the most wild speculations. Unsurprisingly, part of the problem was me, though that leaves the tool to take some of the blame.

At the end of the last episode we had explored the technical reason you can't simply chain a series of clones of a repository using submodules. Depending on the way submodules are identified, you may be able to work around the limitation (what I've done)¹ and you may be able have all clones use the same (possibly thrid-party) master repository for some submodules. Depending on your use case these two options may be sufficient, and in the case of third party modules the latter may the Right Thing (tm).

At that point I actually had my local server copies in place but it wasn't working right. I'd started my investigation and gotten far enough to write

Only now there is the matter of branches and tags.

without having quite solved it. The symptom I had noticed is that there was a branch, call it develop, on the central server that I couldn't checkout from the local server. If I drilled down on the hosting website to find hashes for the version of the files I wanted to look at I found that those hashes were present on the local server. But the branch wasn't.

So, what is a branch, where do they come from, and how do I make sure that the local server has the ones that are on the central server?

A branch (and indeed a tag) is just the associate of a textual name with a particular commit. This class of objects are called "refs".² And while cloning copies all the contents (commits, trees, and blobs) of the cloned repository it performs some bookkeeping on refs. Moreover exactly what bookkeeping is performed depends on how you run your clone. The gory details are available thanks to stackoverflow user Cascabel and editors, but the long and short of it is I had created the repositories on the local server using git clone --bare when I should have used git clone --mirror.

Sigh.

Thankfully, stackoverflow user onionjake knows the incantation to fix it up in place.

---

¹ In the work-around the non-terminal repositories are in no way unified. You check out the top-level without recursing into the sub-modules and then check out each submodule separately and carefully locate them on your file-system relative the top-level in the way that the terminal repositories are going to expect. The downside of this is that there is no tooling for keeping them in-sync. I suppose I'll write a script. In python, perhaps, because I'm trying to get away from using unix-specific tools for things that could be cross-platform.

Verbing weirds language.

Calvin

² The main difference between branches and tags is how the associated behaves when you git commit. Tags simply don't care, once you set them up they are fixed and always point to the same commit, but branches can move. Git keeps track of what branch you are "on" and when you perform a commit action, it moves that branch to point to the newly created commit object. Of course, using the same word for the noun and verb is not in the least confusing.

Is git submodules really this bad? (part 1)

As I mentioned, I'm "getting" to use git submodules, and my frustration level is making a bid for a new personal best. My sense is it's a little clunky even when you use it exactly as envisioned and breaks completely as soon as you stress it. I hope I'm exagerating or outright wrong because I have no choice but to work wih it.

Here's the problem: we're working on one small piece of a larger project (a plugin, as it happens), and the project management is security conscious enough to:¹

• Use submodules as part of a system to selectively limit access to the repository: I can see all main API headers and only those implementation details I will be working directly with. I don't "need" the rest because I can test my plugin against a binary disribuion of the core program.
• Make it quite a gaunlet to get individual credentials for direct access to the central repository.

To avoid sending each member of my team through the gaunlet as they join I thought "Oh, git is a distributed system,² right? I'll just create a local working repository for my team and we can push back upstream when we're happy."³ Which is, evidently, not something the designers of submodules anticipated.

The core issue is submoules are found by follwing a either a path or a url which has implications for how a clone of a clone works in projects that use submodules. Look at the contents of a .gitmodules file: for each module there will be a url tag. That tag may be formatted as a filesystem path telling git where to look for the sub-repository on the filesystem where it found the super-repository, or as a url telling git where to find the repository on the wider network.

Image that there exists a project on a central.server: central.server:/repos$ ls compoundproject.git includedproject.git utilityproject.git central.server:/repos$ cat compoundproject.git/.gitmodules [submodule "IncludeProject"] path = IncludeProject url = ../includeproject.git [submodule "UtilityProject"] path = UtilityProject url = http://central.server/repos/utilityproject.git and note that I've rigged the two submoules to use different logic about finding their related repos, but that they will both find the one on the central server.

Now I create the repository for my team (there are slight differences if we make this a bare repository): local.server:/home/git$ git clone --recurse-submoules http://central.server/repos/compoudproject.git [...various git output that looks good...] local.server:/home/git$ ls compoundproject local.server:/home/git$ ls -A compoundproject .git .gitmodules IncludeProject UtilityProject [...some top-level contents...] and if we peak in the sub-project directories we'll see the expected contents.

Next a member of my team tries to set up a working repository developerworkstation:/home/developer/Projects$ git clone --recurse-submoules http://local.server/home/git/repos/compoudproject but this is going to fail when it tries to get IncludeProejct because it is going to look for it at http://local.server/home/git/includeproject.git instead of at http://local.server/home/git/compoundproject/IncludeProject, and if we assume that the developer does not have credentials for central.server then it would also fail when trying to get UtilityProject because it gets that from the central source.

Now, I can solve the first problem by (bare) cloning includeproject.git to local.server beside compoundproject.get. The second problem can only be overcome by getting the developers credentials for central.server.

Okay, backup and replace the subjunctive above with my actuall situation. In effect utilityproject.git is actaully reached by relative reference just like includeproject.git. Consequently I have made bare clones of all three projects on local.server and my developers can do a git clone --recurse-submoules http://local.server/home/git/repos/compoudproject.git and get all three. Yeah! Go me!

Only now there is the matter of branches and tags. I'm not sure I understand this, so the saga will have to continue another day...

---

¹ Coming in heavily on the side of security in the security-versus-getting-things-done trade-off is par for the course in my industry. I've been sighing a lot about this but I'm not at all surprised.

² Big selling point, right? Every repository is equivalent and you can move updates from any repository to any other repositorye. Of course, the way people actually use DVCS there is a (or are a few) repositories that are central to the workflow even if they are not special to the underlying software. For that matter in git those ones are usually configured as bare repositories so there's feature support in the tool for the distinction. But it is still better than using SVN.

³ Bear in mind that this plan would work just fine for a plain boring project that used git without any sub-whatevers.

Mystery of the Month Club

With due thanks to Randall Munroe, Comic Genius

Git.

The git parable does a pretty good job of explaining why git makes sense as an abstract tool, and of preparing you to understand other articles on why you should or should not git in various ways.¹

But git will still surprise you. I think I sussed out the bit that confused me today. Maybe. It's all to do with submodules. We've opted not to use them locally because we were convinced we didn't understand all the implications. I think we were right but I've gotten involved in a project to write a plugin for a third party tool which does use submodules, so I'm getting to learn.

Of course, submodules is just one option in the mix-n-match-repositories marketplace (along with subrepos and subtrees as well as various wrappers). Probably because the use case was not part of the original design and all the patch-it-up-afer-the-fact schemes have realy drawbacks.

---

¹ Did you notice that the cherry picking article comes in 10 (!) parts. That's because, elegant though it is, git's underlying graph theoretical model requires you to keep track, not just of a DAG, but of an evolving DAG. And also because the series actually explored multiple issues. The first four or five articles (if I recall correcly) treat the main subject and the rest explore extentions to the basic idea.