I'll have the BeautifulSoup

We've delived version 1.0 of my project at work; or perhaps we're in the process of delievering it.

We've handed over a complete copy of the repository and deployable binary bundles for three targets and VM based development environments for two.

We still have some time to complete and hand over the documentation. Better still, by policy I set very early on all the documentation is in the repository, so they already have it. And I tried to keep us writing and updaing documentation often enough that it didn't get too far out of synch with the code, so the documentation they have is pretty good. But we're taking this time to make it better: more complete, easier to understand, and reflecting the code we handed over even better.

There is a twist: even though the project is entirely in-house, it's paid for by a contract to which we are a subcontractor. This means that the prime contractor has the responsibility to report on the work (and the authority to make demands about that). This particular prime contractor has a tradition of including the user manuals for software projects as appendicies to the report.

But my main user documentation takes the form of a pile of individual html files that are displayed in a on-line help system. There is no linear structure imposed and no single document containing it all.

So I needed to flatten it.

There are only 20 files right now, so I could have done it by hand. But that would have taken a few hours or more to get right (all the internal links!) and I have reason to believe the project will be funded for further development meaning I'll have to do it all again in a year or two. That's excuse enough to automate the task.

The tool I settled on was a python library called BeautifulSoup which I had heard of but never used before. I loaded a html/css scaffold file, looped over a hand-tooled list/tuple structure placing my files into a few categories. In the loop I created a table of contents entry for the file; grabbed the file body, re-wrote the header levels, munged the anchors and links, base64 encoded the images, and wrapped the result in a section for nice formatting; and then appended the new section to the body of the scaffold. When the loop ended prettify and write. It just works. I'm hooked.

On styling text in a QTextBrowser

My project at work is complicated enough to need fairly extensive on-line documentation, which we write as html and diplay to the user in a QTextBrowser. Now, this thing is not a full web browser¹ and only a supports a limited subset of html/css.

Suffice it to say that if you're just had a clever CSS idea it's not supported by QTextBrowser.

At first this didn't bother me too much because I had never done css before. But as I've become more familiar with the tool (and at once awed and appalled by it), I've become aware of just what I'm missing out on. I wrote the quoted sentence as part of our guidelines for writing and formatting the help files.

I'm sticking with QTextBrowser for now because our needs are simple and we can't really expect an incoming junior dev to be able to support a sophisticated pile of css anyway.²

---

¹ That's available as QWebEngine, but it brings in a large module so I didn't want to use it.

² Because of the nature of our work, our target demographic for new hires is "scientists and engineers who can program" more than it is people with a strong development background. Having web skills would be a bonus, but it's not part of the expectation.

Duty and painting oneself into a rhetorical corner

Duty is a debt you owe to yourself to fulfill obligations you have assumed voluntarily. Paying that debt can entail anything from years of patient work to instant willingness to die. Difficult it may be, but the reward is self-respect

Robert Heinlein

"Duty" in the sense of an obligation is a politically contentious idea. I am personaly a fan of Heinlein's definition, but it suffers for some people's purposes from not being something that you can dictate as in "Doing [this thing I'm interested in] is your duty". The best you can do is explain to them why you think their self-respect should depend on doing it.

As a legal reality a few "duties" are required and you can be punished if you don't comply. If you are summoned for jury duty and neither provide an excuse the court finds acceptable nor show up you can be held in contempt and subjected to fines or jail time. If you are a young but adult male in the US and don't register with the selective service you can get in trouble. In the same vein, many people were punsihed during the Veitnam war era for draft dodging, which brings us to the culture wars.¹

The political right in the country largely partakes of a long standing view that military service is and should be required of men "in times of war" (which really means at the discretion of the political leadership). Some who don't go quite that far feel that there simply shouldn't be a way for a "real man" to not want to join the fight which is a slightly different take. In either case the point is that the community needs defense at times, and the costs should be widely borne.

The political left, on the other hand, mostly feels that it is the right of every citizen to question the motives behind and necessity of military action and to demur from participating if they find them unsavory or unecessary. Some go so far as to consider it an obligation to make such judgements. Some have even held unwilling particpants in military misadventures responsible for the failing of the political and military leadership under which they served.

That is a stark divide, and has been the cause of much dislike and mistrust between people on opposing political teams for the whole of my lifetime.

Then comes Covid19 and the issues of wearing masks in public, social distancing, enforced closures of some kinds of business and so on...

Suddenly there talk from the left of a duty to protect others by respecting these rules and talk from the right about their right to judge the urgency of the threat, the effectiveness of the suggested measures, and the motives of the leaders.

Laugh or cry.

---

¹ There is, of course, some tension between this notion and that of religious liberty. Quackers, various Menomnites and other conciencious objectors have been a bone of contension on this matter since the beginning of the nation.

Spherical cows #1: Face masks edition

Let's talk a little about the effectiveness of face masks and the way levels of participation in waering them change the overall outlook of a epidemic. I want to go there because I hear some complaints hear and there that cloths masks don't do "much" and so there is no point in waering them, and I don't think that position is terribly well thought out.

This is the first of a couple of articles I intend to write applying simple models to look for insight into policy choices related to infectous disease in general and aimed at understnding a little about Covid19 in particular. Thought I won't try to make the models numerically faithful to Covid19.

Basic reproduction number

In the jargon of epidemiology the "basic reproduction number" (givern the symbol $R_0$) is the number of people, on average, who any given carrier gives the disease to. In simple models, if this number is larger than one then the disease spreads throughout the susceptible population, if it is lower than one the disease fades away. And we can (again in simple models) further break this number into a product of the avearge number of people a carrier interacts with ($N$) and the average chance that they will transmit the disease in their interactions with a single person ($F$).

For our purposes we can say that masks reduce the chance of tranmission¹ Which means that the effectiveness of using masks translates directly into a reduction in $R_0$.

Guessing at the effectiveness of a properly worn mask

Here we consider "non medical" cloths masks that you make yourself or buy at retail. Now a mask could help in two ways: it can reduce the amount of virus a carrier puts into the environment and it can reduce the fraction of environmental virus that a un-infected wearer takes in. For simplcity I'm going to model these as each generating a equal reduction in the chance of transmission modeled by a multiplicative factor $f$ that is less than one (so it models a reduction) and greater than zero (which would be perfect effectiveness.

So, if a carrier has a single "interaction" with a uninfected but susceptible person the probability of transmission is:

• $F$ if neither is wearing a mask
• $fF$ is one (either one) is wearing a mask
• $f^2F$ if both are wearing masks

But what should $f$ be? I don't really know, so I'm going to guess. To be conservative I'm going to guess that they are not terribly effective. I'm arbitrarily assigning $f = 0.8$. That is, a single mask reduces your chances of getting infected by only one fifth, and both parties being masked gets you down to 64 percent.

What about compliance

Not everybody wears masks. This may be due to a oversight such a failing to grab one on the way out the door, lack of access, or by choice. It dosen't matter to the model. But what does that mean for the overall effectivenes of a masking policy. Not all iteractions will be between two mask wearers. Some will involve one mask and others no masks at all.

Call the rate of mask wearing $r$. If everyone wears them $r$ is one, if only people who were burned by acid wear them $r$ is essentially zero.

Sticking with our "do everything on average figures" approachand assuming that everyone has the smae chance of meeting everyone else² we can compute the overall fraction $\Gamma$ in a simple way \begin{align} \Gamma(f,r) &= \left[ (1-r) + f r \right]^2 \\ &= (1-r)^2 + 2f (1-r) r + f^2 r^2 ] \end{align} And that's the whole basic calculation. It's quadradic in $r$ and concave up. Done and dusted.

Except that we might want to manipulate the math a little in search of further insight. So we continue \begin{align} \Gamma(f,r) &= (1 - 2r + r^2) + f(r - r^2) + f^2 r^2 \\ &= r^2 (1 - f + f^2) + 2 r(f - 1) + 1 \end{align} Also quadratic and concave up in $f$.

Sticking some numbers in

Just looking around me I think it's little optimistic to expect more than 90% mask compliance, so a optimistic number of the overall effect would be $$ \Gamma(0.8,0.9) = 0.672 \;, $$ or a little better than a 30% reduction. That's not great but it is significant. If $R_0$ was as small as 1.45 that alone would be enought to squeeze off the epidemic.³

A more pesimistic value of compliance might be around 60%, leading to $$ \Gamma(0.8,0.6) = 0.774 \,. $$ which is to say less than a 25% reduction in the tranmition proability. For that to be enough we'd need to have started with $R_0$ less than 1.3.

---

¹ We could also say that stay-at-home orders and the like reduce the number of interactions. Social distancing could be modeled as a little of each. I'll say more about this in the next post.

² I'd guess that this is the worst assumption in this calculation, but making it better requires a huge increase in the complexity of the problem and probably necessitates a Monte Carlo (randomized simulation) approach. I'm doing a back-of-the-envelope calculation, so this is what we get.

³ I haven't seen a number recently but early reports suggested something larger than 2.0, and I suspect it might be a little higher than that simple because early on there wasn't much awareness of how many people get it but remain asymptomatic.

You win some, you lose some

On one hand, diapers seem to be out of our lives. On the other hand, potty emergencies are in.

Your messaging isn't reaching the virus

I talked a bit in a previous post about how nerds mostly don't rule the world. One of the big issues is that we think that we live in a world of facts, but most of humanity lives in a world of opinions masquerading as facts. And if enough people believe them then for political purposes they are facts. Politicians get their way in large messure by controlling the way people understand things; spin doctoring works at least some of the time. Which is why message control looms large in the minds of politicians.

But there are some things you can't message your way out of.

When you hear politicians complaining that too much testing is the problem you know they're locked in a mindset where messaging is more important to them than reality. They aren't making any progress in asserting their view of things because of the relentless drumbeat of daily figures, so rather than addressing the hard problem they try to just get rid of the reporting. Rather like the Ravenous Bugblatter Beast of Trall; which is to say "Daft as a bush".

---

Consdier, if you will, what we could do with enough data.

If we could test everyone all at once, we would have a very powerful tool for isolating the virus. Given that we can't do that, if we could at least test everyone who had been in contact with known carriers we would have nearly as good a grip, which is the point of contact tracing (note how quickly South Korea contained their initial outbreak as an example).

What we can actually accomplish is a lot less powerful, and without some help from the population at large it may not be enough to choke off the transmission, but asking for less data is the best way to guarantee that we don't stop this thing.

Squash bugs

My wife and I are gardeners. Not of any particular expertise, but definitely persistent. If I've counted our misses correctly we've been at it seventeen of the last twenty years. At nine sites in five states and six cities.¹ I believe we've only failed to garden at all in one of the eight "permanent" residences we've had in that time (a rental we occupied for less than a year).²

We grow flowers and vegatables in containers and in the ground. Roses, irises, marigolds, geraniums and others. Tomatoes, bell peppers, a variety of chilies, egg plant, leeks, beans, peas, assorted squash, selecetd herbs, occasional greens, the odd brasica and others. We use containers in rentals, when space doesn't leave us with a choice, and (in one case) when suspected soil contamination makes us unwilling to eat food grown in the ground.

By my count we've grown squash at seven of the nine sites. And we've had squash bugs in every single one of them. "Loathing" is simply not a strong enough word to describe my feelings about these little creeps.

They showed up here on Wednesday. I found them as the light was fading from the sky, and I spent almost an hour working my way through all our vulnerable plants trying to determine the extent of the infestation and to stop it. Each of the last few days has included an hour or more of fighting them, and we might be winning. It's only cost us one plant so far, which is not bad.

But the vile awfulness of these critters is not the point of this post. Rather it is the "discovery" I made that first evening.

Back-lit squash leaf at dusk

The hard part of hunting squash bugs is that they generally lay their eggs underneath the leaves, and when they hatch the juvenile stages spend most of thier time their as well. It's a lengthy and painstaking process (not to mention tough on the back and neck) to lift each leaf and look underneath. As the light faded, I fired up the light on my phone, and found that I could see my targets through the leaves.

It's much less work and easier on the neck to slip your phone under each leaf.³

In principle, you could have done this with any decent flashlight, but they generally have the wrong form-factor to manuever around the plant: the shape of a phone is much better for the application.

---

¹ Two of those sites were community garden plots maintained in parallel with gardens at our residences.

² The life of a repeat post-doc is not a stable one. We're good at moving and ready to stop.

³ Sorry, I don't have a photo showing any eggs or nymphs because I don't stop to document my discoveries but move to destroying them at once. It's the only way to be sure. But then, any gardener who's familiar with these things will have no problem spotting the egg arrays and the juvenile stages are pretty obvious as well.