"Objective" assessment by questionnaire

Some time ago we received a recommendation that our daughter get screened for autism. Now, with generally overbooked state of medical specialists around here and the pandemic and all it's taken a year and half to make any headway, but the process is underway at last.

Great.

Part of that process involved a questionnaire for the parents. We were asked to rate our child's willingness to perform various tasks without prompting on a 0-3 scale.

0. The child is unable to perform the task
1. The child is able to perform the task but rarely or never does it autonomously
2. The child is able to perform the task and sometimes does it autonomously
3. The child is able to perform the task and often or always does it autonomously

And there is the options to indicate if we have no actual experience with this task but are extrapolating from related tasks.

So far, so good.

But then we get to list. Honestly it's not bad in general, but the pandemic has played merry havoc with some of its underlying assumptions.

Take "Does your child ring the doorbell or knock when visiting at a friend's house?" Presumably the question is about knowing and respecting social conventions, except that the pandemic has interfered with her ability to find little people to make friends with and prevented her from visiting those friends that she has. Is it time to guess? On what basis?

But that's not the worst of it. Another ran along the line of "Does our child lookup or respond when her name is called?" Well ... always when she's in a good mood, but never if she's really focused on what she's doing or suspects she's not going to like what you have to say (say when she knows it's after bedtime). Is that a 2? But it doesn't really capture the nature of what's going on, does it?

Every time I have to fill out a questionnaire for medical stuff I run into these ambiguities, and even if there is a practitioner around to ask no one knows how you're suppose to handle them.

I find myself assuming that it's down to trying to build nice objective measures out of the chaos that is human behavior. You can't do statistics on a bunch of anecdotes, so you have to covert all that disorder to some nice clean numbers. So, people gather a lot of experience with a phenomenon they want to study, make up some questions and stick the answers on a numeric scale, propose an analysis, test in on a sample of subjects identified by a less numeric mechanism, and check for correlation. If you get a strong enough signal you have a diagnostic mechanism. Tada!

In my fevered imagination that's the whole thing, but I'm pretty sure they try a little harder than that. I know that psychology people like to pepper their questionnaires with distractors, and I'm aware that some of the diagnostics are versioned which implies some kind of feedback and improvement loop. Alas I don't know anything about the details of the validation and improvement process.

Anyway, the problem with the bare process I outlined is that you can get a good correlation as long as there is some uniformity among the way subjects treat ambiguous questions—and anyone who's written assignment and exam questions knows that getting 80 or 90% of people to read a question a single way is easy but making it truly unambiguous is hard—but if you use the same list as a diagnostic you are immediately at the mercy of individual variation. Which offends my sense of order.

Sigh.