When the Measurement Is the Bug

For about a week I was trying to answer one narrow question: could a model we run ourselves replace a paid API inside a computer-vision pipeline I work on? The pipeline asks a few dozen yes-or-no questions about each photograph and expects a single JSON object back. It was a comparison run beside the pipeline rather than a change to it, and I assumed the interesting part would be the models.

The models were fine. Better than I expected in a couple of places. What took the week was that three separate times I produced a number that looked perfectly reasonable and wasn't, and nothing in the output said so. No exception, no empty column. Just a percentage sitting in a table, looking exactly like a measurement.

Every one of them came from the evaluation tooling rather than the model itself.

Three numbers that looked healthy

The first was a sampling accident. I ran the opening few hundred images of a manifest that happened to be ordered in blocks, so my sample came out about 98% indoor while the pool it was drawn from sat closer to 63%. Indoor photographs get roughly twice as many questions asked of them, so every rate I computed off that sample described a mix of work that wasn't the one I was claiming to measure. I noticed because one figure felt off against a rough expectation in my head. I got lucky there. Nothing in the tooling would have caught it.

The second was worse, because it looked like a model behaving badly rather than a tool measuring badly. Reasoning models think before they answer, and the thinking comes out of the same token budget as the answer. When one ran out mid-thought, the server returned finish_reason: "length" and an empty message. My parser read that empty message as the model declining to say yes, and scored it as a negative answer.

So a call that never got to speak was recorded as one with an opinion. In the object-detection task it was sharper still: a truncated call returned an empty list, which reads as I looked and found nothing, and since most things are genuinely absent from most photographs, those silent failures lined up with the reference and counted as agreement. Truncation cost the metric nothing. It quietly improved it.

The third I had been staring at for most of a day without seeing. One model was around four times slower than its sibling in the same setup, and about 65% of the free-text captions I asked it for came back empty. I went through the server logs first. My first instinct was still to blame the model, and I had already written that down as a property of it. It wasn't.

The short tag I had pulled, qwen3-vl:8b, resolved to the reasoning checkpoint rather than the instruction-tuned one, and the packaging it arrived in has no switch to turn the reasoning off. I had spent a day measuring a model I never meant to download. I still don't know whether that tag always pointed there.

The pattern underneath

Looking back, all three failures happened where two parts of the system met. Two things that had to agree were written down separately, and nothing kept them in step.

A config names a model. A tag turns that name into actual weights, and tags move. Nothing checks that the name and the weights still match.

The sampling case had no join at all. The sample and the population it stood for were connected by the order of rows in a file, and nobody had written that down anywhere.

The parser one is worse, because it sits inside the code rather than around it. A parser that can return an answer but has no way to return nothing arrived will invent one every time a call fails, and it invents the cheapest answer available: the absent tag, the empty list. Failures shaped like the majority class are invisible to any metric built on agreement with a reference.

A number becomes useful only when you can explain the conditions that produced it.

So the harness now refuses to produce a number it can't defend. Each run file gets a sidecar recording the digest of the weights it ran against, the rendered prompt and schema, the token budget, and a hash of the bytes the task actually read. Change any of it and the next run stops and names what changed, instead of appending to a file that now describes two experiments.

Truncated and failed calls contribute nothing anywhere, and their share is reported per task. Every rate carries its sample size in the next column.

Agreement is not accuracy

There is one more trap, and it survives every fix above.

When a candidate is evaluated against a system already running in production, there is no ground truth in the room. There are the incumbent's answers. So what is actually being measured is agreement, and in a summary or in your own head, agreement slides very easily into accuracy.

They are not the same whenever the incumbent is wrong. Adjudicating a sample of the disagreements by hand, one photograph at a time, was the only step that could separate the candidate is worse from the incumbent was never that good, and it changed the shape of the answer. The two systems were failing in opposite directions at roughly comparable rates, one over-tagging and the other under-tagging. What the headline metric called a quality gap was two different failure habits.

Without that pass, the honest sentence is that the candidate disagrees with production some of the time and nobody can say who is right.

What I learned

The recommendation itself took an afternoon to write. The week went into being able to defend it.

Running the model wasn't what took the week. Most of the time went into figuring out whether I could trust the numbers I was producing. Nothing crashes when you get the measuring wrong. You just get a clean, well-formatted percentage describing a slightly different experiment than the one you meant to run.

The habit I came away with is to suspect the measurement before the subject whenever a result is surprising. Two of my three bad numbers presented themselves as findings about a model, and both turned out to be findings about my own tooling. The one time I wrote a wrong explanation into a report before checking it, it was because the story about the model was more interesting than the story about my parser.

I still don't trust an evaluation until I can say how every number in it was produced.