← All posts
battlefield

The fix that was accepted and never installed

· 5 min read · by aft3r_d0t

Duotone ink illustration of a black ink blot on a tilted sheet of paper.

Twice in three days, our night runner wrote a fresh, empty report over a full one. The second time, the fix for the first time had already been built, tested and accepted. It just hadn’t been installed.

How the report gets its name

Most nights our runner works through a queue while I sleep: one item per tick, each one built, then checked by someone who didn’t build it. Every tick lands in a report, and in the morning the report is waiting for me.

The report is named after the run’s deadline, which is 08:00 the next morning. That sounds harmless. It means that any run started before the same 08:00 gets the same file name. And when the runner starts, it writes a fresh report before it does anything else, before it even looks at the queue.

Two small decisions, each fine on its own. Together they mean: start the runner twice before the same morning, and the second start wipes the first one’s record.

The first time: 05:41

On 11 October a morning start at 05:41 did exactly that. The night’s record, 34 ticks from 23:25 to 04:17, was replaced by a new file. We got it back from git and saved it as part 1.

The fix was small: when a report already exists, write a new file with the start time in its name instead of overwriting. It went into a bigger change that also taught the runner to stop at a given time. That change was built in a scratch copy, tested and accepted.

Then it sat there. Installing it into the live runner was a daytime step, and it needed my word. The live file’s last change was still from 5 October.

The second time: 00:17

On Monday 12 October we ran the queue in the evening, by hand. The runner normally refuses to start before 18:00, so this start was forced, and a separate timer was set to stop it at 21:30. It started at 17:13 and ended at 21:36: 23 ticks, 104 lines of report. Because the deadline was 08:00 the next day, the report was filed under 13 October.

One of us pointed that out at 17:14, as a naming quirk. Looking back, it was the whole bug, described seven hours early.

At 00:17 someone started the night run. The queue was empty, so the run ended in the same minute. But the start had already written its fresh report over the evening’s 104 lines. At 00:24 the real night started and overwrote that one too.

Why we still have it

The vault commits itself every night at 23:30. That night the commit landed at 23:31:10, 46 minutes before the empty start, and it held the evening’s report. We restored it from that commit, and a comparison with the committed file came back with no differences at all. The commit before it, from 17:30, held only the first 17 minutes of the run. If the empty start had come before 23:30, that’s all we’d have.

So the second rescue was luck with a timestamp. A backup you didn’t plan for is still a backup, but I wouldn’t build a process on it.

The second fix

The first item that night was a second fix for the same bug, written with both incidents in front of it. Three rules.

Never touch an existing report. Claim the next free name, part 2, part 3 and so on, in one atomic step, so two starts can’t both decide the same name is free. And when the queue is empty, write no report at all: a run that does nothing has nothing to say.

The atomic step is the interesting part. “Check whether the file exists, then create it” has a gap between the check and the create. Asking the operating system to create the file only if it doesn’t exist yet closes that gap. In Python the idea fits in a few lines. This is a sketch of the idea, not the runner’s code:

import os

def claim(base):
    n = 1
    while True:
        name = base + ".md" if n == 1 else f"{base}-part{n}.md"
        try:
            fd = os.open(name, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
        except FileExistsError:
            n += 1
            continue
        os.close(fd)
        return name

print(claim("report"))   # report.md
print(claim("report"))   # report-part2.md
print(claim("report"))   # report-part3.md

It was accepted on the first pass, with 126 tests green against a baseline of 117. And again, installing it was left for daytime and my word.

Where the first one went

Now there were two accepted fixes for one bug, both changing the same start code, never tested together. Which one goes in first? Do they even fit together? Nobody had measured it.

On 14 October we went to install them and found nothing. Both had been built in scratch space that is wiped after 24 hours. The accepted work was gone. Only the notes describing it were left: very good notes, with test counts and verdicts, describing code that no longer existed anywhere.

So it was rebuilt once more, this time as a single patch kept where the cleanup can’t reach, with both behaviours and their tests together. That also answered the question nobody had measured: one patch, one test run, both behaviours in it. It was installed the same day while the runner was idle: 117 tests passing before, 165 after, and the installed files matched the patch’s output byte for byte. The live runner now claims its report name with an exclusive create.

What “done” means

Acceptance answers “does this work?”. It doesn’t answer “is this running?”. We had treated the first answer as if it settled the second.

Both answers matter, and they come from different places. The first comes from a test suite on a copy. The second only comes from looking at the live thing: what’s installed, when it last changed, and how many tests pass there. Between the two incidents, the second answer was sitting in plain sight, as a file date of 5 October. Nobody asked it.

An accepted fix that nobody installs is just a well-documented bug, waiting for its next chance.

— Max