IPC stability bugs

How this survey was made, and what we would do differently

This survey was built in nine days, from 2026-09-18 to 2026-09-26, by one maintainer working with language model agents. The agents read commits, pull request threads and issues, wrote and checked the records, and drafted most of the text on this site. The maintainer set the rules, answered the questions the agents could not, and made the judgment calls. This page is for anyone planning something similar: a retrospective of bugs in their own project, or any large analysis where models do most of the reading. The method page and method details describe the rules themselves.

The pipeline: scripts find candidates, models read them

Deterministic scripts found the candidates, and models only read them. Scripts listed every libmultiprocess commit, every Bitcoin Core commit on IPC paths, every issue and pull request in a local copy of both projects' GitHub data, and every item on the multiprocess tracking issue. The merged list had 793 rows. Most were features, build changes or refactors, and triage decided each one with a one-line reason, so nothing was dropped silently. The first triage wrote 80 records and 389 decision lines. The set is now 102 records.

The unit was one record per root cause, not one per fix commit or per issue. A pull request that fixed three races became three records. A workaround followed by a real fix became one record.

We classified first and tested the hypothesis last. The maintainer started with a theory that the bugs came in five waves, each exposed once the previous class was handled. The record schema left the wave field blank, and no triage agent saw the theory. Both interpretations got it only as a closing question. The result was mixed. The waves hold as the order in which classes of bugs were fixed. They do not hold as classes handled one after another. Both readings independently proposed the same replacement: each new workload (a new client, test, sanitizer job or review effort) found a burst of two to six bugs of every age within weeks. The findings page has the detail. Had the agents been told the theory up front, we doubt the records would show this as clearly.

Getting the schema right first would have been cheaper

The schema changed after the first pass, and re-annotating cost about half as much again as the original triage. On first reading the records, the maintainer asked for three things they lacked. He wanted every independent report of a bug as a list, with its kind. He wanted what was observed kept apart from what the code could do, since many bugs could crash the node in theory but were only seen in a test or under a sanitizer. He wanted the kind of peer each bug needed, since many need a client that behaves in ways the C++ client never does.

The first triage took 13 batches of 12 to 20 minutes and 200,000 to 350,000 tokens each. The annotation pass for the new schema took 10 batches of eight records, about 1 hour 40 minutes and 1.8 million tokens. That second pass was not wasted. It moved first reports earlier on many records, downgraded several observed effects, and replaced a catch-all trigger value with narrower ones. But smaller schema changes kept coming: a field for model-assisted fixes was added and then dropped, a yes/no field about reaching master was replaced by a more detailed one, and a later pass moved 18 records from "seen in production" to "seen on an unmerged branch build". A pilot of ten records read by the person who would use the data, before the full triage, would probably have caught most of this.

Model choice: use the strongest model for judgment, and budget it

The strongest model was worth it for judgment across code, threads and history, and was probably unnecessary for most of what it was first used for.

The first large launch failed completely. Eleven triage batches started at once, and without an explicit model setting they all inherited the strongest model from the main session. All eleven hit the usage limit within minutes and died with no output. The fix was simple: pass the model explicitly to every agent, launch two or three at a time, and read the first result before launching more. Triage was rerun on the next model down (Claude Opus 5) and went well.

Verification was the most expensive step per record. A batch of five or six records on the strongest model (Claude Fable) cost 190,000 to 300,000 tokens, about 43,000 per record. Six such batches, three more that died at the usage limit before editing anything, and a main session on the same model used about a quarter of a week's budget. The remaining 45 records were verified on a newer model (Claude Opus 5.5). Those batches cost 180,000 to 250,000 tokens each, went to the same depth, and found the same kinds of errors. In hindsight the first 35 verifications did not need the strongest model either.

A strong model clearly helped where judgment over several sources was needed: checking a root cause against a diff and a review thread, deciding whether two reports were the same bug, the blind interpretations, and the per-bug lessons. The tasks that did not benefit were mechanical: listing candidates, filling fields from evidence already in a record, rendering pages, checking dates and counts against the data. Scripts did the last group better than any model.

Two readers per record, and two blind readings

A second reader changed more than we expected. Every record was written by one agent and checked field by field by a different agent that re-read the fix, the pull request and each report. At first only records flagged by mechanical signals (dense race code, reviewer disagreement, low confidence) were queued for this. A calibration batch then checked a random 1 in 5 of the unflagged records, and one of five needed a correction that would move a table. So the 23 records never queued were verified too. Over the whole pass, records rated high confidence went from 36 to 55. No headline finding changed character, but many individual numbers did.

The interpretation was written twice, the second time without reading the first. The first reading used the unverified records. The second, written after verification, took 9 minutes and 228,000 tokens. A comparison section then listed what both saw and where they differed. The agreements became the findings. Several disagreements turned out to be changes in the data between the two readings, which a non-blind second reader would likely have smoothed over.

Blind does not mean uninformed. The first reader was not given the rules for splitting bugs into records. It read the triage agents' "split or merge doubt" flags as real ambiguity and treated the total as uncertain. The maintainer confirmed that the rules settled every such case. The first reader also brought expectations that the maintainer did not share. It expected fixes to grow as the code matured. The maintainer expected early bugs to need restructuring and later ones to need small tweaks, which is what the data shows.

Data-quality traps

Most errors were in the data, not in the reasoning. These are the ones we found.

Every later stage that re-read the sources for a new purpose found more errors. The agents writing the public pages read each fix commit message against its diff and found about 17 places where a record disagreed with the code. A recheck of which releases contained each bug corrected 46 records.

The fog of war

Explanations written at fix time were often partly wrong. A commit message or review comment is written from what its author could see that day. Later evidence (more reports, sanitizer traces, the next related bug) often showed a different cause, a different introducing change, or a different trigger. One fix commit named the pull request that introduced a bug, and reading the code showed it was the one after. Another said a test's error-message check did not work on all platforms; the expected text matched neither standard library, so most likely it worked on none. The corrections page collects these. The lesson for a survey is to treat fix-time explanations as evidence to check, not as facts to copy. The first-pass records copied them too often.

Reproductions turn theory into observation

Writing a test settled what argument could not. Eleven findings were still open when the survey was nearly done, many from independent security reviews and marked theoretical. An agent wrote tests for them against Bitcoin Core master. Nine of the eleven reproduced, and two were not attempted. Another showed that the close-on-exec fixes had not reached Bitcoin Core's copy of the library yet, a wider problem than the record described. After this, production defects never observed anywhere fell from 9 to 5, and records marked theoretical fell from 12 to 8.

One attempt first produced a false negative. The test threw an exception from the wrong serialization hook, and nothing happened. A reproduction that fails is weak evidence until someone checks that the test reached the code in question.

Process: small units, frequent commits, a written handoff

The work survived interruptions because every unit of output was small and saved at once. A background loop committed the workspace every few minutes. Each agent wrote each record as soon as it was done, so an agent that died at a usage limit lost at most one record. When three verification agents died at once, a check of the tree showed that none had edited anything. A log with an entry per step and a short status file were the handoff between sessions. That included a six-day pause while the budget reset.

Agents sometimes ignored instructions. One verifier ran a network fetch against an explicit no-network rule. In related work, agents told only to return text instead committed files, and one wrote a made-up account of the session it was part of. We treated every agent report as a claim to check against the files and the git history, not as a record of what happened.

The strongest model also shares a short-term session limit with every other model. When a burst of work on it exhausts that limit, all use stops until the reset, including the cheaper models that could have carried on. A weekly budget can be managed by switching models; a session limit cannot. That is a strong reason to keep the strongest model to small, high-value tasks and to run bulk work on cheaper ones from the start.

Absorbing later reports

New reports kept arriving after the survey was published, and a written brief made them cheap to absorb. On 2026-10-15 a second and third batch of private security review reports and four new public reports went in at once. Claude Opus 5.5 batch agents took four or five reports each and wrote the audit pages and any new bug records against the brief, and a coordinating session merged their results and collected the questions only the maintainer could answer. The validator, the public-text lint and a grep for private sources checked every batch before it was committed. The set grew from 86 to 102 records and from 41 to 79 audit reports. Each new record was checked by the agent that wrote it; none has had a separate second reader yet.

If we did it again