Findings

What the 102 records say about stability bugs in libmultiprocess and Bitcoin Core's IPC code. Every number comes from the verified records; the explorer lets you slice them, and the method page says how they were built. Counts depend on how bugs are split into records: counted differently, the total is anywhere from 94 to 104.

In short

• 102 records: 82 defects in production code, 13 bugs in test code, 7 CI, build or example problems. 69 are fixed or worked around, 33 are open.
• About half are about tearing things down, and nearly all fail by stopping: crashes, asserts, hangs and deadlocks rather than wrong answers.
• Most defects are old. They were found after 2024 because new clients, tests and tools started reaching code that had not changed since 2019.
• Eight defects were reported from real deployments, all in the mining interface.
• Detection moved from people running the software, to CI and sanitizers, to code review. Fixes got smaller over time, and eleven fixes caused a later bug.

What the bugs are

The failures are fail-stop. Of 102 records, 87 are crashes, asserts, hangs, deadlocks, data races or undefined behavior, and three more are IPC listeners that stop accepting connections for good. Only one describes the node silently doing the wrong thing: submitSolution accepting a block whose coinbase commitment was never rechecked.

About half of them involve tearing something down. Of the 82 production defects, 42 are triggered by a disconnect, a shutdown or a signal. The most common trigger, in 17 production defects, is a client going away uncleanly: a process killed in a test, a Rust or Python client closing its socket without calling destroy, or an operator pressing Ctrl-C. All 17 are in server code. The share of disconnect bugs is falling, from 11 of 13 production records first reported in 2024 to 17 of 41 in 2026, but the number is not.

The root causes come down to a few sentences:

• Something was used after what it depended on was gone. A ProxyServer outlived its Connection (crash on disconnect during a call); a thread map keyed by a Connection pointer found a new connection at a reused address (stale pointer).
• Two teardown paths ran in the wrong order. Cleanups ran first-in-first-out when the dependency needed last-in-first-out (wallet hang); the event loop exited before its last wake-up write (broken pipe at shutdown).
• Code written for callers in the same process was called from another process. This one keeps coming back: for the GUI in 2021 and 2022, and for the mining and node interfaces from 2025 on. Examples are a client connecting before the chainstate is loaded (startup assert) and a client sending default option values the in-process code never sees (createNewBlock options assert).

By record count the root causes are: an assumption violated 27, a lifetime error 16, an unhandled exception 13, an ordering error 13, a missing lock 8, and smaller groups. By side, 48 production defects are in server code, 10 in client code, 11 in both, 7 in the event loop and 5 in process spawning.

Old code, new exposure

The count by year measures how hard the code was being exercised, not how old it was. Of the 73 production defects whose introducing commit is known, 37 date from the library's first commits in June to August 2019. Eighteen of the 41 production defects found in 2026 had been in the code for seven years. From the merge that introduced a bug to the merge that fixed it, the median was 996 days, and 31 of 63 lasted more than three years.

What changed was the workload. Reading each record's account of what exposed it gives a sequence, and each step was followed within weeks by a burst of two to six records:

At the start of 2024, 41 production defects already in the code had not been reported. At the start of 2026, 28 had not.

How they were found

Detection shifted twice: from people to tools, then to review.

Through 2025 most production defects were found by someone running the software: the maintainer, reviewers testing branches, a Rust client developer, Stratum v2 operators after v30.0. In 2026 only two were, both by developers testing the unmerged pull request that serves the Chain interface over IPC.

Sanitizer finds came in two bursts tied to new CI jobs: MSan and UBSan on the IPC unit tests in late 2024 (a downcast in a constructor), and the library's own TSan jobs in 2025. Antithesis is the first finder of three records and appears in six. What it added was a reproducible trace of a rare interleaving, and its finds were real: the getParams() use-after-free also crashed a Rust client three weeks later.

Review found 20 of the 30 production defects first reported since July 2026. Nineteen of the 24 found by review in 2026 have a report that says, or reads as if, an LLM helped; 18 came from the private security reviews. Across all report entries, none in 2024 and 1 of 53 in 2025 are marked LLM-assisted, against 42 of 88 in 2026. Most of these marks are judgments of writing style, so the trend is firmer than any single one.

The 2026 reviews also brought a kind of testing the history barely had: fault injection, forcing an allocation, a thread start or a system call to fail at a chosen point to see what the error path does. Thirteen of the audit reports rely on it. Six describe hardening only, because nothing but the injected failure reaches the code, and four became low-confidence records whose trigger is a failure that rarely happens on its own. Three are real stability bugs: a listener that stops for good after one failed accept(), a terminate when a signal interrupts the wait for a spawned process, and the client hang when building a request throws, which an earlier report had already found.

How fast they were fixed

Once a bug was recognized, work started fast: the median bug waited 1 day for a fix PR and 8 more days for it to merge. Tool-found bugs closed fastest.

The long tail, 17 bugs over 90 days, is mostly branch effects. Two wallet-process bugs reported in 2020 waited 1992 days for the branch that fixed them to merge (Ctrl-C reaches the wallet process), and the Windows socketpair hang waited a year for the Windows port.

Fixes got smaller as the design settled. The early fixes restructured: giving every proxy its own reference to the event loop, evicting clients at shutdown. Later ones mostly moved a lock or reordered two steps inside those mechanisms, such as the one-call prefetch that closed an Antithesis race. Among production fixes, new code paths and refactors were 5 of 21 in 2025 and 1 of 20 in 2026, and the median fix fell from 24 lines to 10. The median merged production fix is about 17 lines; 25 of the 69 merged fixes are under 10. Two recent fixes are the exceptions and the largest in the set: request cancellation (267 lines, February 2026) and disconnect-then-drain at shutdown (288 lines, pending). Small fixes do not mean small bugs: a one-line lock move can close a window a few instructions wide in code that ships in every release.

Eleven production defects were introduced by the fix for another record. The cancellation path produced three found within three weeks of merging (lock missing on cancel, getParams() after cancel, cancel callback after finish). The 2024 thread map fix led to a deadlock eleven months later and a thread-exit race a year after that.

What it takes to trigger them

The libmultiprocess C++ client is methodical. It never abandons a request, never disconnects before calling destroy, and shuts down in order. A C++ peer used as designed can trigger 70 of the 82 production defects, cannot trigger 11, and for one it is unknown. For many of the 70 that means the peer being killed in the middle of a long call.

The eleven that need a foreign or misbehaving peer are of two kinds:

• Timing windows a few instructions wide around request dispatch and thread start, which only a peer that sends and then disconnects programmatically will hit (worker-thread init).
• Things the C++ client never sends: default option values, two requests on one thread, a malformed argument (empty coinbase), or a call after destroy (null implementation).

Rust and Python clients appear in the peer list of 23 production records.

Seen, or only possible

A record's symptom says what the code can do, which is often worse than what anyone saw. Of the 82 production defects, 9 were seen failing in production: in a node or GUI built from merged code and run by a person. Another 20 were seen failing only in a node, wallet or GUI built from an unmerged branch, such as the multiprocess wallet and GUI branches, mining branches or the Windows port. 47 were seen only in tests or under tools, and 6 never. Ten of the 47 were never more than a sanitizer report. Fifteen records needed a sanitizer or Antithesis to be seen at all, and ten are theoretical: found by reading code and never triggered outside analysis.

In September 2026 the open findings from the security reviews were tested against Bitcoin Core master. All seven bug records tried were reproduced, four of which had been theoretical: a client hang, a crash after destroy, a use-after-free during destroy and a leaked descriptor. The same tests showed that Bitcoin Core master still has the older, wider form of a socket leak fixed in libmultiprocess, until its subtree is updated. Each bug page has a Reproduction section.

Where they lived

• On master: 78 of 82 production defects were on a master branch at some point.
• Reachable on master: 44 had a trigger in merged code, 13 possibly (only through the hidden test-only echoipc RPC, or only if the process runs out of file descriptors or another resource at the wrong moment), 24 none, and one is unknown. Most of the 24 are Bitcoin-side code that only the unmerged wallet and GUI branches reach.
• In releases: 43 shipped in releases with IPC binaries (v30.0 and later), 23 only in earlier source releases where IPC was an opt-in build, and 16 never.

Reports from the field

Eight records have reports from real deployments. All eight are in the mining interface, and all were hit by Stratum v2 software. None is in the wallet or GUI. Seven were in released binaries, and two are still open.

• A crash when a killed client's waitNext call completes
• A node that stopped answering IPC after 25 minutes
• An assert on two calls to one thread
• A hang on Ctrl-C while a client waits
• An assert when a client connects during startup
• An assert on Ctrl-C with the template provider attached (open)
• Memory growth on a small server (open)
• The getParams() crash on a master build

The most-reported bugs are the costly ones: the event loop relocking after exit was seen five times on three platforms over ten months. Fifty-seven bugs were reported once, 30 twice, 15 three or more times.

Test, CI and build problems

Twenty records are not production defects. They are the cost of the tools that found the others, and they arrive right after new tests: five from the library's sanitizer CI work in 2025, four from the Python functional tests, others from new spawn, connect and concurrency tests. One test race, a lost wake-up in the simultaneous-calls test, hung CI intermittently for six months. Thirty-nine records have at least one CI-failure report. The fix PRs drew 527 force-pushes and 2061 comments from people.

The waves idea

The survey started from a guess that the bugs came in five waves, each exposed once the previous class was handled: missing disconnect handling; unclean disconnects from aborting peers and non-C++ clients; racy disconnects needing new code paths; small leftover fixes, with a server-side sub-wave; and rare or theoretical races found by assisted review and Antithesis.

Two independent readings of the data reach the same verdict. What holds:

• The waves describe the order in which the classes were fixed.
• The foreign-client part of wave 2: the Rust client in December 2024, then Python and fuzzing in September 2025.
• Wave 3 did need new code paths.
• Wave 4 on fix size, and the server-side sub-wave on a small base: three records from one contributor's tests in July 2026.
• Wave 5 is the clearest: of 32 records first reported since July 2026, 20 are review finds, 25 have an LLM-assisted report, and 7 were theoretical when reported (3 still are; tests reproduced the other 4 in September 2026).

What does not hold:

• The classes overlap from the start. Clean disconnects did not come first: the oldest records in the family, from 2020, are unclean.
• Foreign-client findings kept arriving after the racy-disconnect fixes began.
• The racy-disconnect work was not one step but fourteen months of PRs.
• The "small leftover fixes" period contains the two largest fixes in the set.
• Antithesis finds were confirmed by crashes or CI failures, not theoretical.

The picture that replaces the waves is the exposure sequence above: each new workload reached code that had been there since 2019.

What is still open

How sure these numbers are

Confidence is high for 68 records, medium for 19 and low for 15. The most cautious view, leaving out marginal and low-confidence records, has 83 records and 68 production defects. First-report dates can come years before the fix work started. Three pairs of records share one report, which double-counts the delay of one event. The security-review findings that became bug records by September 2026 have all been reproduced (one only under ThreadSanitizer). The eleven added in October 2026 rest on the reviewers' own tests and analysis, which the survey has not repeated, and most of the other audit reports have not been tested, and the LLM-assistance marks are mostly judgments of style. Method details lists the remaining doubts.