Method details

The full rules behind the records: what each field on a bug page means, how bugs were split into records, how every record was checked, and what is still uncertain. The short version is enough for most readers.

What counts as a bug

A stability bug is a crash, assert, hang, deadlock, data race, undefined behavior, memory or file descriptor leak, or timeout, in libmultiprocess, in Bitcoin Core's IPC code, or in Bitcoin Core code that would not exist without IPC.

Included:

Bugs only ever seen in tests (mptest, ipc_tests, interface_ipc*.py) when the defect was in production code. These count in full and are the main interest.
Bugs entirely in test code, and CI or build problems that cost time on IPC jobs. Intermittent CI failures are part of the developer cost the survey set out to measure.
Bitcoin-side bugs where the honest answer to "would this exist without IPC?" is no, and the effect is a crash, hang, deadlock, undefined behavior or leak in the node, wallet, GUI or an IPC client.
Open bugs: reported and unfixed, or fixed only in an unmerged pull request.
Regressions from refactors done to prepare for the multiprocess split. They are included but marked marginal so they can be filtered out.

Not included: compiler warnings and static-analysis findings with no runtime effect, documentation, wrong return values with no stability effect, feature work, and bugs in Rust or Python clients outside Bitcoin Core. A static-analysis fix does count when the defect it found could really execute. Borderline cases were recorded with low confidence rather than left out.

One bug per root cause

Each record is one root cause.

One fix commit is usually one bug. mp#249 fixed three races, so it has three records: getParams() after cancel, cancel callback after finish and worker-thread init.
A commit that fixes two independent defects, with different triggers and mechanisms, is two records. mp#298's one commit fixed a use-after-free and a leak that hung the event loop.
A bug first worked around and later fixed properly is one record with both commits. The server object that outlived its connection was worked around in mp#118 in 2024 and fixed in mp#160 in 2025.
A library fix shipped to Bitcoin Core by a subtree update is one record. The update PR is listed as where it shipped.
Tests added with a fix belong to that fix's record. A commit that only fixes a flaky test is its own test-code record.
An issue that reports several distinct problems is several records.

Splitting is a judgment. Merging the one pair split from a single commit gives 101 records. Also merging the three pairs that share one report (a killed wallet process seen from both sides, for example) gives 94. Splitting records that merge several sightings gives up to 104.

Classification fields

Each bug page ends with its classification. The values, in plain words:

Status

fixed: the fix is merged. workaround: a workaround is merged and no further fix is planned.
open, fix pending: a fix exists in an unmerged pull request.
open, unfixed: no fix yet.

Scope

Where the defect was, not where it was seen.

production: library or node code that ships.
test-code: the test itself was wrong.
ci-infra, build, example: CI configuration, the build, or the example program.

Symptom

What the defect can do, which may be worse than anything anyone saw.

crash: the process died from a signal, such as a segfault or abort from a library check.
assert: an assert, Assert or Assume failed and aborted the process.
hang: the process stopped making progress or would not exit, with no lock cycle identified. deadlock: two or more threads each wait for the other.
data race: two threads touch the same memory without synchronization, usually reported by TSan.
undefined behavior: code that C++ does not define, such as a use-after-free or a null argument to memcpy, usually reported by a sanitizer.
leak, fd leak, out of memory: memory or file descriptors not freed, or memory use growing until the process is killed.
timeout: a test or CI step ran out of time.
service loss: the process keeps running but a listener stops accepting new connections for good.
wrong result: the code returned or accepted something wrong. Only kept when the result affects stability.

Observed effect and where seen

Where seen lists every place the failure actually happened: a node, wallet or GUI built from merged code and run by a person (production), a build of an unmerged branch, an external client, a functional test, a unit test, mptest, a fuzz target, a CI build step, or never.
Observed effect is the worst thing actually seen: a crash, hang, deadlock or assert; a sanitizer report with no failure without the sanitizer; a test assertion only; a log line only; or nothing.
Needs a sanitizer: yes when every sighting needed a sanitizer or Antithesis to show up.
Theoretical: found by reading code and never triggered outside analysis.
Reproducible: deterministic, intermittent, only under a tool, or never reproduced.

Trigger and phase

The trigger is the event that sets the bug off.

client unclean disconnect: a client's connection dropped while it still held objects or had a call in flight: a killed process, or a Rust or Python client closing its socket without calling destroy.
server unclean disconnect: the same, seen from the client when the server goes away.
clean disconnect: the peer released its objects and closed in order.
local disconnect: this process closed its own connection while the peer was alive, for example the node dropping its clients at shutdown.
shutdown, signal: process shutdown, or a signal such as Ctrl-C.
node startup, Qt startup, test fixture: during startup of the node, the GUI, or a test setup.
concurrent calls: two calls at once, on one object or one thread.
fork and exec: process spawning.
non-C++ client: something only a client written in another language does.
unusual input: an argument or sequence the usual client never sends.
ordinary path: nothing special; the code was wrong on its normal path.
logging: triggered by log output.
other: explained on the page, for example a resource running out at the wrong moment.

The phase is when in the process's life it happens: startup, steady state, shutdown, or any time. A node dropping its clients at shutdown is trigger local disconnect, phase shutdown.

Side and component

Side says where the defective code runs: the server that implements an interface, the client that calls it, both, the event loop itself, or the code that spawns processes. It is not about who disconnected. Component names the part of the code: in the library, for example the proxy and connection code (proxy-io), the per-call context handling (type-context), the thread map, the event loop, spawning, or mpgen; on the Bitcoin side, init, the IPC glue in src/ipc, the mining interface, RPC or signals; or the upstream Cap'n Proto library, the compiler toolchain, or the test harness.

Root cause kind

assumption violated: code relied on something that stopped being true, often an in-process assumption meeting a caller in another process.
lifetime: an object used after it was destroyed, or destroyed while still in use.
ordering: two steps ran in the wrong order.
missing lock, lock order: shared state without its lock, or two locks taken in an order that can deadlock.
refcount: a reference count kept something alive, or let it die, at the wrong time.
missing cancellation: work not stopped when its caller went away or shutdown began.
unhandled exception: an exception escaped where nothing caught it.
blocking the event loop: code on the event loop thread waited for work that needs the event loop.
API misuse, resource leak, uninitialized, off by one: as named.
unknown: not yet diagnosed. other: explained on the page.

Peer

The libmultiprocess C++ client is methodical: it never abandons a request, never disconnects before calling destroy, and shuts down in order. Other clients do all of these.

Needs a foreign peer: yes when a libmultiprocess C++ peer used as designed could never trigger the bug. Being killed counts as something a C++ peer can do; a window a few instructions wide that only a program sending and then disconnecting on purpose will hit counts as foreign.
Peer kind: the libmultiprocess C++ client, Cap'n Proto C++ without the library, Python, Rust, a test harness, a spawned child process, or none.
Peer behaviour: what the peer did: a sudden disconnect, an abandoned request, no destroy before disconnecting, an out-of-order shutdown, two requests on one thread, a malformed argument, default values the C++ client always fills in, an unexpected call order, or a reconnect.

Where it lived

On master: whether the defect was ever on the libmultiprocess or Bitcoin Core master branch.
Trigger on master: whether merged code could set it off. "Possibly" means only through the hidden test-only echoipc RPC, or only if the process runs out of file descriptors or another resource at the wrong moment.
Released: binary if it shipped in a release with IPC binaries (v30.0 and later), source if only in an earlier release where IPC was an opt-in build, no if never.

Fix

Fix kind: a one-line tweak, a local fix, a new code path, a refactor, a workaround, a configuration or dependency change, a test-only change, or none yet.
Fix lines: lines added and removed by the fix commits, not counting tests or preparatory refactors. For test and CI records the test or CI change is the fix, so it is counted.
Stability impact: what the bug could affect: the node, wallet, GUI, a mining client, only tests, only the build, or nothing in practice.
Counterfactual: "partially" marks the marginal refactor regressions.
Confidence: how sure the second reader is of the classification.

Reports

Each independent report is listed with its date, reporter and kind: a CI failure, testing a pull request, real use of a deployed node or client, noticed in passing, code review, Antithesis, fuzzing, a sanitizer run, a new test, or later recognized as the same bug when another report was examined. Repeated CI failures on one thread are one report. Each report is also marked for whether it says a language model helped ("stated"), reads as if one did ("likely"), or neither. The "likely" marks are judgments of writing style.

Dates

Introduced and fixed are the UTC dates the change was merged to master. Author dates are ignored, because pull requests change a lot between writing and merging. A commit made directly on master uses its commit date.
First report is the date of the earliest report of this bug, even years before work on a fix began. The page says when work started if it differs.
An Antithesis sighting counts as a test, not as production.

How each record was checked

Candidates. Scripts listed every commit, pull request and issue in both repositories that might be a stability fix, plus the multiprocess tracking issue and the maintainer's notes. All data came from local copies; nothing was fetched while writing.
First reader. Each candidate was read with its diff and discussion and either written up as a record or set aside with a reason. Records needing a closer look were flagged by fixed rules: fixes to core teardown code, two or more threads, a disputed mechanism, or a doubtful split.
Added fields. After a first look at the data, each record gained a list of every independent report, what was actually observed, what kind of peer it takes, and a phase. Each record was read again to fill them.
Second reader. Every record was checked again against every fix commit, the pull request body and review comments, and every linked report. Each field was confirmed or corrected, and the reason written down. Questions only the maintainer could answer were sent to him; he settled twelve. The 16 records added in October 2026 were checked against the code and the reports by the agent that wrote them, and have not had a separate second reader yet.
Public text. The bug pages were written from the checked records. Where the code or logs contradict a commit message, pull request or review comment, the page follows the evidence and says so. Those cases are collected on the corrections page.

What the second pass changed, for example:

It moved first reports earlier when an older sighting was the same bug. The event loop relock at exit is now dated from an April 2024 CI failure, and the wallet scheduler deadlock from a tester's report three weeks before the CI failure.
It found the commit that introduced the request-thread deadlock and the thread-map double destroy: the 2024 thread map fix, not later changes.
It corrected which process fails: the spent-coin assert is in the node, not the GUI.
Across the set, it moved 26 introduced dates to merge dates, dropped one record, added seven, six of them from the security reviews, and changed confidence from 36 high, 29 medium and 15 low to 60, 15 and 11.

Two independent readings

The patterns on the findings page come from two interpretations of the finished set. The first was written from the first-pass records. The second was written later from the checked records, without reading the first, and ends with a comparison. They agree on the main results: failures are fail-stop, disconnects are the central trigger, shutdown bugs appear in every period, fixes cause later bugs, detection follows new tools, and the "waves" describe the order of fixes rather than successive classes. They disagreed on details where the data had changed in between: whether the client side was tested before 2026 (it was), the ranking of root causes, and how large the review-driven 2026 group is. The findings follow the current data.

Known limitations

Found, not present. The records list bugs that were found. Finding was concentrated after 2024, so the count by year measures effort as much as quality.
Reports are from GitHub. Diagnoses made on IRC or in private appear only as notes, and small finds fixed in passing by other people are probably under-counted.
"Open" mixes three things: unfixed bugs seen in the field, fixes waiting for review, and review findings with no reproduction.
Most audit reports are untested. The security-review findings that had become bug records by September 2026 were reproduced against Bitcoin Core master that month (one only under ThreadSanitizer), and so were two audit-only reports. The eleven records added from the reviews in October 2026 rest on the reviewers' own tests and analysis, which were not repeated, and the rest have not been tried. Ten records overall are theoretical.
Branch builds are not production. Only 9 production records were seen failing in a build of merged code. Another 20 were seen failing in a running node, wallet or GUI only on builds of unmerged branches, such as the wallet and GUI process branches; they are counted as branch builds, and they depend on how much testing those branches got.
Shared reports. Three pairs of records share every report, so the delay of one event is counted twice. The 1992-day wait of the 2020 wallet-process pair is one event.
Retroactive links. Some first-report dates rest on a later judgment that an older report was the same bug.
Upstream defects. A few production records are defects in Cap'n Proto or the MinGW runtime that IPC exposed. They add four to the production count and three to the release counts.
Style judgments. Whether a report was LLM-assisted is mostly judged from its style.

Acknowledgements

Every bug on this site was found, reported, diagnosed or fixed by someone, and each bug page credits the people involved, with links to their reports. Thanks to all of them, including the Stratum v2 developers and operators whose reports from real deployments are the most direct evidence here, the developers who wrote the Rust and Python clients and tests that exercised the code in new ways, and the Antithesis team. Thanks also to an independent red team for the 2026 security reviews, whose reports are published on the audit reports pages with permission.