IPC stability bugs

Patterns

A pattern here is the mechanism behind a bug, stated in words a developer who has never seen libmultiprocess would recognise: work running on the wrong thread, an object outliving what it points to, a wait that nothing wakes. Each of the 102 bug records has a lesson (what would have prevented it, what would have found it sooner, what made it hard to debug), and each lesson names a fine-grained pattern, 60 in all. This page groups those patterns into eleven mechanisms, so that bugs whose symptoms look unrelated sit next to each other by cause.

How to read the page: the summary table lists the groups with their counts. Each group section gives the mechanism in plain words, why it kept happening in this code, the rule the fixes imply, and the bugs that belong to it. Below the groups are the counts across all lessons: what would have prevented them, the earliest check that would have caught them, and how hard they were to diagnose and to fix.

Every bug page has a Lessons box that links back here, to the bug's group. The Rust page asks, group by group, whether writing this code in safe Rust would have prevented these bugs.

Would Rust have prevented these? Over all 102 records: 29 yes, 28 partly, 38 no, and 7 not applicable. The Rust page goes through it group by group.

For what would make the IPC code more reliable today, see What would help now.

The 11 patterns

Production defects: the group's bugs in production code, and their share of all production defects. Rust: how many of the group's bugs safe Rust would have prevented, partly prevented, or not prevented.

PatternBugsThe rule we learnedProduction defectsRust: yes / partly / no
Work or cleanup on the wrong thread8State owned by one thread is read, written and destroyed only on that thread.7 (9%)5 / 2 / 1
An object outliving what it points to12Whatever an object touches in its destructor or its callbacks it must own or keep alive.11 (13%)9 / 2 / 1
Built or torn down in the wrong order5Keep the order the language would have used.3 (4%)4 / 0 / 1
Serving before ready, freeing before done8Decide which lifecycle states each entry point may be called in.8 (10%)2 / 3 / 3
Locks, waits and wakeups11Every reader and writer of a guarded field takes the lock.10 (12%)4 / 2 / 5
A contract changed and a caller was left behind6When you widen what a function can return, narrow what it guarantees, or replace what it uses, visit every caller and every assertion on the old contract in the same change.4 (5%)2 / 2 / 2
An in-process interface trusting what now comes from another process14On the day an interface goes over the wire, re-read every assert reachable from its arguments, its configuration and its call order and turn each into validation or an error.14 (17%)2 / 6 / 6
Error paths that skip cleanup or completion16A call across a process boundary can fail at any moment.16 (20%)0 / 10 / 6
Process spawning, file descriptors and signals6Do all the work before fork() and nothing but system calls after it, or use posix_spawn.5 (6%)1 / 1 / 4
Test timing, and tests that cannot fail6Wait for the observable event, never for a duration.0 (0%)0 / 0 / 6
CI, toolchain and platform10Build against the floor of what you claim to support as well as the ceiling, one toolchain per link.4 (5%)0 / 0 / 3 (7 n/a)

Work or cleanup on the wrong thread

8 bugs · 7 production defects (9% of all) · Rust: 5 yes, 2 partly, 1 no

In this code one thread, the event loop, owns the connections, the proxy objects and the per-connection maps of worker threads, and Cap'n Proto's own objects carry reference counts that are not atomic. C++ destructors and cleanup hooks do not know that. They run on whichever thread drops the last reference, and a precondition check runs on whichever thread starts a call. For six years that thread was the loop thread by accident: connections outlived every proxy, and the only peers were child processes the node stopped in order, so nothing ever destroyed a proxy or checked a connection from a worker thread while the loop was tearing the connection down.

Peers that disconnect mid-call changed that. A worker thread erased a map entry through an iterator the loop thread had already invalidated; a client destructor removed a cleanup callback off the loop while ~Connection walked the same list; a shutdown destructor on the loop thread waited for a scheduler thread that needed the loop; a getResults() call on a worker read connection state the loop was rewriting. The results were deadlocks, corrupted maps and reference counts, and ThreadSanitizer reports that appeared on the first sanitized run of the first disconnect test.

The rule we learned. State owned by one thread is read, written and destroyed only on that thread. Hop to the loop first, and assert the affinity in the callee (assert(loop->isLoopThread())), so the ordering that holds by construction today is enforced when lifetimes change. The library adopted this rule for its thread maps in 2025; the bugs before that date are the ones the assert would have caught on the day they were written.

Representative bugs:

The bugs, by fine-grained pattern

Cleanup runs on a thread that does not own the state

A single-thread object used from another thread

A check moved away from the use it guards

A precondition checked on the wrong thread

Prevented by
a thread-affinity assertion (7), a design rule (6), the type system (1), a review checklist item (1), a written spec or documentation (1)
Earliest detector
a disconnect test (3), ThreadSanitizer (3), a test that kills a process (1), UBSan (1)

An object outliving what it points to

12 bugs · 11 production defects (13% of all) · Rust: 9 yes, 2 partly, 1 no

A server object kept a raw pointer to the connection that created it. A cancellation callback captured locals of a frame and was stored in an object that cleared it only after the frame returned. A per-thread map was keyed by a Connection* with nothing removing the entry when the connection died. A disconnect handler that deletes the connection was armed before the client that owns the connection existed. EventLoop::post relocked a mutex inside an object the loop thread was free to destroy the moment loop() returned. The common shape is a longer-lived holder with a reference to a shorter-lived object, safe only under an unwritten assumption about who goes first.

Remote ownership is what broke the assumptions. When a client can drop a capability, abandon a request, reconnect, or disconnect without calling destroy, the destructor of the server-side object runs at a moment the process did not choose, and the comment saying "the connection is always destroyed last" stops being true. Every C++ client in the tree behaved politely, so the first Rust client, the first client someone could interrupt with Ctrl-C, and the first disconnect tests found bugs five to seven years old: use-after-free, double delete, and the m_loop assert of an EventLoopRef whose loop was gone.

The rule we learned. Whatever an object touches in its destructor or its callbacks, it must own, keep alive, or check is alive before use. A comment saying "X is always destroyed before Y" is a bug report; the fix is ownership (the EventLoopRef each proxy now holds, a weak token in a queued handler) or an assert that fires when the order breaks, not a longer comment. Fixes that extend a lifetime instead need checking for the opposite failure: a loop that never exits because something still holds a reference.

Representative bugs:

The bugs, by fine-grained pattern

A child object outlives its parent

A callback outlives what it captured

A cleanup hook armed before its owner exists

A stored reference to a temporary

A map keyed by an address that gets reused

A reference count that drops to zero between users

An object touched after allowing its destruction

Prevented by
a clear ownership model (9), a design rule (6), a review checklist item (4), a different API shape (2), a written spec or documentation (2)
Earliest detector
a disconnect test (5), a unit test (2), ThreadSanitizer (1), a test with a non-C++ client (1), static analysis (1), shutdown test (1), startup test (1)

Built or torn down in the wrong order

5 bugs · 3 production defects (4% of all) · Rust: 4 yes, 1 no

C++ constructs members in declaration order and destroys locals and members in reverse. Cap'n Proto destroys the objects of a broken connection newest first for the same reason: a later object may depend on an earlier one. Code that takes that order into its own hands has to get it right by hand. A test fixture declared its std::thread member first, so the event loop thread wrote to members the main thread had not yet initialized, and a std::promise was declared after the fixture whose destructor joins the thread that reads it. Connection::addAsyncCleanup inserted at the front of its list instead of the back, so deferred destructors ran oldest first and a wallet loader blocked the cleanup thread waiting on notifications queued behind it. The RpcSystem member whose disposal calls back into the connection's cleanup lists was declared before them, so implicit destruction ran those callbacks into lists already gone. And the base proxy class reached its derived class from its own constructor and destructor, where the derived object is not alive.

None of these needed a remote peer. They needed a kill test or a sanitizer on code that had run for years; a write into a destroyed member of a still-live object is invisible to everything but MSan, which is why one sat through every ipc_tests run for five years.

The rule we learned. Keep the order the language would have used. A thread-owning member is the last line of the class and the first thing destroyed; deferred teardown runs newest first; a hook that a base class calls from its own constructor or destructor takes the base object, never the derived one; and a member whose destructor can call back into its owner is destroyed explicitly at the top of the owner's destructor, while everything it may touch is still alive.

Representative bugs:

The bugs, by fine-grained pattern

A thread started before the members it uses

Teardown in the wrong order

A base class reaching a derived object that is not built yet or already gone

Prevented by
a design rule (4), a different API shape (1), a review checklist item (1)
Earliest detector
other (2), a test that kills a process (1), ThreadSanitizer (1), UBSan (1)

Serving before ready, freeing before done

8 bugs · 8 production defects (10% of all) · Rust: 2 yes, 3 partly, 3 no

At the two ends of a process's life, work and state overlap. At startup, -ipcbind opened the node's listening socket early in initialization, before the chainstate was loaded or the first tip connected, while the Mining methods were written for in-process callers that only exist after both. A client that connects the instant the socket appears, as production supervisors that restart node and client together do, reached an Assert on a null chainstate manager or a null tip. Inside the library, a worker thread published a handle to itself before taking the lock its destructor needs, so it could be torn down before it was ready.

At shutdown the overlap runs the other way. The node stopped waiting for clients to disconnect, which fixed a hang, and then freed the chainstate while a method body already handed to a worker thread was still running on it. A peer's destroy request reset a server object's implementation while another thread was inside one of its methods. The parameters of a cancelled request were read after cancellation had freed them. Before any of that, shutdown waited on the event loop's reference count, which strangers could hold, so a client that sat connected hung the node's exit. Cancelling and joining are different operations, and each of these bugs treated "told to stop" as "stopped".

The rule we learned. Decide which lifecycle states each entry point may be called in, and either open the listener last or make every method tolerate the state the process is in when the socket opens. Going down, stopping new work is not the same as waiting for work already running: wait for it, or refuse the release, before freeing what it uses; make the stop API return only when that wait is done; and wait only on things the process controls.

Representative bugs:

The bugs, by fine-grained pattern

Serving calls before startup finishes

Cancelling work without waiting for it to stop

Freeing state while work still uses it

Reading a request after it was cancelled

Shutdown waits on a peer it does not control

Prevented by
a design rule (7), a review checklist item (5), a different API shape (3), a clear ownership model (1)
Earliest detector
a disconnect test (3), a test with a non-C++ client (3), a stress test (1), startup test (1)

Locks, waits and wakeups

11 bugs · 10 production defects (12% of all) · Rust: 4 yes, 2 partly, 5 no

A lock protects a field only if every accessor takes it, and a condition variable wakes a waiter only if the state change and the notify happen under the waiter's mutex. The records hold both halves of that sentence broken. Three fields in the disconnect path were written under a lock on the event loop thread and read bare on a worker; a flag the lock guarded was written through an unnamed Lock{} temporary that unlocked on the same line; a log statement read a pointer another thread was nulling; a thread map had two destroyers, one of which never took the lock; and two threads making a process's first IPC call could both start an event loop, because the check for an existing loop took no lock. On the wakeup side, a test issued notify_all() without the mutex and lost the wakeup on slow machines; a shutdown refactor left the notify for waitTipChanged on one of two shutdown paths, so Ctrl-C set the interrupt flag and woke nobody; a failed startup woke the mining waits without setting the shutdown flag they check, so they went back to sleep; one interrupt() released only one of several waits, because the first waiter to wake cleared the flag; the event loop's "done" byte was sent with async work still queued, so the loop exited and the next write hit a closed pipe; and the async cleanup thread exited on a snapshot of "done" that one more connection could reverse.

The symptoms were data races under ThreadSanitizer, and hangs whose backtrace shows one thread parked in a wait and every other thread idle. Clang's -Wthread-safety annotations turn the unlocked accesses into compile errors; the library added them after the first round of these bugs, and the next round was the fields that pass had missed.

The rule we learned. Every reader and writer of a guarded field takes the lock, and the compiler is told so with GUARDED_BY; the waited-on state changes and the notify happens under the waiter's mutex, and the condition variable is annotated too; a "done" signal is sent for the whole condition, once; and a worker exits on an event that cannot be undone.

Representative bugs:

The bugs, by fine-grained pattern

A field locked by some accessors and not others

Consumed broadcast flag

An interrupt that wakes nobody

A notify sent without the waiter's lock

A done signal sent too early

A done state that can become false again

Unsynchronized lazy init

Wakeup without interrupt

Prevented by
a design rule (9), lock annotations (6), a different API shape (1), the type system (1), a review checklist item (1)
Earliest detector
a stress test (3), ThreadSanitizer (3), static analysis (2), a disconnect test (1), a test that kills a process (1), startup test (1)

A contract changed and a caller was left behind

6 bugs · 4 production defects (5% of all) · Rust: 2 yes, 2 partly, 2 no

A function's contract changed and a caller written against the old one kept running. createNewBlock began returning null when shutdown interrupts its wait, for the benefit of IPC clients, and three RPC callers still carried a CHECK_NONFATAL written when null was impossible. SetupServerArgs stopped setting node.args as a side effect, and every entry point except bitcoin-qt was updated. submitSolution stopped copying the template's block and wrote into it in place, in a commit titled a refactor, which turned an object shared with every client thread into one that races. A locale refactor swapped a printf for std::cout chains and lost the per-call locking that had made the log callback safe to run from any thread. Two of the drifts came from two correct PRs that merged without a textual conflict, so neither author saw the combination.

Each change was reviewed as what it said it was, a refactor or a mining fix, rather than as a change to what other code could assume. That is the mechanism: the assumptions live in the callers, and a diff shows only the callee. The assertions guarding the old contract in callers are exactly the list of places that needed a visit, and a return type that cannot be ignored (std::optional, std::expected, a distinct "interrupted" result) would have made the missed callers compile errors instead of shutdown-time aborts.

The rule we learned. When you widen what a function can return, narrow what it guarantees, or replace what it uses with a look-alike, visit every caller and every assertion on the old contract in the same change; treat "modify a copy" becoming "modify in place" as a change of contract rather than a cleanup; and test the merged tree, not the branches.

Representative bugs:

The bugs, by fine-grained pattern

A function's contract changed under its callers

A refactor that dropped a guarantee

A shortcut that only held for the original caller

Prevented by
a review checklist item (4), a different API shape (2), the type system (2), a written spec or documentation (1), other (1)
Earliest detector
a unit test (3), ThreadSanitizer (2), code review (1)

An in-process interface trusting what now comes from another process

14 bugs · 14 production defects (17% of all) · Rust: 2 yes, 6 partly, 6 no

Bitcoin Core's Node, Mining and wallet interfaces were written for callers in the same process and later served over Cap'n Proto to processes with their own arguments, their own language and their own call order. Nothing in the interfaces changed, so their in-process assumptions became things a peer could violate. Range checks on BlockCreateOptions and a negative-timeout check on waitTipChanged stayed Asserts, so a zeroed struct or a stray double aborted the node. Node::walletLoader() asserted that the node had a wallet loader, which became a fact about a differently configured peer. Chain methods asserted that a requested height existed and passed a transaction argument on without a null check. The server assumed one outstanding request per thread because the C++ client never sends two, and the fix for that still assumed it for a thread reached through two proxies. A call after destroy, which the C++ client cannot send, went through a null implementation. A test-only escape hatch on the interface let GUI code reach node internals that do not exist across a process; a bool plus out-parameter return was serialized in its failure case; a byte-span encoder was added without its decoder; an empty byte vector reached memcpy with a null pointer; and nothing bounded how many block templates a peer could hold or how much of a message the debug log would print.

The library's own C++ client never does any of this, which is why each of these waited for the first Python or Rust client, the first fuzzer at the boundary, or a security review reading the dispatch code.

The rule we learned. On the day an interface goes over the wire, re-read every assertion reachable from its arguments, its configuration and its call order, and turn each into validation; encode "no result" in the return type; define both halves of every encoding together; bound what a peer can make the server hold or log; and test with a client that does not share your headers, because your own client always sends what you expect.

Representative bugs:

The bugs, by fine-grained pattern

An assertion on what is now input from a peer

An interface that only works inside one process

Empty input

Assuming every peer behaves like our own client

Unbounded log output from peer data

No limit on what a peer can make the server hold

An encoder without its decoder

Prevented by
a different API shape (9), a design rule (8), a review checklist item (4), the type system (2), static analysis (1)
Earliest detector
a test with a non-C++ client (4), a unit test (3), code review (2), a stress test (1), UBSan (1), fuzzing (1), static analysis (1), other (1)

Error paths that skip cleanup or completion

16 bugs · 16 production defects (20% of all) · Rust: 10 partly, 6 no

The success path was written and the failure path was assumed not to happen. Shutdown() called stop() on each chain client as if the call could not fail, which was true while every chain client was in-process; once one was a proxy, the peer being gone at shutdown was the normal case; likewise, an error returned by a client's Chain notification handler escaped the node's scheduler thread and terminated the node. A proxy destructor made a destroy round trip to the peer from a noexcept context, and other noexcept destructors started the cleanup thread or waited for a spawned child, so a thread limit or an interrupted waitpid terminated the process. A function posted to the event loop that threw skipped the two lines that cleared the slot and notified the posting thread, which then waited forever, and EventLoop::sync() logged and dropped a failure to wrap a new socket, so the caller went on with a null connection. A worker thread ran the method body under no handler, so a malformed request terminated the process, and the thread that builds Bitcoin Core's event loop had no way to report a failure to the caller waiting for it. A constructor that threw left a connection that only the destructor would have released; a function that created two socket descriptors leaked one when it threw between them, and failed IPC setup leaked its socket and left a spawned child unreaped; a listener's connection slot was released from a hook that fires only when the peer closes, not on a local close; and the accept loop's only error handler logged, so one dead socket, or one accept() that ran out of descriptors, ended the listener for good.

Nothing in the tree ever took these paths, so no test did either. The failures arrived with a Rust client that dropped its connection, a review that asked what happens if this throws, a platform that returned an error the code had never seen, and reviews that forced allocations, thread starts and system calls to fail on purpose.

The rule we learned. A call across a process boundary can fail at any moment, and cleanup, shutdown and destructors catch and continue. Every "done" signal owed to another thread is sent from a scope guard on every exit path, including exceptions, and the waiter learns that the work failed. Every thread entry point converts exceptions to an error result; resources acquired before a fallible step are owned by something with a destructor; and the error path is tested with a callee that actually fails, when the primitive is written.

Representative bugs:

The bugs, by fine-grained pattern

Assuming a remote call cannot fail

One error ends a service loop

Fallible work in destructor

A throw skips the done signal

A throw skips cleanup

An exception escaping a thread

Cleanup on only one of two exit paths

A constructor throw skips cleanup

Swallowed setup error

Prevented by
a design rule (9), a different API shape (6), a review checklist item (4), a clear ownership model (3)
Earliest detector
a unit test (8), a disconnect test (2), code review (2), a test that kills a process (1), a test with a non-C++ client (1), signal test (1), startup test (1)

Process spawning, file descriptors and signals

6 bugs · 5 production defects (6% of all) · Rust: 1 yes, 1 partly, 4 no

Spawning a child process from a multithreaded program touches operating-system state that no type system models. After fork() only async-signal-safe calls are allowed in the child, because every lock in the parent is copied in whatever state it was in; SpawnProcess ran a caller-supplied callback that allocated, and mpgen did the same with perror and std::filesystem. A descriptor's close-on-exec flag belongs to the process, not to the thread that set it, so clearing it in the parent "for the child" leaked the socket into every other thread's children until the parent closed it. The loop that closes "everything else" before exec used a documented inclusive maximum with <, leaving the top slot open. The spawned child stayed in the terminal's process group, so Ctrl-C reached the wallet process, whose default SIGINT handling killed it under a node about to call it. And a teardown written for unix sockets, where close() delivers queued bytes and an EOF, lost its final Release messages on Windows, where a closing socketpair discards unsent data.

These are old rules from the POSIX manual, and each was applied one site at a time as it was rediscovered: the post-fork fix left mpgen and one perror alone. A search for fork( after the first fix would have listed every site.

The rule we learned. Do all the work before fork() and nothing but system calls after it, or use posix_spawn so the question never arises; create every descriptor close-on-exec atomically and clear the flag only inside the child that should inherit it; a child whose shutdown another process drives installs its own signal handling before it serves, and says so in its output; and a protocol says goodbye explicitly, with a shutdownWrite() before the stream goes, rather than relying on what an operating system does with a closing socket's buffer.

Representative bugs:

The bugs, by fine-grained pattern

Unsafe work between fork and exec

A descriptor leaked into another thread's child

An off-by-one loop bound

Ctrl-C reaches the child process

Relying on a platform behavior nobody stated

Prevented by
a design rule (5), a different API shape (3), a written spec or documentation (2), a review checklist item (1)
Earliest detector
a unit test (3), a test that kills a process (1), code review (1), other (1)

Test timing, and tests that cannot fail

6 bugs · 0 production defects (0% of all) · Rust: 6 no

Tests in this code coordinate two or three processes and several threads, and a test that passes on a developer machine can be asserting a scheduling guarantee the system does not make. A functional test checked a second node's tip height straight after submitSolution returned on the first, before the block had propagated. The IPC mining test passed literal millisecond timeouts straight to the node, outside the framework helpers where the --timeout-factor scaling lives. A disconnect test slept a fixed 0.1 seconds instead of waiting for the "executing request" log line it needed to be past. A post-fork safety test gave the child one second, sized for a healthy run rather than the slowest CI runner. A connect test asserted which of two correct error strings a race would produce. Each failed weeks later on a loaded or sanitized runner, on someone else's PR.

One more was different in kind: a test compared an exception's what() text to a string no standard library produces, and passed anyway, because an expectation failure on a worker thread only printed a message. A test that cannot fail looks like coverage and is worse than none.

The rule we learned. Wait for the observable event, never for a duration; when a test hands a timeout to the system under test, scale it the way the framework would and size it for the slowest machine, since a healthy run never waits it out; assert the property, not the interleaving, or make the interleaving deterministic first; make an expectation failure on any thread fail the run; and review the test hunk in a bug-fix PR as carefully as the fix, because a flaky test costs every later PR. When a test fails on a missing log line, look for the line the code logged in its place before adjusting timeouts.

Representative bugs:

The bugs, by fine-grained pattern

A test that assumes timing

A test check that cannot fail

Prevented by
a review checklist item (6), a design rule (3), a different API shape (2)
Earliest detector
code review (3), a stress test (2), a unit test (1)

CI, toolchain and platform

10 bugs · 4 production defects (5% of all) · Rust: 3 no, 7 n/a

These records have their cause below the project's code. A CI job tuned to sit just under its memory limit was killed linking bitcoin-gui when an unrelated PR grew the build; a stress workflow sized its parallelism for a cheap test and ran forty copies of an expensive one at once. A job that linked GCC-built dependencies with a clang-built program met an i686 ABI disagreement over an empty struct in a kj header, and a second 32-bit crash inside kj's exception formatter is still undiagnosed. Cap'n Proto 0.8, the oldest version the project claimed to support, recursed forever under C++20's rewritten comparison candidates, and no build had ever combined the two. NetBSD's system header stubbed __has_feature to zero, kj believed exceptions were off, and every error path in the library became a crash. The mingw-w64 runtime ran a thread_local destructor after freeing its storage, on a construct Bitcoin Core's lint forbids and the library had waived, and on Windows the same kind of destructor waited for the event loop while holding the loader lock, and deadlocked. kj before Cap'n Proto 1.2 aborted when the monotonic clock stepped back, a bug fixed upstream but still in the versions distributions ship. A test's schema search list put the system directory first, so an installed copy shadowed the tree under test.

What the project owned in each case was the exposure rather than the defect: a mixed-compiler job nobody labelled as an ABI test, a minimum dependency version CI never built, a waived rule, a build-time assumption never asserted, a resource margin never measured. Diagnosis was the hard part, often days with gdb far from the cause.

The rule we learned. Build against the floor of what you claim to support as well as the ceiling, keep one toolchain per link, and assert build-time assumptions such as "exceptions are enabled" at compile time with an #error. Measure a CI job's peak against its limit when the job is first made heavy and leave a margin. When a PR's own run dies in a new way, treat it as the PR's problem until shown otherwise.

Representative bugs:

The bugs, by fine-grained pattern

A CI job at the edge of its limits

Objects from two compilers in one program

A thread-local variable with a destructor

A feature macro detected wrongly

An installed copy found before the tree under test

Stale dependency bug

A minimum dependency version CI never built

Prevented by
a design rule (4), nothing practical (4), other (2)
Earliest detector
a unit test (5), other (3), code review (1), an assertion (1)

What would have prevented these bugs

Counts over the 102 bugs that have lessons written.

What would have prevented them

A bug can list more than one way it could have been prevented.

Prevention Bugs Share
Design rule 61 60%
Review checklist item 31 30%
Different API shape 29 28%
Clear ownership model 13 13%
Thread-affinity assertion 7 7%
Lock annotations 6 6%
Type system 6 6%
Written spec or documentation 6 6%
Nothing practical 4 4%
Other 3 3%
Static analysis 1 1%

The earliest check that would have caught them

The cheapest existing or plausible check that would have caught each bug first, and whether the project had that check when the bug was introduced.

Earliest detector Bugs Check existed then Did not exist then Not recorded
Unit test 25 3 22 0
Disconnect test 14 0 14 0
ThreadSanitizer 10 0 10 0
Code review 10 8 2 0
Test with a non-C++ client 9 0 9 0
Stress test 7 0 7 0
Other 7 2 5 0
Test that kills a process 5 0 5 0
Static analysis 4 0 4 0
Startup test 4 0 4 0
UBSan 3 0 3 0
Fuzzing 1 0 1 0
Assertion 1 0 1 0
Shutdown test 1 0 1 0
Signal test 1 0 1 0

Was the catching check written with the fix?

Written with the fix Bugs
Yes 43
No 53
Not recorded 6

How hard to diagnose, and how hard to fix

Rows: how hard the root cause was to recognize once the failure was seen. Columns: how hard the right fix was to design once the cause was known.

Diagnosis Fix easy Fix moderate Fix hard Fix not recorded Total
Easy 42 15 0 3 60
Moderate 16 10 3 2 31
Hard 2 3 5 1 11

What helped, or would have helped, debugging

Debugging aid Bugs
Reproduction script 36
Logging 30
Clear assertion message 24
Sanitizer output 24
Symbolized stack traces 21
CI artifacts 12
None 9
Hook to force the timing 8
Thread names 6
Other 6
Core dumps 3
Lock annotations 2

The lesson is useful outside libmultiprocess and IPC for 101 of 102 bugs.

Would Rust have prevented these? · What would help now · Findings