Stability bugs in libmultiprocess
and Bitcoin Core IPC

102 bugs, 2019 to 2026: what they were, how they were found, and what is left

Ryan Ofsky
ipcbugs.ofsky.org

This is the written version of a talk about every stability bug the IPC code in Bitcoin Core has had so far, from the first libmultiprocess commits in 2019 to October 2026.

I went through the whole history of libmultiprocess and of the IPC code in Bitcoin Core and wrote up every stability bug I could find. There are 102 of them.

The talk spends a few minutes on what the bugs are and why they showed up when they did, then walks through a deadlock and a chain of cancellation bugs in code. After that it looks at where the bugs lived, whether my own theory about them held up against the data, and what is still open. Every bug mentioned here has its own page on ipcbugs.ofsky.org, with links to the PRs, issues and review comments behind it.

Before getting to the bugs, it is worth being precise about what was counted.

What the survey is

102 stability bugs: crashes, asserts, hangs, deadlocks, data races, undefined behavior, leaks
In libmultiprocess, and in Bitcoin Core code that exists only because of IPC
One record per root cause, usually one fix commit; test-code and CI bugs kept as their own classes
Sources: both git histories, a local mirror of both GitHub trackers, tracking issue bitcoin#28722, my notes, two private security reviews
Every record written by one reader and checked field by field against the diffs by a second

Models did the triage, verification and drafting; I reviewed and made the judgment calls. Method: ipcbugs.ofsky.org/method

The first question is what counts as a stability bug, and what counts as one bug.

Stability here means the process stops working: it crashes, hits an assert, hangs, deadlocks, races, runs into undefined behavior or leaks. A method that returns a wrong value is out of scope. The survey covers libmultiprocess and the Bitcoin Core code that exists only because of IPC; a Bitcoin Core bug counts only if it would not exist without IPC.

Each record is one root cause, which usually means one fix commit, so a PR that fixes three races produces three records. Bugs in test code and CI are kept as their own classes, because they cost real time even though no user sees them. Split a different way, the total would be anywhere from 94 to 104.

Models did the enumeration, the first drafts and an independent second check of the first 86 records against the diffs. I reviewed them and made the judgment calls. Confidence is high for 68 records, medium for 19 and low for 15, and the method page has the details. Here is what came out.

The numbers

102

stability bugs, 2019 to 2026
94 to 104 depending on how they are split

82

in production code
13 in test code, 7 CI, build or example

87

fail-stop: crash, assert, hang, deadlock, race, UB
one silent wrong result (53)

69

fixed or worked around
33 open, 13 of them with a fix in review

8

reported from real deployments
all Stratum v2 software on the mining interface

12

days, median from first report to merged fix
mean 123; 17 took over 90 days

The whole survey fits in six numbers: 102 bugs, nearly all of them fail-stop, most of them fixed, and only eight ever reached someone running the software for real.

Of the 102, 82 are defects in production code. The rest are in test code (13) or in CI, the build or examples (7). They fail loudly: 87 are crashes, asserts, hangs, deadlocks, races or undefined behavior. Only one production record silently did the wrong thing, bug 53, where submitSolution accepted a block with an uncommitted coinbase.

69 are fixed or worked around. The other 33 are open, 13 of them with a fix in review and 20 without one. Eight were reported from real deployments, and all eight were Stratum v2 software using the mining interface.

The last tile is about speed. The median bug was fixed 12 days after its first report, but the mean is 123 days, and 17 bugs took more than 90. That long tail is mostly bugs waiting for a branch to merge.

So what are these bugs, concretely?

What the bugs are

42 of 82 production defects need a disconnect, a shutdown or a signal to happen

The mechanisms fall into 11 pattern groups. These five cover 58 of the 82:

Pattern	The rule the fixes imply	defects
An object outliving what it points to	Whatever an object touches in its destructor or callbacks, it must own or keep alive	11
An in-process interface trusting what now comes from another process	Once an interface is on the wire, every assert reachable from its inputs becomes validation or an error	14
Error paths that skip cleanup or completion	Any call across a process boundary can fail; cleanup and “done” signals run on every exit path	16
Work or cleanup on the wrong thread	State owned by one thread is read, written and destroyed only on that thread	7
Locks, waits and wakeups	Every reader and writer of a guarded field takes the lock; notify under the waiter's mutex	10

Six more groups, the largest serving before ready and freeing before done (8): ipcbugs.ofsky.org/patterns

Nearly every bug is about tearing something down while something else is still using it, and the same five mechanisms cover two thirds of them.

[42 of 82] 42 of the 82 production defects need a connection to end: a disconnect, a shutdown or a signal. The most common trigger, 17 records, is a client going away with a call in flight. All 17 are in server code.

[table] Every bug has a lesson that names its mechanism, and the lessons fall into eleven groups. These five cover 58 of the 82.

[first row] The classic lifetime bug: a server object outlives its Connection (30), a map keyed by a pointer finds a new object at a reused address (18).

[second row] The one that keeps coming back with each new client: code written for callers in the same process, now taking input from another one. The GUI in 2021 and 2022, then the Mining and Node interfaces from 2025: default options, bad timeouts.

[last two rows] Wrong thread and locks are where the walkthroughs come from. Bugs 45 and 79 are both wrong-thread bugs.

[footer] The other six groups, with every bug in each, are on the patterns page.

Would a different language have avoided these?

Nearly every bug in the survey is about tearing something down while something else is still using it, and five mechanisms cover two thirds of them.

42 of the 82 production defects need a connection to end before they can happen: a disconnect, a shutdown or a signal. The most common trigger, behind 17 records, is a client going away while a call is still in flight. All 17 of those are in server code, which is why the server side dominates: 48 of the 82 are in server code, against 10 in client code, 11 in both, and 12 in the event loop or process spawning.

Every record has a lesson that names the mechanism behind the bug, and the patterns page sorts them into eleven groups, each with the rule its fixes imply. The five on the slide cover 58 of the 82 production defects.

The first two rows have 11 and 14. An object outliving what it points to is the classic lifetime bug: a server object outlives its Connection (30), a map keyed by a Connection pointer finds a new connection at a reused address (18), a disconnect handler runs after its Connection is gone (77). An in-process interface trusting what now comes from another process is different. Code written for callers in the same process gets called from another process and one of its assumptions breaks: the GUI in 2021 and 2022, then the Mining and Node interfaces from 2025, with clients that pass default options the in-process code never sees or set bad timeouts. That group keeps growing with each new interface and client.

Error paths that skip cleanup or completion (16), now the largest group, are the cross-process version of assuming a call cannot fail: a destructor that throws when the peer is gone, a thread that dies on an uncaught exception, a "done" signal a throw skips. Work on the wrong thread (7) and locks, waits and wakeups (10) are where the walkthroughs later in the talk come from; bugs 45 and 79 are both wrong-thread bugs.

The six smaller groups are on the patterns page. The largest of them, serving before ready and freeing before done, has 8 production defects, among them a client connecting before the chainstate is loaded; cleanups that ran first-in-first-out when they needed last-in-first-out (17) are one of the 3 built or torn down in the wrong order.

A question that comes up with any list of C++ memory and threading bugs is whether a different language would have prevented them.

Would Rust have helped?

24

yes: would not compile in safe Rust
13 of them lifetime or wrong-thread bugs

27

partly: harder to write, or fails differently
an abort becomes a panic, a crash a hang

29

no: ordering, deadlock, protocol, trust
plus 2 toolchain problems, not applicable

Rust would have ruled out about a third of production defects: mostly the lifetime and thread-affinity groups; not the hangs, lifecycle ordering or wire-boundary trust.

Send and Sync stop data races (all 7), not ordering races or deadlocks (none of the 15 hangs and deadlocks)
capnproto-rust's RPC objects are single-threaded, so work on the wrong thread cannot be written
Arc turns a use-after-free into a longer lifetime; here that means a node that never exits

Over the 82 production defects, judged per bug from its fix: ipcbugs.ofsky.org/patterns/rust

Rust would have rejected about a third of the production defects at compile time, and none of the hangs.

[tiles] Each lesson records a one-line judgment made from the fix: would safe Rust, with the pure-Rust Cap'n Proto, have prevented this bug? Over the 82 production defects it comes out close to a third each: 24 yes, 27 partly, 29 no.

[24] 13 of the 24 come from two groups: objects outliving what they point to (9 of 11) and work on the wrong thread (4 of 7).

[first bullet] Send and Sync make each of the 7 data races a compile error. They say nothing about order: an unlocked notify compiles, and holding a lock while waiting on the loop thread deadlocks in either language. None of the 15 hangs and deadlocks gets a yes.

[second bullet] capnproto-rust's clients and connections are Rc-based and not Send, so only the loop thread can touch them. That rules out the wrong-thread group, at the cost of a different model for blocking calls.

[third bullet] The idiomatic replacement for a raw pointer is an Arc. It removes the crash, but here the thing kept alive is often a connection or an event loop reference, and the loop only exits when its last reference drops.

If these are mostly old teardown paths, why did they only show up recently?

Rust would have rejected about a third of the production defects at compile time, and none of the hangs.

Each lesson records a one-line answer to the question, judged from the fix rather than by porting the code, and assuming the pure-Rust Cap'n Proto implementation (capnproto-rust) so there is no C++ layer underneath. Over the 82 production defects the answers split close to a third each: 24 yes, 27 partly, 29 no, and 2 toolchain problems where the question does not apply. The yes answers cluster in the lifetime and thread groups: 9 of the 11 objects outliving what they point to, and 4 of the 7 production bugs from work on the wrong thread. Error paths, process spawning and the wire boundary get almost none: those bugs are decisions about what a peer may send and what an error path does, not memory.

Send and Sync stop data races, not ordering races or deadlocks. All 7 production data races would be compile errors, because a field shared between threads has to live inside a Mutex or be atomic. But Condvar::notify_all takes no guard, so the unlocked notify that lost a wakeup compiles, and holding one mutex while blocking on a hand-off to the loop thread is a lock-order deadlock in either language. None of the 15 production hangs and deadlocks is a yes.

capnproto-rust's clients, connections and call contexts are single-threaded (Rc-based and not Send), so a worker thread cannot hold, destroy or call into one. That rules out the wrong-thread class by construction. The cost is a different model for blocking calls: libmultiprocess runs method bodies on worker threads that block, and in capnproto-rust that work has to be handed to another thread explicitly.

Arc turns a use-after-free into a lifetime extension. Where the C++ code had a raw pointer to a connection, Rust would have an Arc, which removes the crash. In this code the object kept alive is often a connection or an EventLoopRef, and the event loop only exits when its last reference drops, so a lifetime that runs past shutdown is a node that never exits. The survey already has that hang twice, from C++ code that kept references too long. The Rust page has the answer for every group, with examples of each verdict.

If these are mostly old teardown paths, why did they only show up recently?

Old code, new exposure

all bugs production defects production defects from 2019 code

9 first reports before 2024, 13 in 2024, 32 in 2025, 48 so far in 2026

37 of 73 dated production defects were in the library's first commits, June to August 2019

19 of the 41 production defects reported in 2026 had been in the code since 2019

Median bug lived 996 days from introducing merge to fix

The bugs are old. What changed was who and what was exercising the code.

[blue line] Cumulative first reports: 9 before 2024, then 13 in 2024, 32 in 2025 and 48 so far this year.

[green line] Green is production defects that were already in the library's first commits in 2019. It climbs as fast as the others: 37 of the 73 dated production defects come from June to August 2019, and 18 of this year's 41 had been in the code for seven years.

[dotted milestones] Each step follows something new reaching the code: new tests, Rust and Python clients, TSan, the v30 release, Antithesis, reviews. Each was followed within weeks by two to six records.

[996 days] The median bug lived almost three years between the merge that introduced it and the merge that fixed it.

The next question is who, or what, found them.

The bugs are old. What changed was who and what was exercising the code.

The blue line counts first reports over time: 9 before 2024, then 13 in 2024, 32 in 2025 and 48 so far in 2026. The orange line counts production defects only.

The green line is the telling one. It counts production defects that were already present in the library's first commits, from June to August 2019, and it climbs as fast as the others. 37 of the 73 dated production defects come from those three months, and 18 of the 41 production defects reported in 2026 had been in the code for seven years.

The dotted lines mark moments when something new reached the code: new tests, Rust and Python clients, TSan in the library's CI, the v30.0 release, Antithesis, private reviews. Each was followed within weeks by two to six new records. The median bug lived 996 days, almost three years, between the merge that introduced it and the merge that fixed it.

The next question is who, or what, found them.

How they were found

Production defects by first report, per half year (2026 H2 is to mid October)

2026: 2 of 41 production defects first found by someone running the software

20 of 30 production defects since July 2026 came from review

first found by	median days to fix
TSan	5
review	18
intermittent CI	25
user report	53.5
manual testing	188

Detection moved from people running the software, to tools, to review, and the bugs found by tools were the cheapest to fix.

Each bar is a half year of production defects, split by what first found them. Through 2025, no source found more of them than people running the software, the blue segments. In the two 2026 bars there are only two, both found by developers testing an unmerged branch.

Review, the pink segments, took over instead: 20 of the 30 production defects reported since July 2026 came from review. Nineteen of the 24 that review found in 2026 came with a report that says, or reads as if, an LLM helped.

The table on the right is the cost side. A bug first found by TSan was fixed in a median of 5 days. Review finds took 18 days, intermittent CI failures 25, user reports 53.5, and bugs found by manual testing 188. Red CI is a cost as well: 39 records drew at least one CI failure report, 59 reports in all, and each one is time someone spent looking at a failure.

To show what these bugs look like in code, the next slide first explains how the library uses threads.

The black box, in one slide

Event-loop thread

all socket I/O, all promises
owns Connection and Cap'n Proto connection state
on disconnect: destroys the Connection, runs its cleanup callbacks

post the request

loop->sync(fn)
worker blocks until the loop has run fn

Worker threads

one per thread the client asks for
run the method body of every call with an mp.Context (all mining calls)
may still be running when the client goes away

Where the fixes converge: touch connection state only on the event-loop thread, and never hold a lock while waiting for it.

Before the code walkthroughs, one slide on how libmultiprocess uses threads, because almost every bug is a disconnect racing between two kinds of thread.

Each process has one event-loop thread that does all the socket I/O and resolves all the promises. It owns the Connection object and Cap'n Proto's connection state, and when a peer goes away it destroys the Connection and runs its cleanup callbacks.

Method bodies run on worker threads instead, one per thread the client asks for, so a long call like waitNext does not block the loop. Every call with an mp.Context runs this way, which includes all the mining calls. A worker that needs the loop calls loop->sync(fn) and blocks until the loop has run fn. Workers may still be running when the client goes away.

So a disconnect is always a race: the loop tears things down while workers may still be in the middle of a call. The box at the bottom of the slide states the rule the fixes converge on: touch connection state only on the event-loop thread, and never hold a lock while waiting for it.

The first bug breaks the second half of that rule.

Bug 45: two threads erase one entry mp#201, 2025

Worker thread, after the method returns: PassField, include/mp/type-context.h

Event-loop thread, run by ~Connection after a disconnect: callback from SetThread, src/mp/proxy.cpp

Erasing destroys a ProxyClient<Thread>, whose cleanup waits for the event loop: ProxyClientBase, include/mp/proxy-io.h

lock taken (same mutex: threads.mutex is Waiter::m_mutex) destroys the entry or waits

Two threads destroy the same map entry, and one of them holds a lock while waiting for the other.

The server keeps a map, request_threads, from each client connection to a proxy object for that client's thread. Two code paths erase the same entry, with nothing ordering them.

[first block, yellow line] On the worker thread, after the method returns, a KJ_DEFER takes the Waiter mutex, [red line] and erases the entry for this connection.

[second block] On the event-loop thread, when the connection goes away, ~Connection runs a callback that takes a mutex and erases the same entry. [footer] It is the same mutex: threads.mutex is Waiter::m_mutex.

[third block] The last piece: erasing the entry destroys a ProxyClient<Thread>, and its cleanup calls loop->sync. It waits for the event-loop thread.

Both erase paths came in with mp#106 in 2024, the fix for bug 18, where this map was keyed by a Connection pointer whose address got reused. So this is also a fix that caused a later bug. It will not be the last one today.

Put the three pieces on a timeline and you get a cycle.

The first walkthrough is bug 45 (mp#201, 2025), a deadlock in which two threads destroy the same map entry and one of them holds a lock while waiting for the other.

The server keeps a map, request_threads, from each client connection to a proxy object for that client's thread. Two code paths erase the same entry, and nothing orders them.

The first block runs on the worker thread after the method returns. A KJ_DEFER takes the Waiter mutex (highlighted in yellow) and then erases the entry for this connection (in red). The second block runs on the event-loop thread: when the connection goes away, ~Connection runs a callback that takes a mutex and erases the same entry. As the footer points out, it is the same mutex, because threads.mutex is Waiter::m_mutex.

The third block is the last piece. Erasing the entry destroys a ProxyClient<Thread>, and its cleanup calls loop->sync, so it waits for the event-loop thread.

Both erase paths came in with mp#106 in 2024, the fix for bug 18, where this map was keyed by a Connection pointer whose address got reused. So this is also a fix that caused a later bug, and it is not the only one in this talk. Put the three pieces on a timeline and they form a cycle.

Bug 45: the lock cycle

Found: TSan on Sjors's CI (2025-07); mptest hanging on ARM CI; a segfault in CI

In production: a v30.0 node stopped answering IPC after about 25 minutes under a Rust Stratum v2 client

The worker holds the mutex and waits for the loop; the loop waits for the mutex. Neither moves.

[left column, top to bottom] The client disconnects just as a method returns. The worker takes Waiter::m_mutex, erases the entry, and the destructor calls loop->sync. It now waits for the loop while still holding the mutex.

[right column] The loop thread is inside ~Connection, running the disconnect callback, which wants the same mutex.

[red arrows] Each waits for the other. With slightly different timing the two erases overlap instead, and you get the TSan race or a segfault.

[bottom right] The production case had no disconnect at all. The Rust client sends each request on the same thread, so the loop was handing the next getBlock call to that worker in Waiter::post, which takes the same mutex. A v30.0 node stopped answering IPC after about 25 minutes. I reproduced it in about a minute, and gdb showed the same two stacks.

[bottom left] Four reports: Sjors's CI under TSan, mptest hanging on ARM CI, where TheCharlatan needed about 50,000 runs to reproduce it, a CI segfault, and that node.

The fix is not a lock order.

Laid out as a timeline, the three pieces of bug 45 form a lock cycle: the worker holds the mutex and waits for the loop, and the loop waits for the mutex.

Read the left column from top to bottom. The client disconnects just as a method returns. The worker takes Waiter::m_mutex, erases the entry, and the destructor calls loop->sync, so it now waits for the loop while still holding the mutex. In the right column, the event-loop thread is inside ~Connection, running the disconnect callback, which wants the same mutex. The two red arrows show each thread waiting for the other. With slightly different timing, the two erases overlap instead, and the result is the TSan report or a segfault.

The production case had no disconnect at all. The Rust client sends each request on the same thread, so the loop was handing the next getBlock call to that worker in Waiter::post, which takes the same mutex. A v30.0 node stopped answering IPC after about 25 minutes. I reproduced it in about a minute, and gdb showed the same two stacks.

There were four reports in all: TSan on Sjors's CI in July 2025; mptest hanging on ARM CI, where TheCharlatan needed about 50,000 runs to reproduce it; a segfault in CI; and that node. The fix is not a lock order.

Bug 45: the fix 4a269b21b8c8, 76 lines

include/mp/type-context.h

Every add, erase and callback removal for these entries runs on the loop thread, including SetThread and ~ProxyClient<Thread>
The node is destroyed after the mutex is released

Merged 2025-09-17; reached 30.x after v30.0 shipped
The thread-exit path of the same map (37) is still open: mp#361

A lock order would not fix it. Moving the destruction onto the loop thread does.

Move the destruction onto the event-loop thread, rather than trying to order the locks.

[green lines] The worker now calls loop->sync without holding the mutex. Inside, on the loop thread, it extracts the map node under the mutex, and the node is destroyed after the mutex is released.

[left bullets] Every add, erase and callback removal for these entries now runs on the loop thread, the same thread as ~Connection. So the disconnect callback runs either entirely before the erase, which then finds nothing, or after it, when the callback is already unregistered.

[right bullets] Merged 2025-09-17, after v30.0 shipped. The sibling, 37, a thread exiting while the loop destroys its Connection, was reported in the same issue and is still open. Its fix is in mp#361.

[blue box] If you keep one thing from this talk: the loop thread is a shared resource, and anything that holds a lock while waiting for it can deadlock the process.

The second walkthrough is a chain of three bugs, where each fix led to the next find.

The fix for bug 45 does not order the locks. It moves the destruction onto the event-loop thread.

In the diff, the green lines show the worker calling loop->sync without holding the mutex. Inside the sync block, on the loop thread, it extracts the map node under the mutex, and the node is destroyed after the mutex is released.

The same change moves every add, erase and callback removal for these entries onto the loop thread, including SetThread and ~ProxyClient<Thread>. That is the thread that runs ~Connection, so the disconnect callback now runs either entirely before the erase, which then finds nothing, or after it, when the callback is already unregistered.

The fix, commit 4a269b21b8c8, is 76 lines. It was merged on 2025-09-17 and reached the 30.x branch after v30.0 shipped. Its sibling, bug 37, a thread exiting while the loop destroys its Connection, was reported in the same issue and is still open; its fix is in mp#361.

If you keep one thing from this talk, keep the box at the bottom of the slide: the event-loop thread is a shared resource, and anything that holds a lock while waiting for it can deadlock the process. The second walkthrough is a chain of three bugs, where each fix led to the next find.

The cancellation chain: 3, then 5, then 79

BUG 3 · mp#240

A killed client's `waitNext` returns into a deleted `Connection`

Field reports from Rust Stratum v2 clients, v30.0rc3 to v30.2, from 2025-10

Fix: cancel requests on disconnect. 267 lines, 2026-02-20

caused

BUG 5 · mp#249

The cancel frees the params; the worker reads them anyway

Antithesis, 2026-03-09; sanitizer CI; a Rust client on master

Fix: read params on the loop thread, after the cancel check. 9 lines, 2026-03-27

slipped past

BUG 79 · mp#349

The worker's first `getResults()` reads capnp state the loop is tearing down

Antithesis with TSan, 2026-08-19; in code since 2019

Fix: call it once on the loop thread. 19 lines, 2026-08-26

mp#240 also caused 4 and 6, found within three weeks of merging. 4, 5 and 6 never reached a release.

One structural fix, request cancellation, fixed a field crash and then needed two small follow-ups.

[first card] Bug 3. A Rust Stratum v2 client is killed while waitNext is running on the server. Nothing cancelled a server request when its connection went away, so when fees rose and waitNext returned a new template, the worker wrapped it for a Connection that was already deleted. Five reports from the field, v30.0rc3 to v30.2.

The fix, mp#240, cancels every pending request when its Connection is destroyed. At 267 lines it is one of the two largest fixes in the survey.

[second card] Bug 5. Cancelling a request frees its parameters, and the worker read them before checking for cancellation. Antithesis found it less than three weeks after the merge.

[third card] Bug 79 is from 2019 and was not caused by mp#240. But the cancellation never runs on its path, so it slipped past both fixes.

[line at the bottom] mp#240 also caused 4 and 6. All three regressions were fixed before any release had them.

Here is bug 5 in code.

First there was a structural fix, request cancellation, for a crash that Stratum v2 users were hitting in the field. It then needed two small follow-up fixes, and each fix in the chain led to the next find.

The first card is bug 3. A Rust Stratum v2 client is killed while waitNext is running on the server. Nothing cancelled a server request when its connection went away, so when fees rose and waitNext returned a new template, the worker wrapped it for a Connection that was already deleted. There were five reports from the field, from v30.0rc3 to v30.2, starting in October 2025. The fix, mp#240, cancels every pending request when its Connection is destroyed. At 267 lines, merged on 2026-02-20, it is one of the two largest fixes in the survey.

The second card is bug 5, which that fix caused. Cancelling a request frees its parameters, and the worker read them before checking for cancellation. Antithesis found it on 2026-03-09, less than three weeks after the merge, and the fix was 9 lines.

The third card, bug 79, has been in the code since 2019 and was not caused by mp#240. But cancellation never runs on its path, so it slipped past both fixes until Antithesis, running with TSan, found it in August 2026. Its fix was 19 lines.

As the line under the cards says, mp#240 also caused bugs 4 and 6, found within three weeks of merging. All three regressions were fixed before any release had them. The next slide shows bug 5 in code.

Bug 5: read first, check second mp#249, 2026

Worker thread: PassField, include/mp/type-context.h, before the fix. The sync() lambda runs on the event-loop thread.

Fix (9 lines): drop the early read; read the params inside sync(), after the check

Loop: posts the request → Loop: client disconnects, request cancelled, params freed → Worker: getParams() on freed memory

Read first, check second: once requests can be cancelled, the order of those two lines matters.

[red line] Before mp#240, requests were never cancelled, so reading the params at the top of the worker was safe.

[yellow line] After it, a disconnect cancels the request and Cap'n Proto frees the params. The cancellation check is here, inside loop->sync on the loop thread, but the worker has already read them.

[sequence at the bottom] The loop posts the request; the client disconnects, the request is cancelled and the params freed; then the worker calls getParams() on freed memory.

Antithesis reported it first, as an ASan use-after-free. Then the library's sanitizer CI, then a plain segfault with a Rust mining client on a master build, about one run in two or three.

[green line] The fix is 9 lines: read the params inside the sync block, after the check. Cancellation also runs on the loop thread, so once the check has seen "not cancelled", the params cannot be freed before the read returns.

The last bug in the chain is inside Cap'n Proto itself.

Bug 5 (mp#249, 2026) comes down to two lines in the wrong order: once requests can be cancelled, reading the parameters before checking for cancellation is a use-after-free.

Before mp#240, requests were never cancelled, so reading the params at the top of the worker, the red line, was safe. After it, a disconnect cancels the request and Cap'n Proto frees the params. The cancellation check is there, on the yellow line inside loop->sync, which runs on the event-loop thread, but by then the worker has already read the params.

The sequence under the code spells out the order: the loop posts the request; the client disconnects, so the request is cancelled and its params freed; then the worker calls getParams() on freed memory. Antithesis reported it first, as an ASan use-after-free. Then the library's sanitizer CI hit it, then a plain segfault with a Rust mining client on a master build, about one run in two or three.

The fix, the green line, is 9 lines: read the params inside the sync block, after the check. Cancellation also runs on the loop thread, so once the check has seen that the request is not cancelled, the params cannot be freed before the read returns.

The last bug in the chain is inside Cap'n Proto itself.

Bug 79: inside Cap'n Proto mp#349, 2026

Worker thread, after the method returns: ServerRet::invoke, include/mp/proxy-types.h

Cap'n Proto, first call only (later calls return a cached response): RpcCallContext::getResults, capnp/rpc.c++ 1.5.0

Event-loop thread, abrupt disconnect: RpcConnectionState::disconnect, two steps

Fix: fill the cache on the loop thread, at the end of PassField's existing loop.sync block

Even Cap'n Proto's own objects reach into state the loop owns, so the fix does that one read on the loop thread.

[first block] After the method returns, the worker calls getResults() to start building the reply.

[second block] The first time it is called, Cap'n Proto checks whether the connection is still up, to decide where to build the reply. Later calls return a cached object, so only the first call is racy.

[third block] On an abrupt disconnect, the loop thread tears the connection down in two steps: it moves the live connection out, leaving a null pointer, then marks the field disconnected. A worker reading between the two sees "connected" and dereferences null.

Antithesis found it with a TSan build, by killing the client during createNewBlock. No crash was ever seen, only the TSan report, but that interleaving would take down the node.

[green line] The fix is one call: getResults() on the loop thread, before the worker gets the request. 19 lines with the test, merged in libmultiprocess on 2026-08-26. It is not in a Bitcoin Core subtree update yet, so v30.0 through v31.1 have the race.

If there is time, one more bug: a server object that outlives its connection. If not, skip to slide 16.

Bug 79 (mp#349, 2026) shows that even Cap'n Proto's own objects reach into state the event loop owns, so its fix does that one read on the loop thread.

In the first block, after the method returns, the worker calls getResults() to start building the reply. The second block is Cap'n Proto's side. The first time getResults() is called, it checks whether the connection is still up, to decide where to build the reply. Later calls return a cached object, so only the first call is racy.

The third block is the event-loop thread during an abrupt disconnect. It tears the connection down in two steps: it moves the live connection out, leaving a null pointer, and then marks the field disconnected. A worker that reads between the two steps sees a connected state and dereferences null.

Antithesis found it with a TSan build, by killing the client during createNewBlock. No crash was ever seen, only the TSan report, but that interleaving would take down the node.

The fix, in green, is one call: getResults() on the loop thread, at the end of the existing sync block that runs before the worker gets the request. It is 19 lines with the test and was merged in libmultiprocess on 2026-08-26. It is not in a Bitcoin Core subtree update yet, so v30.0 through v31.1 have the race.

One more walkthrough follows, which is optional in the talk: a server object that outlives its connection.

Optional: bug 30, a server outliving its connection

~Connection, src/mp/proxy.cpp: the assumption

Worker, after a long call returns: PassField (before mp#118) and ~ProxyServerBase (before mp#160)

2024-10-16	Ctrl-C a mining client during `waitFeesChanged`: segfault. Same-day workaround leaks the server object
2025-06-10	darosior's Rust client in a reconnect loop: ASan use-after-free on the other path
2025-06-19	Fix, mp#160: every proxy holds its own event-loop pointer, `m_context.loop`

This is the clearest case of an object outliving its transport, and of a workaround that bought time at a price.

Optional; skip it if short on time.

[first block, yellow line] ~Connection assumed that resetting the RPC system destroys every server object. That holds unless a call is in flight: the call keeps its ProxyServer alive, so the Connection goes first.

[second block, red lines] When the long call returns, the worker reaches the event loop through the freed Connection.

[table] tdb3's Ctrl-C of a mining client segfaulted in October 2024. The same-day workaround avoided the crash by leaking the server object. In June 2025 darosior's Rust client hit the other path under ASan. The fix, mp#160, gave every proxy its own pointer to the event loop.

It took eight months and 133 lines, and the new disconnect tests that came with it exposed bug 45 within weeks.

Writing these up, one thing surprised me: how often the fix commit was wrong about what it fixed.

Bug 30 is the clearest case of an object outliving its transport, and of a workaround that bought time at a price. In the talk this slide is optional.

The first block is from ~Connection, and its comment states the assumption, highlighted in yellow: resetting the RPC system garbage-collects every server object that was not freed before the connection closed. That holds unless a call is in flight. The call keeps its ProxyServer alive, so the Connection goes first. The red lines in the second block show what happens next: when the long call returns, the worker reaches the event loop through the freed Connection, in PassField before mp#118 and in ~ProxyServerBase before mp#160.

The table gives the history. On 2024-10-16, tdb3 pressed Ctrl-C on a mining client during waitFeesChanged and the node segfaulted. The same-day workaround avoided the crash by leaking the server object. On 2025-06-10, darosior's Rust client, running in a reconnect loop, hit the other path as an ASan use-after-free. The fix, mp#160 on 2025-06-19, gave every proxy its own pointer to the event loop, m_context.loop.

It took eight months and 133 lines, and the new disconnect tests that came with it exposed bug 45 within weeks.

Writing these bugs up, one thing surprised me: how often the fix commit was wrong about what it fixed.

Fog of war: what the fix commits said

Commit said	Later analysis found
196e6fcdbc1f (2024, bug 30) "prevent null pointer dereference"	A use-after-free of the freed `Connection`; ASan reported it that way in 2025
c685fa9bedaf (2019, mp#7) "onDisconnect events are now cancelled as Connection objects are destroyed"	A handler already re-posted to the event loop is not cancelled: bug 77, found 2026-08, fix in mp#361
c4762c7b513a (2026, mp#240) "does not change behavior other than changing log messages"	Running the call inside `kj::runCatchingExceptions` is the fix for 57: an exception on a worker thread terminated the node

18 commit messages and 9 PR or review statements are corrected on the bug pages, each with the evidence. Several are mine.

The explanation written at fix time is often incomplete, so check a diff against its commit message.

This is not about blame: most of these commits are mine.

[first row] The 2024 workaround for bug 30 called it a null pointer dereference. It was a use-after-free of the Connection, and ASan said so in 2025.

[second row] A 2019 commit says disconnect handlers are cancelled when the Connection is destroyed. A handler already re-posted to the loop is not. That window was found seven years later: bug 77.

[third row] A commit in mp#240 says it only changes log messages. It is the fix for bug 57, an exception on a worker thread that terminated the node.

Two slides will be added after this one: "Rules we learned the hard way" and "Would Rust have prevented these?". Until they land, go straight on.

Now: where did these defects actually live, and who did they reach?

The explanation written when a bug is fixed is often incomplete, so it pays to check a diff against its commit message.

This is not about blame: most of these commits are mine.

In the first row, the 2024 workaround for bug 30, 196e6fcdbc1f, called the problem a null pointer dereference. It was a use-after-free of the Connection, and ASan said so in 2025.

In the second row, a 2019 commit from mp#7, c685fa9bedaf, says disconnect handlers are cancelled when the Connection is destroyed. A handler that has already been re-posted to the event loop is not. That window was found seven years later, in August 2026, as bug 77, and its fix is in mp#361.

In the third row, a 2026 commit in mp#240, c4762c7b513a, says it only changes log messages. In fact, running the call inside kj::runCatchingExceptions is the fix for bug 57, where an exception on a worker thread terminated the node.

Across the survey, 18 commit messages and 9 PR or review statements are corrected on the bug pages, each with the evidence. Next: where these defects actually lived, and who they reached.

Where the defects lived

of 82 production defects	n
ever on master	78
trigger reachable in merged code	44 +13
shipped in a release with IPC binaries (v30+)	43
shipped only in source releases	23
never released	16

8 hit real deployments. All are the mining interface, all driven by Stratum v2 software.

crash when a killed client's waitNext completes (3)
node stops answering IPC after 25 minutes (45)
assert on two calls to one thread (47); Ctrl-C hang (52); connect during startup (58)
still open: shutdown assert (44), memory growth (55)

Most defects shipped in some release, but what users actually hit is narrow: eight reports, all on the mining interface, all from Stratum v2 software.

The table follows the 82 production defects outward from the code. 78 were on master at some point. 44 had a trigger reachable in merged code, and 13 more only through a hidden test-only RPC or a resource running out. 43 shipped in releases with IPC binaries, which means v30 and later, 23 shipped only in source releases, and 16 were never released.

The right side is what reached real deployments. Eight defects were reported from the field, mostly relayed by plebhash, and seven of them were in released binaries. They include the crash when a killed client's waitNext completes (3), the node that stops answering IPC after 25 minutes (45), an assert on two calls to one thread (47), a hang on Ctrl-C (52) and a problem connecting during startup (58). Two are still open: the shutdown assert (44) and the memory growth (55).

The wallet and the GUI do not appear because nobody runs them over IPC outside of branches yet. Expect a new batch of bugs when they do.

When I started the survey, I had a theory about why the bugs came in the order they did.

The waves idea: the hypothesis

My starting guess: each class of bug surfaced once the previous class was handled. Tested only after every record was classified.

Wave	Class of bug	Why it appeared then
1	Disconnect handling missing	The library assumed subprocesses live as long as the process and never disconnect before shutdown
2	Unclean disconnects	Once clean disconnects worked, only clean ones were expected. Aborting C++ peers, and Rust or Python clients that skip destructors, broke that
3	Racy disconnects	With the common unclean cases handled, the races left needed new code paths and refactoring: the cancellation work, mp#240 and mp#249
4	Residual small bugs	With every disconnect path in place, the rest are one-line fixes: refcounts, lock order
4.5	Server-side disconnects	Earlier waves were clients leaving servers. Servers leaving clients was less tested until xyzconstant added tests and fixes
5	Theoretical and rare races	Found by LLM-assisted review, other reviewers and Antithesis

My starting theory: the bugs came in waves, each class exposed once the previous one was handled.

[table, top to bottom] Wave 1: the library assumed subprocesses never disconnect before shutdown. Wave 2: once clean disconnects worked, unclean ones broke things: aborting C++ peers, Rust and Python clients that skip destructors. Wave 3: the races left over needed new code paths, the cancellation work. Wave 4: after that, small fixes, a refcount here, a lock order there. Wave 4.5: servers disconnecting from clients, which was less tested until xyzconstant's tests. Wave 5: rare or theoretical races, found by assisted review and Antithesis.

I deliberately did not use this to classify anything. Every record was classified first; then two independent readings checked the theory against the data.

Here is what survived.

My starting theory was that the bugs came in waves, each class exposed once the previous one had been handled.

The table lists the waves as I first described them. In wave 1, disconnect handling was missing, because the library assumed subprocesses live as long as the process and never disconnect before shutdown. In wave 2, once clean disconnects worked, unclean ones broke things: C++ peers that abort, and Rust or Python clients that skip destructors. Wave 3 was the racy disconnects that remained once the common unclean cases were handled, and they needed new code paths and refactoring: the cancellation work in mp#240 and mp#249.

Wave 4 was what was left once every disconnect path was in place: small one-line fixes, a refcount here, a lock order there. Wave 4.5 was servers disconnecting from clients, the reverse of the earlier waves, which was less tested until xyzconstant added tests and fixes. Wave 5 was theoretical and rare races, found by LLM-assisted review, other reviewers and Antithesis.

I deliberately did not use this theory to classify anything. Every record was classified first, and only then did two independent readings check the theory against the data. Here is what survived.

The waves idea: what held, what did not

Held

allAs the order the classes were fixed: the fix dates follow the waves
2Foreign clients: the Rust client (2024-12) for 40, 41, 28, 8; Python and fuzzing (2025-09) for 42, 47, 57
3New code paths were needed: mp#160, mp#240
42026 disconnect fixes are small: lock moves (4, 64), a reorder (7), one prefetch (79)
4.5xyzconstant's July 2026 tests found 72, 73, 80 in five weeks
5Clearest in the data: of 32 records since July 2026, 20 from review, 25 reported with LLM help

Did not hold

allAs successive classes: by first report they overlap from the start
1Clean disconnects did not come first: 24, 16, 17 were earlier; the oldest (38, 39, 2020) are unclean
2Not done before wave 3: foreign-client finds kept coming (3, 42, 47, 57; 81, 82, 84)
3Not one step: fourteen months of PRs, mp#186 to mp#361
4It holds the two largest fixes: 3 (267 lines), 44 (288, pending)
5Antithesis finds were real crashes and CI failures, not theoretical

What replaces it: the exposure timeline. New workloads reached code that had been there since 2019.

The waves describe the order the bugs were fixed in, not the order they appeared.

[left column] What held: the fix dates follow the waves. Foreign clients do mark wave 2: the Rust client in December 2024, Python and fuzzing in September 2025. Wave 3 did need new code paths. The 2026 disconnect fixes are small. And wave 5 is the clearest thing in the data: of the 32 records first reported since July, 20 are review finds and 25 were reported with LLM help.

[right column] What did not: by report date the classes overlap from the start. The oldest disconnect records, from 2020, are unclean ones. Foreign-client bugs kept arriving after the race fixes began. The "small fixes" period has the two largest fixes.

[blue box] So the replacement is the exposure timeline from earlier: new workloads reaching old code.

That leaves what is still open.

The waves turned out to describe the order in which the bugs were fixed, not the order in which they appeared.

The green column is what held. The fix dates do follow the waves. Foreign clients mark wave 2: the Rust client in December 2024 led to 40, 41, 28 and 8, and Python and fuzzing in September 2025 led to 42, 47 and 57. Wave 3 did need new code paths, mp#160 and mp#240. The 2026 disconnect fixes are small: lock moves (4, 64), a reorder (7) and one prefetch (79). xyzconstant's July 2026 tests found 72, 73 and 80 in five weeks. And wave 5 is the clearest thing in the data: of the 32 records first reported since July 2026, 20 came from review and 25 were reported with LLM help.

The red column is what did not hold. By first report date, the classes overlap from the start. Clean disconnects did not come first: 24, 16 and 17 came earlier, and the oldest disconnect records, 38 and 39 from 2020, are unclean ones. Wave 2 was not done before wave 3 began, since foreign-client finds kept coming (3, 42, 47, 57; 81, 82, 84). Wave 3 was not one step but fourteen months of PRs, mp#186 to mp#361. The supposedly small wave 4 holds the two largest fixes, 3 at 267 lines and 44 at 288, still pending. And the Antithesis finds in wave 5 were real crashes and CI failures, not theoretical ones.

What replaces the waves is the exposure timeline from earlier: new workloads reaching code that had been there since 2019. That leaves what is still open.

Still open, and what would help

33 open

Fix in review: 44 (bitcoin#35932 + mp#335), 37, 77, 78 (mp#361), 75 (bitcoin#36011), 56 (bitcoin#36047), 90, 92, 103 (bitcoin#36520), 83 (bitcoin#36525), 89 (mp#368), 93 (mp#369), 16 (branch)
Undiagnosed: 55, memory growth under Stratum v2; 48, 32-bit segfault in kj::str
Private review findings: 81, 82, 84, 86, 87, reproduced by tests in September 2026; 88, 91, 94 to 102, added in October
79 is merged in mp, not yet in a subtree bump

What would help

Tests with foreign peers: abandon calls, skip destroy, two requests on one thread. 11 production defects need such a peer
Server-side disconnect tests: the July 2026 tests found three defects in five weeks
Sanitizer coverage: 15 records could only be seen with a sanitizer or Antithesis

33 bugs are open. Thirteen only need review, and most of the rest would be found sooner by test peers that misbehave.

[left column, first bullet] Thirteen have a fix waiting for review. The most important is 44, the shutdown assert with mining calls in flight, which Stratum v2 users hit.

[second bullet] 55, memory growth under Stratum v2 clients, is the one I worry about most: a field report with no diagnosis.

[third bullet] Six come from the private security reviews. Tests against master reproduced all six this month.

[right column] What would help: our C++ client is too polite. It never abandons a call, never skips destroy, and shuts down in order. Ten production defects need a peer that does not behave that way. Tests that disconnect from the server side found three in five weeks.

Which brings me to the questions I would like to discuss.

33 bugs are still open. Thirteen of them only need review, and most of the rest would be found sooner by test peers that misbehave.

Thirteen have a fix waiting for review: 44 (bitcoin#35932 and mp#335), 37, 77 and 78 (mp#361), 75 (bitcoin#36011), 56 (bitcoin#36047), 90, 92 and 103 (bitcoin#36520), 83 (bitcoin#36525), 89 (mp#368), 93 (mp#369) and 16 (on a branch). The most important is 44, the shutdown assert with mining calls in flight, which Stratum v2 users hit.

Two are undiagnosed. Bug 55, memory growth under Stratum v2 clients, is the one I worry about most, because it is a field report with no diagnosis. Bug 48 is a 32-bit segfault in kj::str. Sixteen more come from the private security reviews: tests against master reproduced 81, 82, 84, 86 and 87 in September 2026, and the eleven added in October (88, 91, 94 to 102) rest on the reviewers' own tests and analysis. Bug 79 is merged in libmultiprocess but not yet in a Bitcoin Core subtree update.

The right column is what would help. Our C++ client is too polite: it never abandons a call, never skips destroy, and shuts down in order. Eleven production defects need a peer that does not behave that way, so tests with foreign peers that abandon calls, skip destroy or send two requests on one thread would find them sooner. Tests that disconnect from the server side found three defects in five weeks. And sanitizer coverage matters, because 15 records could only be seen with a sanitizer or Antithesis.

That leads to the questions for discussion.

Discussion

Should we own more of the Cap'n Proto layer (wrap CallContext, patch capnp) instead of relying on "only on the loop thread"?
Should 55, memory growth under Stratum v2, block a release? Should mining IPC fixes be backported?
What would a misbehaving Rust or Python test client in CI look like?
How should we triage unreproduced findings from LLM-assisted and private reviews?

ipcbugs.ofsky.org · all bugs · findings · corrections · explorer
Bugs shown: 45, 3, 5, 79, 30 · tracking issue bitcoin#28722

The talk ends with four open questions for the room.

The first is about Cap'n Proto. The fix for 79 came down to two options: stop worker threads from touching connection state, or patch Cap'n Proto. Is a rule like "only on the loop thread" enough, or should the library wrap CallContext so workers cannot touch that state at all?

The second is about releases. Bug 55 is undiagnosed, and the fix for bug 3 was not backported to 30.x because the mining interface is experimental. Is that still right now that Stratum v2 deployments depend on it?

The third is about testing: could a deliberately rude Rust or Python client, or fuzzamoto, run in CI? The fourth is about triage. Review finds now dominate, and some are unreproduced. How much maintainer time should an unreproduced finding get before it has a test?

The links on the slide lead to the site, the full bug list, the findings and corrections pages, the explorer, the pages for each bug shown in the talk, and the tracking issue, bitcoin#28722.

Stability bugs in libmultiprocessand Bitcoin Core IPC