IPC stability bugs

Bug 82

Destroying an object during a running call frees it under the call

Status
Open
Symptom
Data race; worst effect seen: other
Where seen
Libmultiprocess tests (mptest)
Trigger
Several calls running at the same time
First reported
Reported privately in an independent security review, 2026-08-13
Fixed
Not yet
In released binaries
Yes (v30.0 onward)
Defect ever on master
Yes

Summary

If an IPC client calls destroy on an object from one thread while a call on the same object is still running on another server thread, the server frees the object's implementation under the running call. For BlockTemplate, destroying a template while its waitNext is outstanding makes bitcoin-node read the freed template for the rest of the wait, a use-after-free that can crash the node. The libmultiprocess C++ client cannot do this, so only a multi-threaded client written with another Cap'n Proto library (pycapnp, capnp-rust) can trigger it. It was reported privately in an independent security review. A test that forces the interleaving reproduces the use-after-free; it has not been seen in practice. The code has been this way since the library's first commit; released binaries from v30.0 on, which accept external IPC clients with -ipcbind, are affected. No fix yet.

What happened

Nothing has been observed outside a test. The race was reported privately in an independent security review of Bitcoin Core's IPC code (2026-08-13), found by reading the code. The same report described a related crash from calling a method after destroy, which is tracked separately as its own bug. No crash was demonstrated. The report's proposed fix adds a lease that makes destroy fail while calls are active.

Root cause

On the server, each object a client holds is wrapped in a ProxyServer, whose ProxyServerBase::m_impl is a std::shared_ptr<Impl> (include/mp/proxy.h) to the real implementation. ProxyMethodTraits::invoke (include/mp/proxy.h) reads m_impl.get() and calls the method on the worker thread named by the call's Context. ProxyServerBase::invokeDestroy() (include/mp/proxy-io.h) does m_impl.reset() with no lock and no count of calls in flight, on the worker named in the destroy call's own Context, which can be a different server thread.

For BlockTemplate, the ProxyServer is the only owner of BlockTemplateImpl, so the reset deletes it. The clearest case is waitNext: BlockTemplateImpl::waitNext (src/node/interfaces.cpp) passes m_block_template and m_interrupt_wait by reference to BlockTemplateManager::WaitAndCreateNewBlock (src/node/block_template_manager.cpp), which loops for up to the client's timeout reading block_template->block.hashPrevBlock and vTxFees. After the template is destroyed, each later iteration reads freed memory. If the other call is only just entering invoke, its read of the shared_ptr and the reset are an unsynchronized read and write of the same object, which is undefined behavior by itself.

A libmultiprocess C++ client sends destroy only from ~ProxyClientBase, and destroying a proxy while another thread is calling it would already be undefined behavior in the client. A pycapnp or capnp-rust client that shares a capability between tasks can do it, for example by dropping a template when a new tip arrives without calling interruptWait first. Such clients do run several server threads at once: @plebhash described, in an issue about concurrent requests to one thread, a Rust client that calls submitSolution, getHeader, getCoinbaseTx, getCoinbaseMerklePath and waitNext on separate threads.

Fix

No fix yet; tracked in mp#378 (opened 2026-10-15), which covers both destroy bugs. The private report proposes a lease on m_impl that makes destroy fail while calls are active.

Reproduction

The test adds a makeCallback method to the test interface in mptest, the libmultiprocess unit test program. It returns a FooCallback object that the server owns outright, as it owns a BlockTemplate. One client thread calls call on it, and a hook holds that call inside the implementation. The test then creates a second server thread with makeThread and sends destroy on it as a raw Cap'n Proto request. Once destroy returns, the hook releases the first call. It was run on 2026-09-26 against Bitcoin Core master bfdcd97, which carries libmultiprocess v13 in its subtree.

Patch: mptest-repro.patch. It applies to Bitcoin Core master; last checked against badc957 on 2026-10-15. From a Bitcoin Core checkout:

curl -sO https://ipcbugs.ofsky.org/repro/mptest-repro.patch
git apply mptest-repro.patch
cmake --build build --target mptest
build/src/ipc/libmultiprocess/test/mptest -f test.cpp:662

destroy returned while the call was still running, and the call then read the freed object. In a build configured with -DSANITIZERS=address,undefined:

destroy returned while call was still running
ERROR: AddressSanitizer: heap-use-after-free ... READ of size 4 ... thread T4
    #0 ... in mp::test::OwnedCallback::call(int) foo.h:72:16
freed by thread T5 ... in mp::ProxyServerBase<...FooCallback...>::invokeDestroy() proxy-io.h:674

Without a sanitizer the call returned -258435454 instead of 102. The hook makes the result the same on every run. In real use the timing decides; with a blocking method such as waitNext the window is the whole wait. The test patch is available on request and will be included in the GitHub issue for this bug when it is opened.

Next steps

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern Serving before ready, freeing before done: Freeing state while work still uses it

  • Prevented by a clear ownership model, a design rule
  • Earliest detector a test with a non-C++ client (did not exist then)
  • Diagnosis easy
  • Fix design moderate
  • Debugging aids sanitizer output, a hook to force the timing
  • Rust yes

Would Rust have prevented it? Yes. Resetting m_impl while another thread calls into it is an unsynchronized write to shared state, so the impl would sit behind a Mutex or each call would hold an Arc clone; the Arc version turns the use-after-free into a lifetime extension where waitNext keeps running on an object destroy() no longer frees, up to the client's timeout, and capnproto-rust's single-threaded servers would need an explicit hand-off to a worker anyway. More on Rust

Could it have been prevented?

Yes, by an ownership model in which a running call keeps the implementation alive. invokeDestroy resets the shared_ptr<Impl> with no lock and no count of calls in flight, on whichever worker thread the destroy request names, while ProxyMethodTraits::invoke on another worker reads the same pointer and runs the method through it. Either a per-call strong reference taken under a lock (so destroy only drops the server's reference and the object dies when the last call returns) or the in-flight lease the private report proposes (so destroy fails while calls are active) makes the interleaving harmless. The rule behind both, that a remote peer may release an object at any moment and release must therefore wait for or exclude in-flight work, was knowable when bitcoin#30510 exposed BlockTemplate.destroy to -ipcbind clients in 2024; the code is unchanged since the initial commit (06e1045baba0).

Could it have been found sooner?

Only by a client that is not the library's own. The C++ client sends destroy from the proxy destructor, and destroying a proxy while another thread calls it is already undefined on the client side, so no C++ test can reach this. A pycapnp or capnp-rust client sharing one capability between tasks reaches it naturally, and Rust Stratum v2 software already runs several server threads at once (bitcoin#33923), which is why the waitNext case matters: the window is the whole wait. The survey's 2026-09-26 mptest case, with a hook holding one call while destroy runs on a second server thread, gets an ASan heap-use-after-free every time.

What made it hard to debug?

Reproducing it, not reading it. The independent security review described the mechanism from the code without a demonstration, and without a hook the race depends on destroy landing during a running call, which in the field would show as a garbage result or a crash somewhere inside WaitAndCreateNewBlock with no obvious link to a destroy on another thread. The deterministic hook in the test and ASan's "freed by thread T5 in invokeDestroy" line are what make the cause legible; a node-side log of destroy requests would be the equivalent aid in production. The fix design is the open part: lease versus per-call reference, and how destroy should answer while calls run.

Lesson

When a remote peer can release an object, the release must either wait for calls already running on it or be refused while they run; resetting the only owner on one thread while another thread is inside a method is a use-after-free that only the peer's timing decides. Test with a client that can interleave calls on one object, because a client that serializes them will never show this.

Notes

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom data-race
Worst effect observed other
Seen in mptest
Scope production
Trigger Concurrent calls (concurrent-calls)
Phase steady-state
Side server
Component proxy-io
Root cause kind lifetime
Fix kind none
Detection review
Needs a peer that is not libmultiprocess C++ yes
Needed a sanitizer to see no
Defect ever on master yes
Released binary
Confidence medium