IPC stability bugs

Bug 75

Node spawned by the GUI never exits after shutdown because disconnected callbacks stay alive

Status
Open, fix under review
Symptom
Hang; worst effect seen: hang
Where seen
Bitcoin Core functional tests
Trigger
A peer disconnects cleanly
First reported
2026-08-14 (first report)
Fixed
Not yet
Fix
bitcoin#36011
In released binaries
No
Defect ever on master
Yes

Summary

When bitcoin-gui spawned bitcoin-node and was told to stop, bitcoin-node finished its shutdown and then never exited, and bitcoin-gui waited for it forever. The GUI registers callbacks with the node over IPC, and the node wraps each one in an object that keeps the node's IPC event loop running while it exists. Those wrappers stayed alive after the GUI disconnected them, because btcsignals (Bitcoin Core's replacement for boost::signals2) did not destroy a disconnected callback. The btcsignals behavior is on master, but only the multiprocess GUI in the unmerged bitcoin#10102 registers signal callbacks over IPC, so no released binary is affected. The proposed fix is bitcoin#36011, not yet merged.

What happened

Root cause

The eleven leaked references belonged to the node-side callback wrappers: ProxyClient<InitMessageCallback>, NotifyBlockTipCallback and the other handle* callbacks the GUI registers. A callback registered over IPC is a capability pointing back into the GUI process, so on the node side it is a ProxyClient, and each one holds an EventLoopRef through its proxy context. Each wrapper lives in a std::function stored in a btcsignals signal such as uiInterface.*.

The GUI's shutdown was clean. ClientModel is deleted and the Handler.destroy() calls run, so every node-side HandlerImpl is destroyed and its scoped_connection disconnected. But btcsignals::connection::disconnect() (src/util/btcsignals.h) only marked the slot as disconnected. The callback object stayed in the signal's connection_holder until two things happened: a later connect() on the same signal garbage-collected it, and every connection handle referencing it was released. After shutdown nothing connects to those signals again. So the wrappers and their EventLoopRefs lived forever, EventLoop::done() never became true, CapnpProtocol::serve() never returned, and bitcoin-node never exited. The GUI's waitSpawned cleanup (mp::WaitProcess in src/ipc/interfaces.cpp) blocked in waitpid. boost::signals2, which btcsignals replaced in bitcoin#34495 ("Replace boost signals with minimal compatible implementation"), destroys a disconnected slot's function object as soon as no emission is running it.

The hang is deterministic once the test runs. The stop RPC with wait=0 finishes node shutdown in about 100 ms, before the GUI's 200 ms shutdown-poll timer starts its teardown, so no signal is touched again after the handlers disconnect.

Fix

No fix is merged yet. bitcoin#36011 ("btcsignals: destroy callbacks on disconnect", open, by ryanofsky with Claude) restores the boost::signals2 behavior. Its fix commit c735a8ec4ac7 gives connection::liveness an atomic shared_ptr<const void> m_owner that owns the callback. disconnect() resets it, destroying the callback at once. Each emission holds m_owner for the duration of the call, so a callback disconnected during a concurrent emission is destroyed when that call returns. a16a941b429e first adds tests that describe the old destruction timing, which the fix commit updates. 6a3d490d826a adds a fallback for standard libraries without __cpp_lib_atomic_shared_ptr. @maflcko agreed it can wait for a later release since nothing released is affected.

Next steps

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern An object outliving what it points to: A child object outlives its parent

  • Prevented by a written spec or documentation, a review checklist item
  • Earliest detector a unit test (did not exist then)
  • Diagnosis moderate
  • Fix design moderate
  • Debugging aids a reproduction script, logging
  • Rust no

Would Rust have prevented it? No. When a disconnected slot's closure is dropped is a policy of the signal registry; a Vec of boxed closures that defers cleanup until the next connect compiles fine in Rust and holds its EventLoopRef just as long. More on Rust

Could it have been prevented?

Yes. btcsignals replaced boost::signals2 as a "minimal compatible implementation" (bitcoin#34495), and compatibility was judged on call semantics: which slots run, in what order, what disconnect does to future emissions. One property was left out: boost::signals2 destroys a disconnected slot's function object as soon as no emission is running it, while btcsignals kept it until a later connect() on the same signal garbage-collected it. For in-process callbacks that is invisible; for a callback that is an IPC proxy holding an EventLoopRef, destruction timing is the difference between a node that exits and one that spins forever. The fix's first commit adds tests describing the old timing before changing it, which is the spec that was missing: when replacing a library, list the lifetime guarantees callers depend on, not only the observable calls, and a review item for any callback registry is "when is the callable destroyed after disconnect".

Could it have been found sooner?

Yes, by a unit test on btcsignals that connects a callable owning a counted resource, disconnects, and asserts the resource is released; the fix adds one. Nothing on master could trigger the hang, because only the unmerged GUI and Chain interfaces register callbacks over IPC, so the defect sat for four months until the first push of interface_gui.py on the bitcoin#10102 branch, where it failed on every run of the two jobs that use bitcoin -m. Once a test reached it, detection was immediate and deterministic.

What made it hard to debug?

The first suspect was wrong: the same push updated the libmultiprocess subtree, so the hang looked like a library regression. What settled it was gdb on both processes, bitcoin-gui in waitpid() and bitcoin-node in EventLoop::loop() with m_num_refs = 11 and no connections, then identifying the eleven holders as the node-side callback wrappers and tracing the GUI's teardown to confirm it was clean. The node log gave no hint beyond "Shutdown done" followed by silence. A diagnostic the loop could print when asked to exit with references outstanding (the count, and which objects hold them) would have named the wrappers without a debugger; the 2400 s CI timeout with no trace is the same blind spot every hang in this ledger shows.

Lesson

A drop-in replacement must preserve when things are destroyed, not only what they do, because callers can hold resources in the objects it manages; and a refcounted loop that "exits when the last reference drops" needs a way to say who still holds one.

Notes

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom hang
Worst effect observed hang
Seen in functional-test
Scope production
Trigger Clean disconnect (clean-disconnect)
Phase shutdown
Side server
Component bitcoin-signals
Root cause kind refcount
Fix kind local-fix
Detection new-test
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released no
Confidence high