IPC stability bugs

Bug 107

IPC stays open for most of shutdown

Status
Open
Symptom
Data race; worst effect seen: other
Where seen
Bitcoin Core functional tests
Trigger
Shutdown
Introduced
955d4077aac6, 2024-09-06
First reported
Reported privately in an independent security review (audit report AR-43), 2026-09-26
Fixed
Not yet
Tracked in
Not tracked on GitHub yet
In released binaries
Yes (v30.0 onward)
Defect ever on master
Yes

Summary

When bitcoin-node shuts down, it stops HTTP RPC first, before it tears anything down, but it keeps serving IPC clients until almost the end. In between, it shuts down the P2P layer and resets its objects, flushes and dumps state, and stops and deletes the indexes. IPC clients can keep making calls the whole time, including RPC calls over IPC, which HTTP clients can no longer make. Code in that stretch assumes nothing else can still reach the objects it deletes, so an IPC call made then can race with the teardown. This is less a single bug than a window that makes the other IPC shutdown bugs easier to hit. The fix is to treat IPC like HTTP: stop accepting IPC calls and disconnect clients at the start of shutdown. It was reported privately in an independent security review (audit report AR-43), whose test got an RPC answer over IPC after HTTP RPC had stopped. No crash has been shown. Release binaries from v30.0 on are affected, and RPC over IPC from v32.0. No fix is merged.

What happened

An independent security review reported on 2026-09-26 (AR-43, which records the report as it was first written and later revised) that IPC clients stay connected during node teardown. Its functional test kept two IPC clients connected, called stop through one, and still got an answer to gettxout through the other after the node logged that its HTTP RPC server had stopped. The report rated it as hardening. It was later rewritten around calls already running at shutdown, and kept the change described below.

Root cause

Shutdown() in src/init.cpp begins by stopping HTTP RPC, REST and the HTTP server, which waits for HTTP worker threads to finish. After that it resets the P2P objects, under the comment "After the threads that potentially access these pointers have been stopped, destruct and reset all to nullptr" (init.cpp), and later stops and deletes the indexes. That assumption holds for HTTP worker threads, but not for libmultiprocess worker threads, which serve IPC calls until disconnectIncoming() runs near the end.

RPC over IPC makes this easy to reach. Rpc.executeRpc dispatches through CRPCTable::execute(), which checks for warmup but not whether RPC has been stopped. An RPC that reads node.connman (through EnsureConnman()) or g_txindex while the shutdown thread resets it is a data race, with no lock on either side. Mining calls are exposed the same way, though the state they use most, the chainstate and mempool, is reset only after the disconnect.

There are three IPC shutdown problems with overlapping fixes. This one is about when IPC stops admitting calls. Calls already running when IPC is disconnected need to be waited for. New connections need the listeners stopped. Disconnecting earlier makes the first window small; the other two are needed for it to close fully.

IPC has stayed open through shutdown since bitcoin-node gained the -ipcbind option in bitcoin#30509 (2024, first released in v29.0). Until bitcoin#32345 (2025, v30.0), shutdown waited for every IPC client to leave after tearing everything down. bitcoin#32345 added disconnectIncoming() near the end of Shutdown(), where the shutdown of IPC clients was then needed, rather than next to the HTTP shutdown. RPC over IPC, added in bitcoin#32297 (first released in v32.0), made the window reachable by any RPC.

Fix

No fix merged. The change proposed with AR-43 moves disconnectIncoming() to the start of Shutdown(), before the HTTP shutdown, and stops the IPC listeners there so no new client can connect. Its 2026-10-30 revision adds a functional test that pauses shutdown right after the disconnect: without the change, a new bitcoin-cli call over IPC gets an answer after HTTP RPC has stopped; with it, the call is refused. An earlier version also waited for calls already running; the revision leaves that to bitcoin#35932, the open fix for calls already running, which leaves disconnectIncoming() where it is, so it fixes that crash but not this window.

Next steps

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern Serving before ready, freeing before done: Freeing state while work still uses it

  • Prevented by a design rule
  • Earliest detector shutdown test (did not exist then)
  • Diagnosis easy
  • Fix design easy
  • Debugging aids logging
  • Rust no

Would Rust have prevented it? No. Which requests a server still admits during shutdown is an ordering decision; Rust would make the unsynchronized pointer reset a compile error, but not the late disconnect itself. More on Rust

Could it have been prevented?

Yes, with one rule for every way into the node: stop admitting requests from all of them at the same point, before tearing anything down. Shutdown() followed it for HTTP and then relied on it ("after the threads that potentially access these pointers have been stopped"), but IPC was added later with its own, later disconnect.

Could it have been found sooner?

By a shutdown test that keeps an IPC client making calls while the node stops, which is what the report that found it did. Shutdown tests used clients that were idle or gone.

What made it hard to debug?

Nothing has failed yet. A failure would look like a crash in an RPC or index during shutdown, far from the IPC code that let the call in.

Lesson

When a server has several front doors, close them all at once at the start of shutdown; a second door added later tends to get its own, later closing time, and teardown code written for the first door's guarantee no longer holds.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom data-race
Worst effect observed other
Seen in functional-test
Scope production
Trigger Shutdown (shutdown)
Phase shutdown
Side server
Component bitcoin-init
Root cause kind ordering
Fix kind none
Detection review
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released binary
Confidence low