Bug 106

Node does not exit after shutdown when an IPC client reconnects

Status

Open

Symptom

Hang; worst effect seen: hang

Where seen

A production node

Trigger

Shutdown

Introduced

7f65aac78b95, 2025-04-24

First reported

2026-10-20 (first report)

Fixed

Not yet

Tracked in

mp#385

In released binaries

Yes (v30.0 onward)

Defect ever on master

Yes

Summary

When bitcoin-node shuts down, it disconnects its -ipcbind clients but leaves its IPC listeners accepting. A client that connects again in that window, as a mining client does when it loses its connection, keeps the IPC event loop running, and the process then waits for that client forever: it logs "Shutdown done" and never exits. A v32.0rc3 node with a mining client connected appears to have hit this after a fatal disk-full error, reported in bitcoin#36585, and stayed down for 5.7 hours until it was killed by hand. That report lists only thread names, so the cause of that particular hang is not confirmed, but the code allows it and an independent security review reproduced the reconnect case (audit reports AR-21 and AR-43). Releases from v30.0 on are affected. No fix yet.

What happened

On 2026-10-20 mweinberg reported, in bitcoin#36585, a pruned mainnet node running v32.0rc3 whose chainstate compaction filled the disk. The node hit a fatal error, ran its shutdown sequence, and logged "Shutdown done", but the process did not exit. With the process still alive, the service manager never restarted it, so a short outage became a 5.7-hour one. When the process was finally killed, the threads left were the main thread (named b-shutoff), two b-capnp-loop threads and one named bitcoin-node. The node had a mining client connected over IPC. The report is mainly about the compaction; it flags the hang as possibly a separate issue.

The same day, an independent security review linked the issue in comments on two of its earlier reports, one about stopping listeners (AR-21) and one about shutdown (AR-43). The comments say it reproduced the reconnect case with a test and with a full node that accepted a reconnect after "Shutdown done" and stayed alive until the socket closed, and that a normal shutdown with one active waitNext call completed promptly. Those runs have not been repeated for this page.

Root cause

Shutdown() calls disconnectIncoming(), which closes the current incoming connections and returns. Nothing stops the listeners: CapnpProtocol::listen() calls ListenConnections(), which returns nothing, and _Listen() keeps an accept pending on the event loop. A connection accepted after that point is served normally, and each Connection holds an EventLoopRef, a reference that keeps the event loop running. When the node's context is destroyed after "Shutdown done", ~CapnpProtocol releases its own reference and joins the loop thread, which only stops when no references remain. The process then waits for the new client to disconnect. A mining client that reconnects when its connection drops and then stays connected never does.

Signals do not help at that point, because the node has already finished its shutdown sequence; in the report, SIGTERM had no effect and systemd had to send SIGKILL.

The window comes from bitcoin#32345 ("ipc: Handle unclean shutdowns better", merged 2025-08-18, first released in v30.0). Its commit 7f65aac78b95 ("ipc: Avoid waiting for clients to disconnect when shutting down") made shutdown disconnect clients instead of waiting for every one of them, but did not stop new ones from connecting. bitcoin#35932, which makes disconnectIncoming() wait for calls in progress, lists new connections during shutdown as a known limitation.

The code

Quoted from Bitcoin Core master aa76bd2. Shutdown() in src/init.cpp disconnects the current clients:

// If any -ipcbind clients are still connected, disconnect them now so they
// do not block shutdown.
if (interfaces::Ipc* ipc = node.init->ipc()) {
    ipc->disconnectIncoming();
}

CapnpProtocol::listen() in src/ipc/capnp/protocol.cpp starts a listener and keeps no handle to it, so nothing can stop it later:

void listen(mp::SocketId listen_fd, interfaces::Init& init, std::optional<size_t> max_connections) override
{
    startLoop();
    if (::listen(listen_fd, /*backlog=*/5) != 0) {
        throw std::system_error(errno, std::system_category());
    }
    mp::ListenConnections<messages::Init>(*m_loop, listen_fd, init, max_connections);
}

and its destructor waits for the event loop thread, which runs while any connection holds a reference:

~CapnpProtocol() noexcept(true)
{
    m_loop_ref.reset();
    if (m_loop_thread.joinable()) m_loop_thread.join();
    assert(!m_loop);
};

Fix

No fix merged yet; tracked in mp#385 (opened 2026-10-28). Two parts are needed, and both have been written, as changes proposed with two private reports from an independent security review that are not yet public PRs:

• libmultiprocess: a way to stop a listener. The change proposed with AR-21 makes ListenConnections() return the listener and adds StopListening(), which prevents further accepts: a pending accept that completes afterwards is not served, and disconnect callbacks no longer resume accepting at the connection limit. It does not close the listening socket. This is the approach suggested in the review of mp#269.
• Bitcoin Core: stop the listeners at shutdown. The change proposed with AR-43, built on the first, keeps the listeners CapnpProtocol::listen() creates and stops them at the start of disconnectIncoming() and in ~CapnpProtocol. It also moves disconnectIncoming() to the start of Shutdown() (a related gap). A revision on 2026-10-30 dropped its own wait for calls in progress, leaving the in-progress call crash to bitcoin#35932, and added a functional test that pauses shutdown right after disconnectIncoming(): without the change, a new bitcoin-cli call over IPC still connects and gets an answer; with it, the call is refused. The test does not check that such a client keeps the process alive.

Next steps

• Confirm the rest of the mechanism with a functional test: the AR-43 test shows a client can still connect after the disconnect; also keep that client connected and check that the process does not exit until it leaves (and does exit with the fix). ryanofsky asked the reporter whether the mining client's log shows a reconnect, and for a backtrace if it happens again.
• Bring the two proposed changes upstream: a libmultiprocess PR with the stop API from AR-21 and its test, coordinated with mp#384, which changes the same accept loop; then a Bitcoin Core PR with AR-43's listener handling and a test that reconnects after the disconnect.

Timeline

• 2025-08-18: bitcoin#32345 merged: shutdown disconnects -ipcbind clients instead of waiting for them, leaving the listeners open (first released in v30.0).
• 2026-08-16: an independent security review reports that a listener cannot be stopped (AR-21), then as an API gap with no effect on Bitcoin Core.
• 2026-10-20: a v32.0rc3 node stays alive after "Shutdown done" with a mining client connected, reported in bitcoin#36585; the review links it to the open listener the same day.
• 2026-10-28: mp#385 opened to track the fix.
• 2026-10-30: the proposed Bitcoin Core change (AR-43) is revised to stop the listeners only, with a test showing a client connecting after the disconnect.
• Releases: v30.0 and later have the bug.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.