IPC stability bugs

Bug 58

Mining client connecting during node startup crashes the node

Status
Fixed
Symptom
Assertion failure; worst effect seen: assertion failure
Where seen
A production node
Trigger
Node startup
First reported
2025-12-02 (first report)
Fixed
2026-02-27
Fix
bitcoin#34661
In released binaries
Yes (v30.0-v30.3)
Defect ever on master
Yes

Summary

bitcoin-node aborted at startup with chainman: Assertion `m_node.chainman' failed when an IPC mining client connected and called mining methods before the node had loaded its chainstate (the block index and UTXO set). The node starts listening on -ipcbind sockets before it creates the ChainstateManager, so a Stratum v2 client restarted together with the node, or started before it, could crash it. Released v30.0 through v30.3 bitcoin-node binaries are affected, as are v29.0 source builds with IPC enabled; fixed by bitcoin#34661 ("ipc mining: Prevent Assertion `m_node.chainman' failed errors on early startup"), which is in v31.0.

What happened

Root cause

AppInitMain in src/init.cpp calls ipc->listenAddress() for each -ipcbind address early, before InitAndLoadChainstate constructs the ChainstateManager and sets node.chainman. An external client can therefore call Init.makeMining and then Mining.createNewBlock, waitTipChanged or checkBlock while m_node.chainman is still null, and MinerImpl::chainman() in src/node/interfaces.cpp is *Assert(m_node.chainman). In-process callers never hit this, because they are created after loading.

The code

Quoted from a7cabf92e4de^. AppInitMain in src/init.cpp starts listening on each -ipcbind address before it loads the chainstate:

if (interfaces::Ipc* ipc = node.init->ipc()) {
    for (std::string address : gArgs.GetArgs("-ipcbind")) {
        try {
            ipc->listenAddress(address);
// ...
// Chainstate initialization and loading may be retried once with reindexing by GUI users
auto [status, error] = InitAndLoadChainstate(

InitAndLoadChainstate is what assigns node.chainman. Meanwhile a client's Init.makeMining call (src/init/bitcoin-node.cpp) hands out a MinerImpl at once:

std::unique_ptr<interfaces::Mining> makeMining() override { return interfaces::MakeMining(m_node); }

and every MinerImpl method that touches the chain goes through an accessor in src/node/interfaces.cpp that asserts it exists:

ChainstateManager& chainman() { return *Assert(m_node.chainman); }

A client that connects while the node is still starting, for example one restarted together with the node, calls makeMining and then createNewBlock, m_node.chainman is still null, and the Assert aborts bitcoin-node.

The fix makes MakeMining wait until the chainstate is loaded (bbc8f1e0a7e5). The preceding commit a7cabf92e4de creates node.notifications once, before listenAddress, and sets the chainstate_loaded flag it waits on; that commit, and the wait_loaded=false for the node's internal node.mining, are omitted:

@@ -1017,5 +1017,17 @@ public:
-std::unique_ptr<Mining> MakeMining(node::NodeContext& context) { return std::make_unique<node::MinerImpl>(context); }
+std::unique_ptr<Mining> MakeMining(node::NodeContext& context, bool wait_loaded)
+{
+    if (wait_loaded) {
+        node::KernelNotifications& kernel_notifications(*Assert(context.notifications));
+        util::SignalInterrupt& interrupt(*Assert(context.shutdown_signal));
+        WAIT_LOCK(kernel_notifications.m_tip_block_mutex, lock);
+        kernel_notifications.m_tip_block_cv.wait(lock, [&]() EXCLUSIVE_LOCKS_REQUIRED(kernel_notifications.m_tip_block_mutex) {
+            return kernel_notifications.m_state.chainstate_loaded || interrupt;
+        });
+        if (interrupt) return nullptr;
+    }
+    return std::make_unique<node::MinerImpl>(context);
+}

An IPC client now cannot get a Mining object until node.chainman has been created and loaded, so none of its methods can reach the null chainman; its makeMining call just blocks on its worker thread, or returns null if the node shuts down first.

Fix

In review ryanofsky said a blocking makeMining is "not ideal" but "probably the simplest way of preventing current crashes" (comment), later called wait_loaded a hack, and posted a diff that instead makes each mining method safe during startup (comment); @Sjors had suggested a blocking ready() method (comment). No follow-up has landed, and wait_loaded is still in src/interfaces/mining.h.

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern Serving before ready, freeing before done: Serving calls before startup finishes

  • Prevented by a design rule, a review checklist item
  • Earliest detector a stress test (did not exist then)
  • Diagnosis moderate
  • Fix design moderate
  • Debugging aids a clear assertion message, logging
  • Rust no

Would Rust have prevented it? No. The chainman Assert already turned the early call into a clean abort; an Option unwrap would panic the same way, and the listener-before-load ordering is a lifecycle bug. More on Rust

Could it have been prevented?

Yes. The rule is that a process does not accept requests it cannot yet serve: either the listener opens after the state its interfaces depend on exists, or every method reachable from the socket is written to be valid at every point after the listener opens. -ipcbind placed listenAddress() early in AppInitMain, before InitAndLoadChainstate, and the Mining methods were written for in-process callers created after loading, so MinerImpl::chainman() could keep its Assert(m_node.chainman). Each half was reasonable alone; together they made an abort reachable by any client that connects in the first seconds. The checklist item is the same as for bitcoin#33222: for each method exposed over IPC, list the asserts it can reach, and ask whether each holds at every time a client can call it, not only from the times in-process code calls it. That was knowable when Mining went over IPC in 2024; the reviewers' later preference for making each method startup-safe rather than blocking makeMining shows the rule was recognized as the real fix.

Could it have been found sooner?

Yes, by a test that hammers the startup window, connecting from the moment the socket appears, which is what the fix's run_early_startup_test does with a 5 ms reconnect loop. No such test existed; the functional tests connect after wait_for_rpc_connection, so the window was never exercised, and no C++ client in the tree started before the node. Antithesis found the sibling defect in this window (createNewBlock before the first tip) a month later, which suggests it would have found this one too had it been running against v30. In practice the window was reached by production supervisors (docker compose, the Sv2 Wizard) that restart node and client together, about two months after v30.0 shipped.

What made it hard to debug?

The assert itself names the null member, so the mechanism was clear once someone with the code in mind read a log. The delay was elsewhere: the first report was a relayed Discord message on a feature-request thread (bitcoin#33994) that also carried a different abort (the m_loop assert fixed by mp#240), and it took a second report with a full log in February, showing the abort immediately after "Listening for IPC requests", for ryanofsky to place it in the startup sequence. The fix then needed a small init refactor so a condition variable existed to wait on. Log lines that put the listen call and chainstate load in order were the aid; a startup-phase check on the mining methods would have turned the abort into an error the client could report.

Lesson

Opening a listening socket is a promise that every request on it can be answered; either open it last, or make each handler tolerate the state the process is in when the socket opens. Test by connecting the instant the socket exists, because production supervisors will.

Notes

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom assert
Worst effect observed assert
Seen in production-node
Scope production
Trigger Node startup (node-startup)
Phase startup
Side server
Component bitcoin-init
Root cause kind ordering
Fix kind local-fix
Detection user-report
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released binary
Confidence high