IPC stability bugs

Bug 56

Mining RPCs report an internal bug, or abort, if the node shuts down mid-call

Status
Open, fix under review
Symptom
Assertion failure; worst effect seen: assertion failure
Where seen
Bitcoin Core functional tests
Trigger
Shutdown
First reported
2026-01-12 (first report)
Fixed
Not yet
Fix
bitcoin#36047
In released binaries
Yes (v30.0-v31.1)
Defect ever on master
Yes

Summary

The getblocktemplate and generateblock RPCs (and the generateBlocks helper behind generatetoaddress) fail a CHECK_NONFATAL(block_template) check when the node starts shutting down while they are running. Since 2025, Mining::createNewBlock() returns null when the node shuts down while it waits, so that IPC mining clients never receive a template built on a missing chain tip. These three RPC call sites still assert that it cannot return null. In release builds the caller gets "Internal bug detected: block_template" instead of "Shutting down"; in builds that abort on failed assumptions (such as the Antithesis runs that found it) the node aborts. Released binaries v30.0 through v31.1 have the bug. The proposed fix is bitcoin#36047, not yet merged.

What happened

Root cause

5315278e7c7f in bitcoin#31785 ("Have createNewBlock() wait for tip, make rpc handle shutdown during long poll and wait methods") made createNewBlock() wait for a tip and return nullptr if shutdown interrupts the wait, "instead of the last tip", so that IPC clients never see a template built on a missing tip. It updated the long-poll and wait paths of the RPCs, but left the three CHECK_NONFATAL(block_template) checks in src/rpc/mining.cpp (in generateBlocks, generateblock and getblocktemplate), which had been an unreachable invariant since bitcoin#30440 ("Have createNewBlock return BlockTemplate interface") added them.

In the long-poll case the sequence is:

  1. Interrupt() (src/init.cpp) notifies m_tip_block_cv before it calls InterruptRPC().
  2. The long-poll loop in getblocktemplate wakes with no new tip, breaks out, and passes its IsRPCRunning() check because RPC has not been interrupted yet.
  3. It calls miner.createNewBlock(), which sees m_interrupt set and returns null, and CHECK_NONFATAL(block_template) fails.

Fix

No fix is merged yet. bitcoin#36047 ("rpc: handle createNewBlock() returning nullptr during shutdown", by @chriszeng1010, open) replaces the three checks in its fix commit b1fa37ea8681 (+6/-3). getblocktemplate and generateblock throw JSONRPCError(RPC_CLIENT_NOT_CONNECTED, "Shutting down"), matching the existing long-poll path, and generateBlocks stops its loop and returns the blocks mined so far. The commit message mentions only the first two. A new long-poll shutdown case in mining_getblocktemplate_longpoll.py fails only some of the time on a build without the fix, since the race is timing-dependent.

Two earlier PRs that tried to fix this issue, bitcoin#34357 and bitcoin#35184, were closed because, as a comment on the issue put it, they did not fix the problem.

Next steps

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern A contract changed and a caller was left behind: A function's contract changed under its callers

  • Prevented by a review checklist item, the type system
  • Earliest detector code review (existed then)
  • Diagnosis moderate
  • Fix design easy
  • Debugging aids logging, a reproduction script
  • Rust partly

Would Rust have prevented it? Partly. Changing the return from Box<BlockTemplate> to Option<Box<BlockTemplate>> would fail to compile at every unvisited caller, which is the type-driven audit the lesson asks for; but the C++ type was already a nullable unique_ptr, and a port that started from Option with expect() at those three sites would panic the same way. More on Rust

Could it have been prevented?

Yes. bitcoin#31785 changed the contract of createNewBlock() from "always returns a template" to "returns null if shutdown interrupts the wait", for the benefit of IPC clients, and updated the long-poll and wait paths it had in mind. Three other callers in src/rpc/mining.cpp still carried CHECK_NONFATAL(block_template), added in bitcoin#30440 when null was impossible. The checklist item is mechanical: when a function gains a new return value, visit every caller, and treat an existing assertion on the old contract as a caller that needs the visit most, since it was written to document exactly the assumption being removed. A return type that cannot be ignored (an std::expected carrying the shutdown reason, or a distinct "interrupted" result) would have turned the missed callers into compile errors; a nullable pointer that used to be non-null does not.

Could it have been found sooner?

By review of the PR that changed the contract, since grep createNewBlock finds the three sites. At runtime the window is small: Interrupt() wakes m_tip_block_cv before it calls InterruptRPC(), so a long-poll getblocktemplate has to wake, pass IsRPCRunning(), and call createNewBlock() in that gap. No functional test stopped the node during a long poll, and the fix's new test fails only some of the time on an unfixed build. Antithesis, which explores shutdown windows systematically, found it nine months after the merge; on a release build the effect is a misleading "Internal bug detected" RPC reply, so no user was likely to report it.

What made it hard to debug?

The Antithesis logs named the failing check, but explaining why the RPC's own shutdown guard did not fire took a reading of the order of operations in Interrupt(), and two earlier fix PRs (bitcoin#34357, bitcoin#35184) were closed as not fixing the problem before chriszeng1010 reproduced it by hand and walked through the race. The by-hand reproduction (long-poll in one terminal, stop in another, about one failure in two) is what made the fix reviewable. The severity was also easy to misread: release builds only return a wrong error, but builds that abort on failed CHECK_NONFATAL, including the Antithesis ones, abort the node.

Lesson

When you widen what a function can return, the assertions in its callers are the list of places that believed the old contract; update them in the same change, or use a return type that makes ignoring the new case a compile error.

Notes

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom assert
Worst effect observed assert
Seen in functional-test
Scope production
Trigger Shutdown (shutdown)
Phase shutdown
Side server
Component bitcoin-rpc
Root cause kind assumption-violated
Fix kind local-fix
Detection antithesis
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released binary
Confidence medium