IPC stability bugs

Bug 99

One mining interrupt can leave a second wait blocked

Status
Open
Symptom
Hang; worst effect seen: hang
Where seen
Bitcoin Core functional tests
Trigger
Several calls running at the same time
First reported
Reported privately in an independent security review, 2026-09-27
Fixed
Not yet
In released binaries
Yes (v31.0 onward)
Defect ever on master
Yes

Summary

A mining client can cancel its long waits with Mining.interrupt() (for createNewBlock and waitTipChanged) or BlockTemplate.interruptWait() (for waitNext). Each object keeps a single flag for this, and the first wait to wake clears it. If a client has two waits running on the same object from two threads, one interrupt releases only one of them; the other stays blocked until the next block, its timeout, or another interrupt. The node is not affected, but the client can stall. It was reported privately in an independent security review, with a functional test. No fix is merged yet.

What happened

Nothing has been seen in a deployment. An independent security review reported it privately on 2026-09-27. Its functional test uses a Python client with two server worker threads, starts two waits on one object (two waitTipChanged calls, two waitNext calls, or createNewBlock with waitTipChanged), sends one interrupt, and checks that both return. The report says only one did.

Root cause

Both interrupt methods call BlockTemplateManager::InterruptWait() in src/node/block_template_manager.cpp, which sets a flag (MinerImpl::m_interrupt_mining or BlockTemplateImpl::m_interrupt_wait) under the tip mutex and wakes every thread waiting on the tip condition variable. Each waiting function stops when the flag is set, then clears it and returns:

return tip_changed || m_chainman.m_interrupt || interrupt_wait;
});
if (interrupt_wait) {
    interrupt_wait = false;
    return nullptr;
}

The flag was written for one wait at a time per object. With two, both threads wake, but they take the mutex one after the other: the first sees the flag, clears it and returns; the second sees it cleared, finds nothing else has changed, and waits again. waitTipChanged has no timeout by default, so on mainnet it then waits for the next block. Shutdown is not affected, because the node's own shutdown flag releases every wait.

A side effect of the same design: an interrupt sent while nothing is waiting leaves the flag set, so the next wait returns at once. The interface documentation says the methods interrupt "the current wait".

Two waits on one object need a client that calls from two threads, which libmultiprocess allows and any client, including a C++ one, can do.

Fix

No fix merged yet. The fix proposed with the report replaces each flag with a generation number. Every wait records the number when it starts, an interrupt increases it and wakes all waiters, and each wait whose recorded number is older returns. One pending interrupt is kept for the next wait if nothing is waiting, as today. createNewBlock keeps one record across its stages. A simpler counter that is never cleared would also work; whether an interrupt sent before a wait should cancel it is a separate decision.

Next steps

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern Locks, waits and wakeups: Consumed broadcast flag

  • Prevented by a design rule
  • Earliest detector a stress test (did not exist then)
  • Diagnosis easy
  • Fix design moderate
  • Debugging aids logging
  • Rust no

Would Rust have prevented it? No. A flag that the first waiter clears is a logic error in how a condition is signalled; Rust's Mutex and Condvar accept the same code, and the fix (a generation counter) is the same. More on Rust

Could it have been prevented?

Yes, with a well-known rule for condition variables: a signal meant for every waiter must not be consumed by one of them. The flags added in bitcoin#33676 and bitcoin#34184 were cleared by the waiter that saw them, which is correct only if there is never more than one waiter per object. Over IPC a client can call one object from several threads, so that assumption was never enforced. A generation counter that each wait compares against its starting value, never cleared, follows the rule.

Could it have been found sooner?

Yes, by a test that starts two waits on one object and interrupts once. The tests added with both PRs used one wait at a time. The reporter wrote the two-wait test with a Python client that runs two worker threads.

What made it hard to debug?

Little once suspected: the blocked call is visible in the IPC log as a request that started and never finished. In a real client it would look like an occasional long stall that ends at the next block, which is easy to mistake for normal waiting.

Lesson

When a flag wakes several waiters, do not let a waiter reset it; record a generation number at the start of each wait instead, so every waiter that started before the signal sees it.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom hang
Worst effect observed hang
Seen in functional-test
Scope production
Trigger Concurrent calls (concurrent-calls)
Phase steady-state
Side server
Component bitcoin-mining
Root cause kind assumption-violated
Fix kind none
Detection review
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released binary
Confidence high