Bug 59

Mining IPC test times out on slow CI because its waits ignore the timeout factor

Status

Fixed

Symptom

Timeout; worst effect seen: test failure only

Where seen

Bitcoin Core functional tests

Trigger

Normal use; no special event is needed

First reported

2026-03-12 (first report)

Fixed

2026-03-24

Fix

bitcoin#34727

Summary

interface_ipc_mining.py failed intermittently on slow CI runners because its waitNext and waitTipChanged timeouts were hard-coded millisecond values that ignored the test framework's --timeout-factor, so a 1-second wait for a new block template could expire before the test's fee-bumping transaction arrived. The defect was in test code and did not affect users. Fixed by commit ad75b147b5c3 ("test: scale IPC mining wait timeouts by timeout_factor") in bitcoin#34727.

What happened

@Sjors's new libmultiprocess CI job in mp#253 ("ci: run some Bitcoin Core CI jobs"), which runs Bitcoin Core's IPC functional tests under sanitizers and on macOS with high parallelism, failed once on macOS (2026-03-12) in interface_ipc_mining.py at assert template6 is not None in run_block_template_test. That step waits for waitNext to return a better template after a fee-bumping transaction is sent, and waitNext returned nothing. Sjors put it down to "brittleness in the test itself under high CPU pressure"; @maflcko asked whether the timeout scaled with --timeout-factor, and it did not. The failure was seen once and not reproduced.

Root cause

test/functional/interface_ipc_mining.py set timeout = 1000.0 (milliseconds) and a one-minute long_timeout as literals, while the rest of the functional test framework scales waits by self.options.timeout_factor, which CI raises for slow sanitizer builds. BlockTemplate.waitNext returns null when its timeout expires without a better template, and WaitAndCreateNewBlock in src/node/miner.cpp only builds a new template to compare fees once per 1000 ms tick, so a 1000 ms timeout allows a single fee check, at the deadline. The test submits the fee-bumping transaction after the wait has started; on a loaded runner it was most likely not yet in the mempool when that one check ran. This is a reading of the code rather than a reproduced failure, but the one-check window is plainly too tight.

The test had only run in Bitcoin Core's CI before mp#253 ran it under TSan/ASan with many parallel jobs.

The code

Quoted from ad75b147b5c3^. run_block_template_test in test/functional/interface_ipc_mining.py gives waitNext a fixed one-second timeout, and sends the fee-bumping transaction only after the wait has started:

timeout = 1000.0 # 1000 milliseconds
# ...
        waitoptions.timeout = timeout
        waitoptions.feeThreshold = 1
# ...
        template6 = await wait_and_do(
            mining_wait_next_template(template5, stack, ctx, waitoptions),
            lambda: self.miniwallet.send_self_transfer(fee_rate=10, from_node=self.nodes[0]))
        assert template6 is not None

On the node, WaitAndCreateNewBlock in src/node/miner.cpp checks fees at most once per one-second tick, so with a one-second timeout the only check is at the deadline, and it gives up after that:

const auto deadline = now + options.timeout;
const MillisecondsDouble tick{1000};
// ...
        kernel_notifications.m_tip_block_cv.wait_until(lock, std::min(now + tick, deadline), [&]() EXCLUSIVE_LOCKS_REQUIRED(kernel_notifications.m_tip_block_mutex) {
// ...
    now = NodeClock::now();
} while (now < deadline);

return nullptr;

1. Test: starts the waitNext call, sleeps 0.1 s in wait_and_do, then sends the transaction over RPC.
2. Node: at the one-second deadline builds one new template and compares fees.
3. On a loaded runner the transaction is not in the mempool yet at step 2, waitNext returns null, and assert template6 is not None fails.

The CI jobs raise --timeout-factor for slow builds, but these literals ignore it. The fix scales them (ad75b147b5c3; the same change in run_mining_interface_test and the long waits is omitted):

@@ -172,7 +172,7 @@ class IPCMiningTest(BitcoinTestFramework):
         """Test BlockTemplate interface methods."""
         self.log.info("Running BlockTemplate interface test")
         block_header_size = 80
-        timeout = 1000.0 # 1000 milliseconds
+        timeout = 1000.0 * self.options.timeout_factor

With a timeout factor above 1 the wait covers several ticks, so the node checks fees again after the transaction arrives. At the default factor of 1 the window is unchanged.

Fix

Commit ad75b147b5c3 (@enirox001) multiplies both tests' base timeout by self.options.timeout_factor, and makes the one-minute waits max(timeout, 60000.0) so they are never shorter than the scaled base timeout (+4/-4). It was added to bitcoin#34727 ("test: Add IPC wake-up test and reuse mining context"), which was already open with other test changes.

Timeline

• Introduced: unknown; the fixed timeouts date from the tests' original versions.
• 2026-03-12: failure on the macOS job of mp#253, reported by Sjors.
• 2026-03-24: fix merged in bitcoin#34727.
• Test code only; no release or user affected.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.