IPC stability bugs

Bug 61

CI jobs killed when too many slow IPC tests ran at once

Status
Fixed
Symptom
Crash; worst effect seen: other
Where seen
Bitcoin Core functional tests
Trigger
Normal use; no special event is needed
First reported
2026-03-25 (first report)
Fixed
2026-03-26
Fix
mp#264
Shipped to Bitcoin Core
bitcoin#34804, bitcoin#34952

Summary

The CI jobs in the libmultiprocess repository that build Bitcoin Core with the library change and run its tests were killed with SIGTERM (exit code 143) a few minutes into running Bitcoin Core's IPC functional tests, on every PR and on master, for about a day. They ran up to ten times as many parallel test processes as CPUs, and once mp#263 raised the test timeout factor from 4 to 40, dozens of slow interface_ipc_mining.py runs piled up at once. This was a CI configuration problem and did not affect users. Fixed in mp#264 by lowering the parallelism and run counts.

What happened

Root cause

.github/workflows/bitcoin-core-ci.yml, added in mp#253 to run some Bitcoin Core CI jobs in the libmultiprocess repository, ran test_runner.py with --jobs set to nproc times nproc_multiplier, where the multiplier was 10 (5 on macOS), and repeated the two IPC tests 200-300 times. interface_ipc.py takes 1-3 s but interface_ipc_mining.py about two minutes, so as the cheap runs drained from the queue the runner ended up running about 40 copies of the expensive test at once. mp#263 raised the timeout factor from 4 (10 on macOS) to 40, so slow runs that had failed quickly now kept running, and the step was killed. What sent the SIGTERM was never pinned down; ryanofsky said "I'm still not sure what would actually be sending the sigterm signals". Resource exhaustion on the GitHub-hosted runner is the working explanation.

The over-subscription dates from mp#253; mp#263 is what made it fail reliably.

Fix

336023382c4c (Sjors) lowers nproc_multiplier to 2 on all jobs and the functional test runs to 20-25 (+6/-6). Sjors reported that it then ran for an hour without failing. ryanofsky proposed running the cheap and expensive tests with separate job counts, or a --repeat-for mode in the test runner, as a follow-up.

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern CI, toolchain and platform: A CI job at the edge of its limits

  • Prevented by a design rule
  • Earliest detector other (existed then)
  • Diagnosis moderate
  • Fix design easy
  • Debugging aids CI artifacts, logging
  • Rust n/a

Would Rust have prevented it? Not applicable. CI configuration; no code involved. More on Rust

Could it have been prevented?

Partly. The workflow from mp#253 sized its parallelism for the cheap test (--jobs at ten times nproc, hundreds of repeats) while the queue also held interface_ipc_mining.py, which runs about two minutes and is memory-heavy; once the short runs drained, some forty copies of the long one ran at once. A rule "size a runner's job count to the heaviest test in the queue, not the average" would have set the multiplier at two from the start, which is where 336023382c4c put it. It was not obvious in advance: with the original timeout factor of 4 the slow runs failed fast and the queue never filled, so the configuration looked fine for weeks, and it only tipped when mp#263 raised the factor to 40 to chase a different failure.

Could it have been found sooner?

It was found as soon as it existed: the first exit 143 was on mp#263's own CI run, Sjors reported it the same day, and the change merged anyway because the kill looked like unrelated runner trouble. The detector that mattered was the workflow itself; what was missing was treating a new failure mode on a PR's own run as a reason not to merge. A run of the workflow at the new factor before merging, watched for resource use rather than pass/fail, would have shown the pile-up.

What made it hard to debug?

The failure was a bare Process completed with exit code 143 two to six minutes into the step, with no message from whatever sent the signal; maflcko's sighting added the runner log line "The runner has received a shutdown signal", which says the host stopped the job but not why. ryanofsky collected ten jobs across four branches within a day and worked out from their timing and the test durations that the expensive test was accumulating, but wrote that he was "still not sure what would actually be sending the sigterm signals"; resource exhaustion was the working theory and Sjors's hour-long run of the reduced configuration was the evidence it was right. A memory or load graph from the runner, or a test-runner option that caps the number of concurrent copies of one test, would have turned a day of cross-PR correlation into a glance. The failures the higher factor was meant to fix were a test race that no factor could fix (see mp#253), so the change that caused the kills bought nothing.

Lesson

When a CI job repeats a mix of cheap and expensive tests, the steady state is all expensive ones running at once; set concurrency for that case, and when a PR's own run dies in a new way, treat it as the PR's problem until shown otherwise.

Notes

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom crash
Worst effect observed other
Seen in functional-test
Scope ci-infra
Trigger Normal use (ordinary-path)
Phase any
Side n/a
Component ci
Root cause kind assumption-violated
Fix kind config
Detection ci-deterministic
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see unknown
Confidence medium