IPC stability bugs

Bug 11

Multiprocess CI job runs out of memory linking the debug GUI binary

Status
Fixed
Symptom
Out of memory; worst effect seen: other
Where seen
A CI build
Trigger
Normal use; no special event is needed
First reported
2021-05-24 (first report)
Fixed
2021-05-26
Fix
bitcoin#22061

Summary

The Cirrus "[multiprocess, DEBUG] [focal]" CI task, the only CI job that built Bitcoin Core with IPC support at the time, started failing reliably in May 2021 because the container's memory limit killed the linker while it linked bitcoin-gui. This was a CI resource problem, not a code defect, and it did not affect users. Fixed by bitcoin#22061 ("ci: Bump multiprocess memory"), which gave the task more memory and CPUs.

What happened

Root cause

The multiprocess task was the only CI task that built the template-heavy C++ proxy code that libmultiprocess generates from Cap'n Proto interface files. Since bitcoin#21812 ("ci: Enable D_GLIBCXX_DEBUG for multiprocess task", merged 2021-05-04) it built depends with DEBUG=1 and Bitcoin Core with --enable-debug, turning on -D_GLIBCXX_DEBUG -D_GLIBCXX_DEBUG_PEDANTIC. That PR already hit an out-of-memory failure with gcc in its own CI (comment) and switched the task to clang "to avoid OOM" in commit fad0f21c3cab. Linking the resulting unoptimized, debug-annotated binaries left the default Cirrus container close to its limit, and by 2021-05-24 linking bitcoin-gui crossed it.

@maflcko suggested bitcoin#21869 ("depends: Add missing -D_LIBCPP_DEBUG=1 to debug flags", merged 2021-05-09) as the cause. The code does not support that: its commit fa9249aaccc3 only adds -D_LIBCPP_DEBUG=1 to the depends debug flags, which has no effect on this task, since it builds with clang against libstdc++, not libc++. What pushed the link over the limit that particular week is not known; the laanwj sighting on an unrelated PR shows master itself was near the limit.

Fix

Commit fa0bfc523982 ("ci: Bump multiprocess memory", written by maflcko and opened as bitcoin#22061 by laanwj) sets cpu: 4, memory: 16G and MAKEJOBS: -j8 on the multiprocess task in .cirrus.yml (+3/-0).

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern CI, toolchain and platform: A CI job at the edge of its limits

  • Prevented by other
  • Earliest detector other (did not exist then)
  • Diagnosis easy
  • Fix design easy
  • Debugging aids CI artifacts
  • Rust n/a

Would Rust have prevented it? Not applicable. A CI memory limit hit by a debug link is a build-resource problem; a Rust build has its own (often larger) link-time footprint and the same missing headroom rule. More on Rust

Could it have been prevented?

Partly. Not a code defect: the multiprocess task was the one job compiling the generated Cap'n Proto proxy code, and bitcoin#21812 made it a full _GLIBCXX_DEBUG build three weeks earlier. That PR had already hit an OOM with gcc in its own CI and answered by switching the task to clang (fad0f21c3cab) rather than by raising the limit, which left the link of bitcoin-gui a few hundred megabytes under the container ceiling. The prevention is a headroom rule: when a job change is made "to avoid OOM", record the peak memory and either give the job a margin then or treat the job as one push from failing. The fix (fa0bfc523982, cpu: 4, memory: 16G, MAKEJOBS: -j8) is what that rule would have done in May.

Could it have been found sooner?

Yes, by watching the number Cirrus already showed: the failing task reported a 7.5 GB peak, so a job that printed peak memory against its limit would have shown the margin shrinking as debug builds and the IPC code grew. No such report existed, and the first signal was the linker being killed on an unrelated PR (bitcoin#22017), which laanwj restarted; a restarted intermittent kill hides a deterministic one for anyone whose build is slightly larger, which is what happened to bitcoin#19460 the next day.

What made it hard to debug?

Nothing about the failure: OOMKilled during CXXLD bitcoin-gui says what happened. What was never settled is why that week; maflcko's candidate (bitcoin#21869) only touches libc++ flags the task does not use. The fix did not need the answer, so nobody looked further, which is reasonable for a resource limit but means the growth was never measured.

Lesson

When a CI job is tuned to sit just under a resource limit, it will fail on the next unrelated growth; measure the peak against the limit in the job output and leave a margin when the job is first made heavy, rather than waiting for the kill.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom oom
Worst effect observed other
Seen in ci-build
Scope ci-infra
Trigger Normal use (ordinary-path)
Phase any
Side n/a
Component ci
Root cause kind other
Fix kind config
Detection ci-deterministic
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Confidence low