IPC stability bugs

Bug 25

IPC unit tests abort on 32-bit CI when GCC and clang code are mixed at -O0

Status
Fixed
Symptom
Crash; worst effect seen: crash
Where seen
Bitcoin Core unit tests
Trigger
Normal use; no special event is needed
First reported
2025-01-31 (first report)
Fixed
2025-03-13
Fix
bitcoin#31998

Summary

With depends and the main build both at -O0, ipc_tests aborted deterministically on the i686 (32-bit x86) multiprocess CI job with a kj "Out-of-bounds ArrayPtr::slice()" exception thrown from the EventLoop constructor. kj is the support library that ships with Cap'n Proto. The job built Cap'n Proto with GCC and Bitcoin Core with clang, and on i686 the two compilers pass kj's empty PlacementNew argument differently, so a GCC-built kj::ctor called a clang-built operator new and got back a wrong pointer. This was a CI toolchain problem in third-party headers, not a defect in libmultiprocess or Bitcoin Core code, and it did not affect users. Fixed by bitcoin#31998 ("depends: patch around PlacementNew issue in capnp"), which patches Cap'n Proto in depends.

What happened

@fanquake's bitcoin#29796, which built debug builds at -O0, made the i686 multiprocess job fail in ipc_tests with kj/common.h:1797: failed: expected start <= end && end <= size_; Out-of-bounds ArrayPtr::slice(). and SIGABRT. He reproduced it on master with depends and the main build forced to -O0 and filed bitcoin#31772 (2025-01-31). ryanofsky got a gdb trace into kj::CidrRange::CidrRange called from kj::setupAsyncIo() in mp::EventLoop::EventLoop, then reduced it to a small two-compiler test case (2025-02-04 to 02-06). bitcoin#29796 was closed without being merged, so the failure never reached master CI.

Root cause

kj declares inline void* operator new(size_t, kj::_::PlacementNew, void* p) with an empty struct as the second parameter, and kj::ctor<T>() calls it. GCC and clang use different calling conventions for passing an empty class by value on i686 (a known Itanium C++ ABI gap). At -O0 nothing is inlined, and because libbitcoin_ipc_test.a precedes the kj static libraries on the link line, the linker paired the GCC-generated kj::ctor<unsigned int> from libkj with the clang-generated operator new from the Bitcoin object. The mismatched call returned the wrong pointer, so ctor never wrote its value: ArrayPtr<const char>::findFirst returned a Maybe<size_t> holding garbage, and CidrRange's constructor sliced out of bounds.

The job had mixed GCC-built depends with a clang main build since bitcoin#21812 (2021-05), which enabled _GLIBCXX_DEBUG on the job and switched it to clang to avoid an out-of-memory failure in commit fad0f21c3cab. It had been an i686 job since bitcoin#22923 ("test: Switch multiprocess to clang i686 build", 2021-09). The combination was harmless while optimization inlined operator new, which the fix's patch also notes; bitcoin#29796 removed that.

Fix

ryanofsky wrote a Cap'n Proto patch that moves the empty struct to the last parameter of the placement operator new, so both compilers agree on how the pointer argument is passed, and sent it upstream as capnproto#2235. Commit 1ef22ce33517 (fanquake) applies it in depends as depends/patches/capnp/abi_placement_new.patch (+76). bitcoin#31998 was split out of bitcoin#29796 so it could merge separately.

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern CI, toolchain and platform: Objects from two compilers in one program

  • Prevented by other
  • Earliest detector a unit test (existed then)
  • Diagnosis hard
  • Fix design easy
  • Debugging aids symbolized stack traces, a reproduction script
  • Rust n/a

Would Rust have prevented it? Not applicable. A kj header that GCC and clang lay out differently on i686 is a C++ toolchain problem in which no libmultiprocess or Bitcoin Core code is wrong; a capnproto-rust build has one compiler and no kj, but the question does not apply to the C++ code. More on Rust

Could it have been prevented?

Only on the CI side. The defect is in a kj header (an empty struct passed by value to placement operator new, where GCC and clang disagree on i686), so nothing in libmultiprocess or Bitcoin Core could have avoided writing it. What made it reachable was a job that linked GCC-built depends with a clang-built main program, chosen in 2021 to dodge an OOM (fad0f21c3cab), then moved to i686, then run at -O0. A rule of one compiler per link, or at least "a mixed-compiler job is an ABI test and should be labelled as one", would have kept the combination from being the only 32-bit coverage of IPC. The upstream fix (capnproto#2235) reorders the parameters so the empty struct is last; that is a library-author lesson, not one for a user of the library.

Could it have been found sooner?

ipc_tests found it on the first -O0 run; the test existed and the failure was deterministic. It could not have fired earlier because with any optimization the placement new was inlined and the two compilers never met at a call boundary. So the detector was in place and the exposure was created by bitcoin#29796, which is the right order of events; the only earlier signal would have been a deliberate ABI test of the mixed link, which nobody had reason to write.

What made it hard to debug?

The symptom was an out-of-bounds slice inside kj::CidrRange during EventLoop construction, three libraries away from the cause, and ryanofsky's first hypothesis was a GCC miscompile. It took several days (2025-02-04 to 02-06): a gdb trace into findFirst returning garbage, then noticing that link order paired GCC's kj::ctor with clang's operator new, then a 20-line two-compiler reduction that proved it. The reduction was the turning point; a symbolized trace alone pointed at kj, not at the toolchain. A later, undiagnosed i386 crash in another out-of-line kj function taking an empty struct (bitcoin#34732) suggests the same mechanism may still be live where distributions mix compilers.

Lesson

When a program is linked from objects built by two compilers, every non-inlined call across that seam is an ABI test you did not mean to write; keep one toolchain per link, and when a crash lands deep inside a dependency at -O0 only, suspect the seam before the compiler.

Notes

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom crash
Worst effect observed crash
Seen in unit-test
Scope ci-infra
Trigger Normal use (ordinary-path)
Phase startup
Side n/a
Component toolchain
Root cause kind other
Fix kind workaround
Detection ci-deterministic
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Confidence medium