IPC stability bugs

Bug 43

IPC build hangs forever with Cap'n Proto 0.8 in C++20 mode

Status
Fixed
Symptom
Hang; worst effect seen: hang
Where seen
A CI build
Trigger
Normal use; no special event is needed
First reported
2025-08-12 (first report)
Fixed
2025-08-22
Fix
mp#194
Shipped to Bitcoin Core
bitcoin#33241

Summary

Building Bitcoin Core with IPC enabled against Cap'n Proto 0.8 (for example Ubuntu 22.04's libcapnp-dev) never finished, because the mpgen code generator hung in an infinite loop. The cause was a const char*/StringPtr comparison operator in Cap'n Proto 0.8's kj/string.h that calls itself when compiled as C++20. This was a build-time problem only; programs that did get built were not affected. Worked around in mp#194, which also added a CI job building against old dependencies, and shipped to Bitcoin Core before v30.0.

What happened

Root cause

kj/string.h in Cap'n Proto 0.8 defines

inline bool operator==(const char* a, const StringPtr& b) { return b == a; }

Before C++20 the body converts a to StringPtr and calls the member StringPtr::operator==. C++20 also considers comparison operators with their arguments reversed, and with those candidates the call instead resolves to this same free function, so it recurses forever. The optimizer turns the infinite tail recursion into a self-loop. Generate() in src/mp/gen.cpp compared kj::StringPtr values against string literals (src_prefix == ".", method_name == "construct" and others), which hit this overload. Cap'n Proto fixed it upstream in capnproto#1170, released in 0.9.0.

Bitcoin Core has required C++20 since v27.0, but libmultiprocess's CI and Bitcoin Core's depends build used newer Cap'n Proto versions, so nothing built mpgen against 0.8 with C++20 until this report.

Fix

The workaround commit c4cb758eccb5 wraps the five string literals compared in Generate() as kj::StringPtr{...}, so the member operator is chosen (+5/-5). The same PR adds an olddeps CI job that builds against Cap'n Proto 0.8 (8ceeaa6ae401) and a check for Cap'n Proto versions affected by CVE-2022-46149 (dc3ba2204606).

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern CI, toolchain and platform: A minimum dependency version CI never built

  • Prevented by nothing practical
  • Earliest detector other (did not exist then)
  • Diagnosis moderate
  • Fix design easy
  • Debugging aids symbolized stack traces, a reproduction script
  • Rust n/a

Would Rust have prevented it? Not applicable. A Cap'n Proto header that recurses under C++20 comparison rewriting is a toolchain and dependency problem; no libmultiprocess code is at fault and the question does not apply. More on Rust

Could it have been prevented?

Not practically. The defect is in Cap'n Proto 0.8's kj/string.h, whose operator==(const char*, const StringPtr&) calls itself once C++20's reversed comparison candidates are in play; upstream fixed it in capnproto#1170 for 0.9.0. Nothing in mpgen's five StringPtr == "literal" comparisons was wrong by the library's own rules, and the workaround (c4cb758eccb5) only spells the literals as kj::StringPtr{...} so the member operator wins. The knowable part is policy: the project declared 0.8 supported (it is Ubuntu 22.04's package) while Bitcoin Core had required C++20 since v27.0, and no build ever combined the two.

Could it have been found sooner?

Yes, by a CI job that builds against the oldest dependency versions the project claims to support, which is what mp#194 added as the olddeps job (8ceeaa6ae401). Both libmultiprocess CI and Bitcoin Core's depends build used newer Cap'n Proto, so the first C++20 build of mpgen against 0.8 was fanquake's container while checking bitcoin#31802, days before the first release to ship IPC binaries. The hang is deterministic, so that job would have failed on its first run at any point after the C++20 switch.

What made it hard to debug?

The symptom was a build step that never finished with no output, in a container the reporter had to describe. ryanofsky attached gdb and found mpgen on a jmp to itself inside main, which says "infinite loop" but not where, because the optimizer had folded the tail recursion into a single instruction; the cause came from reading the kj header with C++20's rewritten-candidate rules in mind, the same day. An unoptimized build would have shown a stack overflow with the operator's name on every frame, and a compiler warning for a self-recursive rewritten candidate would have been the ideal aid, but none exists.

Lesson

A supported minimum dependency version that CI never builds is a claim, not a fact; build against the floor as well as the ceiling, especially after raising the language standard, since a newer standard can change what old library code means.

Notes

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom hang
Worst effect observed hang
Seen in ci-build
Scope build
Trigger Normal use (ordinary-path)
Phase any
Side n/a
Component capnproto
Root cause kind other
Fix kind workaround
Detection user-report
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Confidence low