IPC stability bugs

Bug 48

32-bit x86 IPC server crashes while logging an exception thrown by a server method

Status
Open
Symptom
Crash; worst effect seen: crash
Where seen
Libmultiprocess tests (mptest), Bitcoin Core functional tests
Trigger
Unusual input from a peer or caller
First reported
2025-10-03 (first report)
Fixed
Not yet
In released binaries
No; the code was in source releases only
Defect ever on master
Unknown

Summary

On 32-bit x86 builds, an IPC server process crashed with a segfault whenever a server method threw an exception and the server tried to format it for a log message with kj::str(e). The crash is inside kj's out-of-line kj::operator*(kj::_::Stringifier, const kj::Exception&) formatting function, which receives a reference that points to garbage instead of the exception. It was seen in libmultiprocess's gnu32 CI job and in a Debian i386 sanitizer build of bitcoin-node. The cause is not known. There are no official 32-bit x86 release binaries, so only people who build IPC for that architecture themselves are affected. There is no fix. Of the two places that format an exception this way, one now logs the exception differently and avoids the crash; the other still crashes.

What happened

In both cases the server process is gone, and the client sees "Peer disconnected" instead of the text of the remote exception.

Root cause

Not established. In gdb on the gnu32 build, the kj::Exception&& e in serverInvoke is a valid object with the expected description, but two frames deeper, in kj's formatting function, the reference argument points to a different address holding garbage (gdb comment; review thread, where ryanofsky suspected an ABI issue or compiler bug). Holding an extra capability reference, using catch_ instead of then, and explicit lambda return types did not help. Only replacing kj::str(e) with e.getDescription(), which returns the exception's message without calling the formatting function, avoided the crash. maflcko described the Debian case as "another one of the GCC+clang mix things".

A possible explanation, not confirmed by anyone who worked on it: kj::_::Stringifier is an empty struct passed by value as the first parameter of the out-of-line formatting function. That is the same shape as the i686 calling-convention mismatch between GCC and clang for empty classes that broke kj's placement operator new (bitcoin#31772, a 32-bit clang ipc_test failure; see that bug's page). It would fit the Debian build, where a clang-built bitcoin-node calls a GCC-built libkj. It does not explain the gnu32 sighting, where Cap'n Proto and libmultiprocess are both built with GCC, so the cause remains open.

The crash only happens once server code formats a caught exception. libmultiprocess started doing that with mp#218's log line. Bitcoin Core started doing it with the "uncaught exception" log line in PassField, added by 0174450ca2e9 in mp#240 ("Avoid errors from asynchronous (non-c++) clients"). Bitcoin Core's own CI stopped building IPC for i686 in 2025-08, when bitcoin#31802 (which added bitcoin-node and bitcoin-gui to the release binaries) turned the "multiprocess, i686" job into a job without IPC. Since then only libmultiprocess's gnu32 job and custom nightly configurations exercise 32-bit x86 IPC.

Fix

No fix yet. mp#218 was merged (2026-06-02) with its log line using e.getDescription() in serverInvoke (f326c5b1b7b4). That avoids the crash at that site but logs less than kj::str(e) would. It reached Bitcoin Core in bitcoin#35661 (2026-07-07), a subtree update. The kj::str(*exception) call in PassField is still on master, so an affected 32-bit build still crashes whenever a server method throws.

Next steps

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern CI, toolchain and platform: Objects from two compilers in one program

  • Prevented by nothing practical
  • Earliest detector a unit test (existed then)
  • Diagnosis hard
  • Fix design hard
  • Debugging aids symbolized stack traces, core dumps, other
  • Rust n/a

Would Rust have prevented it? Not applicable. An undiagnosed crash inside kj's exception formatter on 32-bit x86, probably a compiler or ABI issue, is a toolchain problem below the library's control; a capnproto-rust build would not contain kj at all, but the question does not apply to the C++ code. More on Rust

Could it have been prevented?

Not practically. The crash is inside kj's exception formatter on 32-bit x86, reached by an ordinary kj::str(e) on a caught kj::Exception, and the cause has not been found: gdb shows a valid exception in serverInvoke and a garbage reference two frames down. The suspected explanations (an i686 calling-convention gap between GCC and clang for the empty Stringifier argument, of the kind that broke kj's placement operator new in bitcoin#31772, or a compiler bug) are below the library's control, and the all-GCC gnu32 sighting does not fit the first one. No rule about how libmultiprocess is written would have avoided calling a documented kj function on a kj exception. The only choice available was the one mp#218 made: log e.getDescription() instead, which loses the exception type and location. The pattern label groups it with the placement-new ABI bug on the strength of the suspected mechanism and maflcko's "GCC+clang mix" reading; it is provisional because the cause is.

Could it have been found sooner?

It was found as early as it could be: the mptest unit test in which a server method throws, run by the libmultiprocess gnu32 CI job, failed the day mp#218 added the first server-side kj::str(e) (2025-10-03), and maflcko's nightly i386 ASan job failed the day after mp#240's PassField log line reached Bitcoin Core (2026-03-04). What is missing is not a detector but coverage: Bitcoin Core's own CI stopped building IPC on i686 in 2025-08, so the second site sat on master for a year, and a server method throwing on an affected 32-bit build still crashes the node. A 32-bit job that runs a test where a server method throws is the check, and it exists only outside the main repository.

What made it hard to debug?

The cause is still unknown after two gdb sessions and several tried workarounds (an extra capability reference, catch_ instead of then, explicit lambda return types). Nothing symbolizes a corrupted reference inside a distribution libkj.so; ASan reported only "SEGV on unknown address"; the third possible sighting (bitcoin#34014) had an unsymbolized stack, so it could not be tied to the same function. What would help is what the ABI diagnosis of bitcoin#31772 needed: a minimal program calling the formatter from the same two compilers, disassembly of both sides of the call, and a build of kj with debug info on the affected platform. Symbolized traces from the nightly jobs and a core dump would at least let the sightings be confirmed as one bug.

Lesson

When a crash lives in a dependency on one platform and only under one mix of compilers, a workaround that avoids the call is a stopgap, not a fix; keep at least one CI job on that platform exercising the error path, so the second site does not sit unnoticed for a year.

Notes

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom crash
Worst effect observed crash
Seen in mptest, functional-test
Scope production
Trigger Unusual input (unusual-input)
Phase steady-state
Side server
Component capnproto
Root cause kind unknown
Fix kind none
Detection ci-intermittent
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master unknown
Released source
Confidence medium