IPC stability bugs

Bug 67

Code generator's child process could deadlock if starting the compiler failed

Status
Fixed
Symptom
Deadlock; worst effect seen: nothing observed
Where seen
Never observed
Trigger
Starting a child process (fork and exec)
First reported
2026-01-08 (first report)
Fixed
2026-07-17
Fix
mp#274

Summary

The mpgen code generator ran the capnp compiler by calling fork() and then, in the child, allocating memory and (if execvp failed) calling perror, iostreams and std::filesystem. In the child of a multithreaded process that can deadlock on a lock another thread held at fork time. It was found in code review and never observed: mpgen is a single-threaded build-time tool, so the precondition does not arise in practice, and nothing that runs in a node or wallet process used this path. Removed in mp#274, which prepared the library for non-Unix platforms, by switching to posix_spawnp.

What happened

Root cause

Generate() in src/mp/gen.cpp invoked the capnp compiler by fork()ing and calling mp::ExecProcess(args) in the child. ExecProcess (src/mp/util.cpp) built the argv vector with MakeArgv, a heap allocation, in the child on every call, and on execvp failure it called perror, std::cerr << and std::filesystem::weakly_canonical before _exit(1). All of these allocate or take stdio locks. After fork() in a multithreaded process only the calling thread exists in the child, so a lock held by another parent thread at fork time stays held forever and the child can deadlock.

Fix

1c6ef7a26c0d (co-authored with Claude) replaces ExecProcess with StartProcess(), which uses posix_spawnp so no user code runs in the child, and gen.cpp becomes WaitProcess(StartProcess(args)) (+12/-25). Spawn failures now surface through KJ_FAIL_SYSCALL in the parent, and the weakly_canonical path in the error message is dropped. The change was needed anyway for the Windows port, which has no fork().

The commit message says "the child process could deadlock if exec() failed". The code shows a slightly wider window: the child also allocated argv with MakeArgv before execvp on every call, not only when exec failed.

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern Process spawning, file descriptors and signals: Unsafe work between fork and exec

  • Prevented by a different API shape, a design rule
  • Earliest detector code review (existed then)
  • Diagnosis easy
  • Fix design easy
  • Rust yes

Would Rust have prevented it? Yes. Safe Rust has no fork(); std::process::Command spawns without running user code in the child, and the pre_exec hook that would allow it is unsafe. More on Rust

Could it have been prevented?

Yes, by the API shape the fix adopts: posix_spawnp runs no user code in the child, so the question of what is safe between fork() and exec() never arises. 1c6ef7a26c0d replaces the inline fork/exec/wait in gen.cpp with WaitProcess(StartProcess(args)) and deletes the child-side perror, iostream and std::filesystem calls along with the argv allocation (+12/-25). The rule itself is POSIX's and was known in 2019 when f89e4b32dc4e wrote the path; the project only started applying it after the same class of defect hung bitcoin-node in SpawnProcess (mp#237), and that fix left this instance on purpose because mpgen is single-threaded. The precondition (a second thread holding an allocator or stdio lock at fork time) genuinely does not arise in mpgen, so this is a rule violation rather than a reachable hang, and the fix came as a side effect of the Windows port needing a fork-free launcher.

Could it have been found sooner?

It was found by review, twice: Sjors labelled ExecProcess "Not safe to call in a post-fork child of a multi-threaded process" in 14e926a3ff37 (2026-01-08) and raised it again when mp#274 touched the function. Nothing else could have: the hang needs a multithreaded parent, no test runs mpgen with one, and no tool in the project's toolchain flags non-async-signal-safe calls in a fork child. A review checklist item "what runs between fork and exec?" is the only detector that would have worked in 2019, and it is the one that worked in 2026.

What made it hard to debug?

Nothing was debugged; the record exists because the mechanism had already been diagnosed in mp#237, where the symptom was a silent child and a 1200 s CI timeout. What this instance shows is the cost of fixing a class of defect one site at a time: mp#237 fixed SpawnProcess, documented ExecProcess as unsafe rather than fixing it, and still left a perror in the SpawnProcess child that mp#312 removed seven months later. A single search for fork( after the first fix would have listed all three.

Lesson

When a defect is a rule violation ("nothing but system calls between fork and exec"), search the codebase for every instance when fixing the first, and prefer an API such as posix_spawn that makes the rule impossible to break over one that asks each caller to remember it.

Notes

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom deadlock
Worst effect observed none
Seen in never
Scope build
Trigger Starting a child process (fork-exec)
Phase any
Side spawn
Component mpgen
Root cause kind api-misuse
Fix kind refactor
Detection review
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Confidence low