IPC stability bugs

Bug 2

Spawned child process can deadlock before it starts running

Status
Fixed
Symptom
Deadlock; worst effect seen: hang
Where seen
Bitcoin Core functional tests
Trigger
Starting a child process (fork and exec)
First reported
2026-01-01 (first report)
Fixed
2026-01-13
Fix
mp#237
Shipped to Bitcoin Core
bitcoin#34363
In released binaries
Yes (v30.0-v30.3)
Defect ever on master
Yes

Summary

A child process spawned with SpawnProcess() could deadlock between fork() and exec(), because the child ran code that allocates memory, and if another thread of the parent held an allocator lock at the moment of fork(), the child inherited it locked. In Bitcoin Core this showed up as an intermittent rpc_misc.py timeout in the echoipc RPC, which spawns a bitcoin-node child. Fixed in mp#237 ("Made SpawnProcess() behavior safe post fork()") by building the argument list before forking. v30.x binaries have the bug, but in releases SpawnProcess is reached only through the hidden echoipc test RPC.

What happened

@maflcko reported that rpc_misc.py failed in CI with 'echoipc' RPC took longer than 1200.000000 seconds (bitcoin#34187, 2026-01-01, from a 2025-12-31 CI run of the pull request bitcoin#33775). The node log showed spawnProcess launching a bitcoin-node child and the Init.construct request being sent, then nothing from the child: it never reached exec, and the RPC thread waited on it until the test framework timed out. @Sjors linked his fix on 2026-01-08; ryanofsky's review said it "could easily explain the echoipc child process randomly hanging". This is the only sighting.

Root cause

SpawnProcess() in src/mp/util.cpp called ExecProcess(fd_to_args(fds[0])) in the child, after fork() and before execvp(). fd_to_args is a std::function supplied by Bitcoin Core that builds a std::vector<std::string> of arguments. ExecProcess allocated a std::vector<char*> and, if execvp failed, called perror, std::cerr and std::filesystem::weakly_canonical; a failed close() in the child also threw std::system_error.

In the child of a multithreaded process, POSIX only guarantees async-signal-safe functions. bitcoin-node has many threads, so if one of them held the allocator lock (or another internal library lock) at the instant of fork(), the child's copy of that lock stayed locked forever and the child's first allocation blocked. The window is a handful of allocations, which is why it took a node with many active threads and frequent CI runs to hit it.

The code

Quoted from 69652f0edfa1^, src/mp/util.cpp. After fork(), the child side of SpawnProcess builds its command line by calling the caller's fd_to_args callback, and ExecProcess builds the argv array:

    pid = fork();
    // ...
    if (!pid) {
        // Child process must close all potentially open descriptors, except socket 0.
        const int maxFd = MaxFd();
        for (int fd = 3; fd < maxFd; ++fd) {
            if (fd != fds[0]) {
                close(fd);
            }
        }
        ExecProcess(fd_to_args(fds[0]));
    }
    return fds[1];
}

void ExecProcess(const std::vector<std::string>& args)
{
    const std::vector<char*> argv{MakeArgv(args)};
    if (execvp(argv[0], argv.data()) != 0) {

fork() copies only the calling thread, but it copies every lock in the process in whatever state it is in. Bitcoin Core's callback returns a std::vector<std::string>, and MakeArgv returns a std::vector<char*>, so the child allocates memory before execvp:

  1. Another thread of bitcoin-node: holds the allocator's internal lock.
  2. Spawning thread: calls fork(); the child's copy of that lock is locked, and the thread that would release it does not exist in the child.
  3. Child: fd_to_args allocates, waits for the lock forever, and never reaches execvp, so the parent waits for a process that never starts.

The fix (69652f0edfa1) builds the arguments in the parent before forking and calls execvp directly in the child. The hunk replacing the child's throw after a failed close() with write() and _exit(126) is omitted:

@@ -122,24 +123,50 @@ int SpawnProcess(int& pid, FdToArgsFn&& fd_to_args)
+    const std::vector<std::string> args{fd_to_args(fds[0])};
+    const std::vector<char*> argv{MakeArgv(args)};
+
     pid = fork();
@@
-        ExecProcess(fd_to_args(fds[0]));
+
+        execvp(argv[0], argv.data());
+        // NOTE: perror() is not async-signal-safe; calling it here in a
+        // post-fork child may deadlock in multithreaded parents.
+        // TODO: Report errors to the parent via a pipe (e.g. write errno)
+        // so callers can get diagnostics without relying on perror().
+        perror("execvp failed");
+        _exit(127);
     }

The child's copies of args and argv are already complete, so on the normal path it only calls close, getrlimit and execvp, which take no library locks. The perror call after a failed execvp is the one remaining exception, as the comment says.

Fix

The fix commit 69652f0edfa1 (Sjors, co-authored by ryanofsky) evaluates fd_to_args and builds argv in the parent before fork(), using the MakeArgv helper from the preceding refactor 14e926a3ff37. It limits the child to close(), execvp() and _exit(), writing a fixed message with write() instead of throwing if close() fails (+38/-10). The test "SpawnProcess does not run callback in child" (5205a87cd90e) checks that the callback is not called after the fork.

The fix left one non-async-signal-safe call, perror("execvp failed") after a failed execvp, with a comment noting it may still deadlock, after ryanofsky's review. 4a56c1837a78 in mp#312 ("report back child errors to parent and throw", merged 2026-08-11) replaced it by reporting the child's error to the parent over a pipe. That path only runs when execvp fails and was never observed to hang.

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern Process spawning, file descriptors and signals: Unsafe work between fork and exec

  • Prevented by a design rule, a different API shape
  • Earliest detector a unit test (did not exist then)
  • Diagnosis moderate
  • Fix design easy
  • Debugging aids logging, core dumps
  • Rust partly

Would Rust have prevented it? Partly. Std::process::Command does what the fix does (argv built before fork, only async-signal-safe calls in the child), so the bug is avoided by using the standard library; a hand-rolled libc::fork() is unsafe FFI where the compiler cannot see the allocation in the child. More on Rust

Could it have been prevented?

Yes. The rule is POSIX's own: in the child of a multithreaded process, only async-signal-safe functions may run between fork() and exec(), because every lock in the parent is copied in whatever state it was in and the thread that would release it does not exist in the child. SpawnProcess() ran a caller-supplied std::function that builds a std::vector<std::string>, then built argv, then on failure called perror, std::cerr and std::filesystem::weakly_canonical, all in the child (f324c666150b, 2019). The fix (69652f0edfa1) is the rule applied: evaluate the callback and build argv before forking, and leave the child with close(), execvp() and _exit(). An API shape that takes a prepared argument list rather than a callback, or posix_spawn, would have made the unsafe version impossible to write. Bitcoin Core had met the same class earlier in bitcoin#33063, outside IPC.

Could it have been found sooner?

Yes, by the test the fix added ("SpawnProcess does not run callback in child", 5205a87cd90e), which checks the rule rather than the deadlock and is deterministic. No spawn test existed until then. The deadlock itself needs another parent thread inside malloc at the instant of fork(), a window of a few allocations, so it stayed invisible for six years and only surfaced when Bitcoin Core CI ran the echoipc RPC often enough from a fully started, many-threaded node (bitcoin#34187, 2025-12-31). A review checklist item "what runs between fork and exec?" would have found it at any time; a tool that flags non-async-signal-safe calls in a fork child does not exist for C++ in the project's toolchain.

What made it hard to debug?

The symptom was a 1200-second RPC timeout with a node log that showed the child being spawned and the Init.construct request sent, then silence: a child that never reached exec produces no output of its own, and the CI job could not attach gdb to it. The cause was inferred from the code rather than traced, which is why the commit says it "likely fixes" the timeout, and it was the only sighting. Sjors credits an LLM with pointing at post-fork unsafety, and ryanofsky's review agreed it "could easily explain" the hang. A core dump or stack of the hung child would have shown the allocator lock directly; the fix knowingly left one perror after a failed execvp with a comment, replaced by an error pipe in mp#312.

Lesson

Do all the work before fork() and nothing but system calls after it; a callback or an allocation in the child of a threaded process is a hang that only the scheduler decides when to show. Write the test that asserts the child runs no user code, since the hang itself is too rare to test for.

Notes

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom deadlock
Worst effect observed hang
Seen in functional-test
Scope production
Trigger Starting a child process (fork-exec)
Phase any
Side spawn
Component spawn
Root cause kind api-misuse
Fix kind local-fix
Detection ci-intermittent
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released binary
Confidence high