IPC stability bugs

Bug 69

Spawned process's socket could leak into another child started at the same moment

Status
Fixed
Symptom
File descriptor leak; worst effect seen: test failure only
Where seen
Libmultiprocess tests (mptest)
Trigger
Starting a child process (fork and exec)
First reported
2026-07-20 (first report)
Fixed
2026-07-30
Fix
mp#311
In released binaries
Yes
Defect ever on master
Yes

Summary

SpawnProcess left the spawned child's socket inheritable across exec for a window in the parent, so a fork+exec on another thread at the same moment (for example a -blocknotify shell command) could leak a duplicate of the socket into an unrelated process. A leaked duplicate keeps the parent from seeing EOF when the spawned process exits, so the connection is never cleaned up. It was found by review and shown with a demonstration test; it was never seen in CI or by a user. Released binaries (v30.0 through v31.1) have an older, wider form of the same window, reachable only through the test-only echoipc RPC. Fixed in mp#311 except for a residual window noted below. Bitcoin Core master is still fully affected: its subtree predates both mp#274 and mp#311, so its SpawnProcess sets no FD_CLOEXEC at all, and any fork+exec on another thread can inherit the socket. A test reproduces that. Bitcoin Core gets the fix only with a subtree update.

What happened

Root cause

SpawnProcess (src/mp/util.cpp) creates a socketpair. Since mp#274 (652934fb7937, SocketPair()), both ends carry FD_CLOEXEC. To let the child's end survive execvp, the parent cleared the flag with fcntl(fds[0], F_SETFD, flags & ~FD_CLOEXEC) before calling fork(), an ordering ryanofsky had suggested in mp#274 review to keep the post-fork code async-signal-safe. From that fcntl until the parent closes fds[0] after fork(), any other thread doing its own fork+exec snapshots the descriptor table with the flag cleared, and its child keeps a duplicate of the socket across exec. (The fix's commit message puts the window between the fcntl and fork(), but the flag stays cleared on the parent's descriptor until the parent closes it.)

As ryanofsky noted in review, the race predates mp#274: before SocketPair() set FD_CLOEXEC at all, every concurrent fork+exec inherited both sockets. mp#274 narrowed the window and mp#311 narrows it further. Released Bitcoin Core never carried the mp#274 version, and neither does Bitcoin Core master: both have the older form, in which the child's end is inheritable from socketpair() until the parent closes it after fork(), and the parent's end for the whole life of the connection.

Bitcoin Core master calls SpawnProcess only from the test-only echoipc RPC. The PR description says it "should not be an actual problem in the way Bitcoin Core uses libmultiprocess today, but it may be with Windows and/or multiple connection support".

The code

SpawnProcess in src/mp/util.cpp, as of 1e0c7ff9a51a^. SocketPair() returns two descriptors with FD_CLOEXEC set; the parent clears it on the child's end before forking:

auto fds{SocketPair()};
// ...
// Clear FD_CLOEXEC on fds[0] before forking so it survives exec in the child.
int fds0_flags;
KJ_SYSCALL(fds0_flags = fcntl(fds[0], F_GETFD));
KJ_SYSCALL(fcntl(fds[0], F_SETFD, fds0_flags & ~FD_CLOEXEC));

ProcessId pid = fork();
// ...
if (close(fds[pid ? 0 : 1]) != 0) {

The flag is a property of the parent's descriptor, so while it is cleared every fork() in the process inherits fds[0] without close-on-exec, not only this one:

  1. Thread A (SpawnProcess): clears FD_CLOEXEC on fds[0].
  2. Thread B: forks and execs an unrelated program (for example a -alertnotify shell command); its child keeps a copy of fds[0] across exec.
  3. Thread A: forks the intended child and closes its own fds[0].
  4. The spawned process exits, but thread B's child still holds fds[0], so the parent never reads EOF on fds[1] and the connection is never cleaned up.

The window runs from the fcntl until the parent's close(fds[0]) after fork().

The fix leaves the flag set in the parent and clears it in the forked child, just before execvp (1e0c7ff9a51a; comments trimmed):

@@ -142,11 +142,6 @@ std::tuple<ProcessId, SocketId> SpawnProcess(SpawnConnectInfoToArgsFn&& connect_
-    // Clear FD_CLOEXEC on fds[0] before forking so it survives exec in the child.
-    int fds0_flags;
-    KJ_SYSCALL(fds0_flags = fcntl(fds[0], F_GETFD));
-    KJ_SYSCALL(fcntl(fds[0], F_SETFD, fds0_flags & ~FD_CLOEXEC));
-
     ProcessId pid = fork();
@@ -172,6 +167,16 @@ std::tuple<ProcessId, SocketId> SpawnProcess(SpawnConnectInfoToArgsFn&& connect_
+        const int fds0_flags = fcntl(fds[0], F_GETFD);
+        if (fds0_flags == -1 || fcntl(fds[0], F_SETFD, fds0_flags & ~FD_CLOEXEC) == -1) {
+            ChildFail("SpawnProcess(child): clearing FD_CLOEXEC failed\n");
+        }
+
         execvp(argv[0], argv.data());

The child's descriptor table is its own after fork(), so clearing the flag there affects only the process that is meant to inherit the socket; in the parent fds[0] stays close-on-exec until it is closed. A shorter window remains inside SocketPair(), between socketpair() and the fcntl calls that set the flag.

Fix

1e0c7ff9a51a (Sjors, co-authored with Claude) keeps both descriptors close-on-exec in the parent for their whole lifetime and has the intended child clear the flag on its own copy between fork() and exec(), using the async-signal-safe fcntl and reporting failure through the ChildFail helper added in the preceding refactor commit (+10/-5).

A residual window remains between socketpair() and the two fcntl calls in SocketPair(), which does not use SOCK_CLOEXEC. ryanofsky named it in review and the PR description repeats it. It is still open on master, now for two socketpairs since mp#312 added an error-reporting pair.

Reproduction

The narrow window left on libmultiprocess master was not tested. What was tested is Bitcoin Core master, whose subtree (libmultiprocess v13, from bitcoin#35720) has neither mp#274 nor mp#311. A test in mptest, the libmultiprocess unit test program, stands in for another thread running a shell hook (Bitcoin Core runs -blocknotify and similar hooks with std::system) while SpawnProcess runs. It calls std::system("sleep 3 &") from the fd_to_args callback, which SpawnProcess calls between socketpair() and fork(), and spawns true. After true exits, the test measures how long the parent waits for EOF, and checks whether a std::system child started after SpawnProcess returned also inherits the parent's end. No library change or timing luck is needed. It was run on 2026-09-26 against Bitcoin Core master bfdcd97, which carries libmultiprocess v13 in its subtree.

Patch: mptest-repro.patch. It applies to Bitcoin Core master; last checked against badc957 on 2026-10-15. From a Bitcoin Core checkout:

curl -sO https://ipcbugs.ofsky.org/repro/mptest-repro.patch
git apply mptest-repro.patch
cmake --build build --target mptest
build/src/ipc/libmultiprocess/test/mptest -f spawn_tests.cpp:169

The parent saw EOF only after 3002 ms, when the unrelated sleep exited, instead of right after true exited. The parent's end has no FD_CLOEXEC, and the later child inherited it too:

parent fd FD_CLOEXEC; (flags & FD_CLOEXEC) != 0 = false; EOF after ms; elapsed.count() = 3002; eof = true; later std::system inherits parent end; later_inherits = true

The result is the same on every run. The leak of the parent's end also works in the other direction: an unrelated child keeps the spawned process's peer open, so the spawned process does not see EOF when the parent closes its end. On Bitcoin Core master only the test-only echoipc RPC spawns a process, so the practical exposure is small, but the fix for Bitcoin Core is a subtree update past mp#274 and mp#311. The test patch is available on request and will be included in the GitHub issue for this bug when it is opened.

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern Process spawning, file descriptors and signals: A descriptor leaked into another thread's child

  • Prevented by a design rule, a review checklist item
  • Earliest detector a unit test (did not exist then)
  • Diagnosis moderate
  • Fix design moderate
  • Debugging aids a reproduction script
  • Rust no

Would Rust have prevented it? No. The FD_CLOEXEC flag is process-wide state that every concurrent fork snapshots, which no type system models; Rust's Command clears the flag in the child as the fix does, but by library convention, and raw fcntl and fork are unsafe FFI. More on Rust

Could it have been prevented?

Yes, by a rule the fix now follows: in a multithreaded process a descriptor is inheritable only in the child that is meant to inherit it, so every descriptor is created close-on-exec (atomically, with SOCK_CLOEXEC or O_CLOEXEC) and the flag is cleared only after fork(), in the child's own table. The 2019 SpawnProcess (f324c666150b) set no close-on-exec flag at all, so every concurrent fork+exec inherited both socket ends; mp#274 added the flag but cleared it in the parent before fork(), on ryanofsky's suggestion, to keep the post-fork code async-signal-safe. That trade was made knowingly and was the wrong way round: the descriptor table is process-wide, so a flag cleared "for the child" is cleared for every thread's children until the parent closes the descriptor. A review question, "what does another thread's fork see between these two lines?", answers itself. The residual window inside SocketPair(), between socketpair() and the fcntl calls, is the same rule not yet applied (no SOCK_CLOEXEC), and is still open on master.

Could it have been found sooner?

Only by a test built for it. Sjors's demonstration test, in which a second thread forks and execs while SpawnProcess runs, fails before the fix and passes after, and a similar test against the pre-mp#274 code in Bitcoin Core master shows the parent waiting three seconds for EOF that an unrelated sleep child was holding. Neither test existed before July 2026, and the test was not merged with the fix, so the check still does not run in CI. Nothing else would have seen it: the effect is a connection that never sees EOF, which looks like a slow peer, and Bitcoin Core only spawns from the test-only echoipc RPC and, on the process-separation branch, once at startup before shell hooks are likely to run. Two reviews looked at this code within a month (mp#274, mp#311) and the second found it by following up the first, which is the detector that actually worked.

What made it hard to debug?

There was nothing to debug in the usual sense; the bug was reasoned out from the code before it was ever observed. What took care was bounding the window: the fix commit says "between that fcntl() and fork()", but the flag stays cleared on the parent's descriptor until the parent's close(fds[0]) after fork(), and ryanofsky's review pointed out that the race predates mp#274 and is wider in released code. The fix also had to move the fcntl into the child, where only async-signal-safe calls are allowed, which is why the preceding commit added a ChildFail helper. Had the leak ever shown up in the field, a log line naming the descriptor and the time between child exit and EOF would have been the only clue, and it would have pointed at the wrong process.

Lesson

A file descriptor's flags belong to the process, not to the thread that set them; in a program that forks from more than one thread, create every descriptor close-on-exec atomically and make it inheritable only inside the child that should inherit it.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom fd-leak
Worst effect observed test-failure-only
Seen in mptest
Scope production
Trigger Starting a child process (fork-exec)
Phase any
Side spawn
Component spawn
Root cause kind ordering
Fix kind local-fix
Detection review
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released binary
Confidence high