IPC stability bugs

Bug 80

macOS IPC listener stops accepting connections after a queued client disconnects

Status
Fixed
Symptom
service-loss; worst effect seen: other
Where seen
Branch-build, libmultiprocess tests (mptest)
Trigger
A client disconnects abruptly, for example because it is killed during a call
First reported
2026-07-17 (first report)
Fixed
2026-09-09
Fix
bitcoin#35796
In released binaries
Yes (v30.0-v31.1)
Defect ever on master
Yes

Summary

On macOS, a bitcoin-node -ipcbind socket stopped accepting IPC connections for good if a client connected and then closed its socket before the node accepted it. The node kept running, but every later connection attempt was refused. The defect is in Cap'n Proto's kj library, not in libmultiprocess or Bitcoin Core: macOS reports EINVAL when kj sets the TCP_NODELAY option on such a dead socket, kj treated that error as fatal, and the exception ended libmultiprocess's accept loop. Released macOS binaries v30.0 through v31.1 are affected. It was fixed by bitcoin#35796, which patches Cap'n Proto in depends.

What happened

Root cause

After accept() returns, kj's acceptImpl (kj/async-io-unix.c++) calls setsockopt(TCP_NODELAY) on the new socket. It tolerates the errors EOPNOTSUPP and ENOPROTOOPT, and EINVAL only when built for FreeBSD. macOS returns EINVAL for a Unix socket whose peer has already closed, so on macOS the call threw. mp::ListenConnections runs its accept loop as a daemonized task, a background promise chain on the event loop that nothing waits on. The exception escaped the accept().then(...) chain, and the task, which is the whole listener, was dropped. Nothing restarts it.

A client has to disconnect while it is queued, before the server accepts it. Listeners normally accept immediately, so this window was almost never open until mp#310's test created it deliberately and bitcoin#35037's max-connections option made a saturated listener easy to produce. Any process that connects and exits while queued, including a killed C++ client, leaves such a socket.

The code

The failing code is in Cap'n Proto, which Bitcoin Core does not vendor, so its lines appear here only through the fix's patch file. The libmultiprocess side is quoted from e07224f2d554^. _Listen in src/ipc/libmultiprocess/include/mp/proxy-io.h accepts one connection at a time, as a task in the event loop's m_task_set, and queues the next accept() only from the success callback:

auto* receiver = listener->m_receiver.get();
loop.m_task_set->add(receiver->accept().then(
    [&loop, &init, listener](kj::Own<kj::AsyncIoStream>&& stream) {
        ++listener->m_active_connections;
        // ...
        _Listen<InitInterface>(listener, loop, init);
    }));

If accept() throws, the callback never runs and the task's error handler in src/ipc/libmultiprocess/src/mp/proxy.cpp only logs it:

void LoggingErrorHandler::taskFailed(kj::Exception&& exception)
{
    KJ_LOG(ERROR, "Uncaught exception in daemonized task.", exception);

kj's acceptImpl sets TCP_NODELAY on every accepted socket and ignores only the errors listed for the platform; EINVAL is listed for FreeBSD only. On macOS, a Unix socket whose client closed while it waited in the accept queue gives EINVAL, so accept() throws, nothing calls _Listen again, and the node stops accepting IPC connections while it keeps running.

The fix adds the upstream Cap'n Proto commit to depends as depends/patches/capnp/macos_accept_dead_socket.patch (e07224f2d554). Its EINVAL hunk, shown as it appears in the new patch file, so every line carries the file's leading +; the addrlen == 0 hunk, the kj tests and the capnp.mk wiring are omitted:

+@@ -1357,8 +1369,10 @@ public:
+               ownFd.get(), IPPROTO_TCP, TCP_NODELAY, (char*)&one, sizeof(one))) {
+           case EOPNOTSUPP:
+           case ENOPROTOOPT: // (returned for AF_UNIX in cygwin)
+-#if __FreeBSD__
+-          case EINVAL: // (returned for AF_UNIX in FreeBSD)
++#if __APPLE__ || __FreeBSD__
++          case EINVAL:
++            // On FreeBSD, EINVAL is returned for AF_UNIX sockets.
++            // On macOS, EINVAL may be returned for sockets that are already dead (due to a race with RST).
+ #endif
+             break;

On macOS EINVAL from setsockopt is now ignored like the other tolerated errors, so accept() returns the dead connection, which fails on first use and is dropped, and _Listen queues the next accept. Builds that use a system Cap'n Proto without this commit are still affected.

Fix

The fix commit e07224f2d554 adds depends/patches/capnp/macos_accept_dead_socket.patch. The patch is the upstream Cap'n Proto commit 85d8e47f. That commit is the 1.x backport (capnproto#2748, opened by xyzconstant and merged upstream on 2026-07-28) of 7df5bd078 from Cap'n Proto's unreleased v2 branch. It tolerates EINVAL on macOS as well as FreeBSD. For a related macOS case, where accept() returns a socket with addrlen == 0, it discards the dead socket and accepts again.

The first version of the PR carried only the two-line EINVAL change. After @hebasto asked why the other non-test changes in the upstream commit were left out, the patch was replaced with the full upstream commit, and ryanofsky checked that it matches upstream. The PR description still describes the earlier version ("based on the EINVAL handling"); the merged commit carries the full upstream change.

In mp#310, xyzconstant first proposed retrying the accept loop inside libmultiprocess. ryanofsky turned it down because an unconditional retry would loop forever on permanent errors such as running out of file descriptors. mp#310 was merged with only the test, which tolerates the error on macOS until a patched Cap'n Proto is in use.

The depends patch only helps builds that use depends, such as release builds. Builds against a system Cap'n Proto need a Cap'n Proto release that contains capnproto#2748, which is why mp#319 stays open.

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern Error paths that skip cleanup or completion: One error ends a service loop

  • Prevented by a design rule
  • Earliest detector a disconnect test (did not exist then)
  • Diagnosis easy
  • Fix design moderate
  • Debugging aids logging, a reproduction script
  • Rust no

Would Rust have prevented it? No. Letting one connection's accept error end the listener loop is an error-classification decision; a Rust accept loop that returns on the first Err does exactly the same. More on Rust

Could it have been prevented?

The defect itself, kj treating macOS's EINVAL from setsockopt on a dead socket as fatal, is upstream Cap'n Proto's and was not preventable here. What this project owns is the amplification: ListenConnections runs accept() as a daemonized task whose only error handler logs, and the next accept() is queued only from the success callback, so any exception from one accepted connection ends the listener for the life of the process. The rule that would have contained it is that a service loop must separate errors belonging to one connection (drop it, continue) from errors belonging to the listener (stop), and never let the first kind end the loop. That is harder than it sounds, which ryanofsky's rejection of a blanket retry (review) shows: an unconditional retry spins forever on fd exhaustion. kj's accept() does not say which kind an exception is, so the safe fix had to be upstream.

Could it have been found sooner?

Yes, by exactly the test that found it: a client that connects and closes before the server accepts, run on macOS. No test disconnected at that point until xyzconstant's server-side disconnect series (mp#310, 2026-07), and Bitcoin Core listeners accept immediately, so the queued-then-dead client window was almost never open in practice. macOS CI existed in both repositories for years; it had nothing to run through this window. Any test that reaches the window fails deterministically, so the gap was purely a missing test case, and the max-connections option of bitcoin#35037 made it reproducible by hand with socat within a week.

What made it hard to debug?

Nothing about the diagnosis: the "Uncaught exception in daemonized task" log line names the kj source line and the errno, ViniciusCestarii found the same case already fixed on Cap'n Proto's v2 branch the next day, and the socat recipe in mp#319 is a two-command reproduction. The effort went into the fix path: the libmultiprocess-side retry was rejected, the upstream fix had to be backported to the 1.x line (capnproto#2748), the depends patch grew from a two-line EINVAL hunk to the full upstream commit after hebasto asked, and builds against a system Cap'n Proto remain affected until a release contains it. Seven weeks from report to merge for a two-line platform check is the cost of a fix that has to travel through a dependency.

Lesson

A background accept loop whose error handler only logs will die silently on the first per-connection error a new platform produces; decide in advance which errors end the loop and which end only the connection, and test the accept path with a peer that disconnects while queued, on every platform you ship.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom service-loss
Worst effect observed other
Seen in branch-build, mptest
Scope production
Trigger Client disconnects abruptly (client-unclean-disconnect)
Phase steady-state
Side server
Component capnproto
Root cause kind unhandled-exception
Fix kind workaround
Detection new-test
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released binary
Confidence high