IPC stability bugs

Bug 88

IPC listener stops for good after one temporary accept failure

Status
Open
Symptom
service-loss; worst effect seen: other
Where seen
Bitcoin Core unit tests, libmultiprocess tests (mptest)
First reported
Reported privately in an independent security review, 2026-10-15
Fixed
Not yet
In released binaries
Yes (v30.0 onward)
Defect ever on master
Yes

Summary

If accepting an incoming IPC connection fails once with a temporary resource error, for example because the process has run out of file descriptors (EMFILE), the libmultiprocess listener gives up for good and serves no new client until the process restarts. New clients hang while other IPC connections stay open, and get "connection refused" after those close. The error is temporary, so the listener should wait and try again. It was reported privately in an independent security review and reproduced with tests; it has not been seen in a running node. Bitcoin Core now counts IPC connections in its file descriptor budget (bitcoin#35037), so running out needs a descriptor leak or other open files, but the listener should survive it. No fix is merged yet.

What happened

Nothing has been seen in a deployment. An independent security review reported the problem privately on 2026-10-15, with a Linux unit test that fills the process's descriptor table, connects a client, waits until the listener's accept() fails with EMFILE, frees the descriptors and shows that the listener never serves the client, while a plain POSIX listener in the same test accepts its queued client once descriptors are free. A smaller test written the same day reproduced it without touching descriptor limits.

Root cause

ListenConnections() (include/mp/proxy-io.h) runs an accept loop: _Listen() waits for one incoming connection with receiver->accept(), and the code that runs when a connection arrives serves it and calls _Listen() again for the next one. The next accept is only ever scheduled from that success path. When accept() fails, the error goes to the event loop's catch-all handler for background tasks, which logs "Uncaught exception in daemonized task" and drops the task. Nothing schedules another accept. What new clients see depends on whether other connections are open. The dropped task held a reference to the listener object, and the only other references are held by open connections, to update the connection count when they disconnect. While any are open, the listening socket stays open: new clients connect, wait in the kernel's queue and are never accepted. When the last one closes, or right away if there were none, the listener is freed, its socket closes and clients are refused. (The report says the socket stays open, which is the first case; the reproduction has no other connections and shows the second. An earlier version of this page said the socket always closes.)

Cap'n Proto's kj library reports EMFILE, ENFILE (the system-wide limit), ENOBUFS and ENOMEM from accept() as OVERLOADED errors, which are temporary by nature, but the loop treats every failure as final.

The same "one failed accept ends the listener" shape caused a listener on macOS to stop accepting after a queued client disconnected, which was fixed in Cap'n Proto for that specific error. A blanket retry was rejected there, in ryanofsky's review of mp#310, because retrying at once on every error would spin forever on a permanent failure.

Fix

No fix merged yet. The fix proposed with the report catches only OVERLOADED errors, logs a warning once per run of failures, waits 100 ms on the event loop's timer and calls _Listen() again; any other error still ends the listener as before, so a permanent failure cannot cause a busy loop. The report's version also depends on an unmerged change that lets an application stop a listener (AR-21); without it those checks are not needed. A standalone version (15 lines in _Listen()) passes the reproduction test below.

Reproduction

A test added to listen_tests.cpp in mptest, the libmultiprocess unit test program, wraps the real listening socket in a kj::ConnectionReceiver whose first accept() fails with an OVERLOADED exception, as kj reports EMFILE, and forwards later calls to the real socket. It passes the wrapper to _Listen(), connects a client, and checks that the client is served. No descriptor limits are changed, so it works on every platform. It was run on 2026-10-15 against Bitcoin Core master badc957, which carries libmultiprocess v13 in its subtree.

Patch: ar-79-accept-retry-test.patch. From a Bitcoin Core checkout:

curl -sO https://ipcbugs.ofsky.org/repro/ar-79-accept-retry-test.patch
git apply ar-79-accept-retry-test.patch
cmake --build build --target mptest
build/src/ipc/libmultiprocess/test/mptest -f listen_tests.cpp

Without a fix the test fails at once: the event loop logs Uncaught exception in daemonized task.; exception = ... overloaded: simulated accept failure, and the client's connect() is refused. With the fix patch also applied, the test passes in about 101 ms, the length of the retry delay. The result is the same on every run.

Next steps

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern Error paths that skip cleanup or completion: One error ends a service loop

  • Prevented by a design rule
  • Earliest detector a unit test (did not exist then)
  • Diagnosis easy
  • Fix design moderate
  • Debugging aids logging
  • Rust no

Would Rust have prevented it? No. An accept loop that stops on its first error is control flow, not memory or thread safety; a Rust server loop written the same way stops the same way, and the fix is the same distinction between temporary and permanent errors. More on Rust

Could it have been prevented?

Yes, with a rule every long-running accept loop needs: decide for each error whether the loop ends, retries later, or skips the connection. The loop in d519e1885990 scheduled the next accept only on success and left failures to a catch-all handler that logs and drops the task, so every failure was final by default. Most servers retry descriptor and memory exhaustion after a short delay, because the condition usually clears on its own. The fix shows the rule: catch only the errors kj marks as OVERLOADED, wait, and listen again, while other errors still end the loop so a permanent failure cannot spin.

Could it have been found sooner?

Yes, by a test that makes one accept() fail and checks that a later client is served. That test is small once the failure is injected through a wrapper around the listening socket rather than by exhausting real descriptors. No test made accept() fail until mp#310 in 2026, and that one exercised a macOS-specific error that was fixed inside Cap'n Proto, so the general case stayed untested. It was found by an independent security review looking at error paths under resource pressure.

What made it hard to debug?

Little, once seen: the event loop logs "Uncaught exception in daemonized task" with the accept error. In a running node that one line would have been the only sign, followed by new clients hanging (while other connections were open) or being refused for as long as the process ran, with no further log entry; a warning that names the listener and says it has stopped would have pointed straight at the cause.

Lesson

A server's accept loop must decide what each error means: retry temporary errors such as running out of descriptors after a delay, end on permanent ones, and log when it stops, because "the loop ends on any error" turns a passing resource shortage into a lasting outage.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom service-loss
Worst effect observed other
Seen in unit-test, mptest
Scope production
Trigger other (other)
Phase steady-state
Side server
Component proxy-io
Root cause kind unhandled-exception
Fix kind none
Detection review
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released binary
Confidence high