IPC stability bugs

Bug 100

IPC process aborts if its cleanup thread cannot be started

Status
Open
Symptom
Crash; worst effect seen: crash
Where seen
Bitcoin Core unit tests
Trigger
A client disconnects abruptly, for example because it is killed during a call
First reported
Reported privately in an independent security review, 2026-09-27
Fixed
Not yet
In released binaries
Yes (v30.0 onward)
Defect ever on master
Yes

Summary

When a client releases a server object without calling destroy, for example by disconnecting with objects still alive, libmultiprocess hands the object's cleanup to a background thread so the event loop is not blocked. That thread is created the first time it is needed, from inside the ProxyServer destructor. If creating it fails, for example because the process has reached its thread limit, the destructor cannot report the error and C++ terminates the whole process. It was reported privately in an independent security review, with a test that injects a failure at that step; it has not been seen in a running process, and the trigger needs a thread limit to be hit at one particular moment. No fix is merged yet.

What happened

Nothing has been seen in a deployment. An independent security review reported the problem privately on 2026-09-27. Its unit test injects an out-of-memory error into the cleanup handoff while a client disconnects and reports that the process terminated. The report names a failure to create the cleanup thread as the other way to reach it. Those runs have not been repeated independently.

Root cause

When a ProxyServer object is destroyed and its client never called destroy(), the server still owns the application object behind it. That happens for interfaces without a destroy method, for clients that simply drop a capability (as Python and Rust clients do), and for every object still alive when a connection drops. The destructor (~ProxyServerBase in include/mp/proxy-io.h) passes the object to EventLoop::addAsyncCleanup() (src/mp/proxy.cpp), which queues it for the asynchronous cleanup thread and, if that thread does not exist yet, creates it with std::thread.

Creating a thread can fail: pthread_create returns an error when the process, its user or its service unit (for example a systemd TasksMax setting) is at its thread limit, or when the thread's stack cannot be mapped. std::thread then throws std::system_error. Destructors in C++ cannot throw by default, so the exception calls std::terminate and the process aborts. The code assumed the handoff could not fail.

The handoff also allocates memory, and the report's test injects its failure there. In Bitcoin Core that is not a separate risk: the node and GUI install an out-of-memory handler that terminates the process before any allocation error is thrown. The same goes for the report's other path, where building a call's result fails on a worker thread and the new server object is destroyed there; that trips an assertion that the cleanup thread is only started from the event-loop thread, but only an allocation failure can cause it. Thread creation is the failure that the out-of-memory handler does not cover.

The thread is created at most once per event loop, at the first such release; since mp#160 it then stays until the loop ends. A connected client can make the node create threads without limit (see AR-04), so a client could reach the limit on purpose; otherwise ordinary limits would have to be nearly exhausted at that moment.

Fix

No fix merged yet. The fix proposed with the report creates the cleanup thread when the EventLoop is constructed, where a failure is an ordinary exception that reaches the caller, so no destructor ever starts a thread. It also replaces the queued std::function with a concrete entry and adds a fallback slot that needs no allocation if adding to the queue fails. The first part alone removes the trigger described here; the rest is hardening for applications that do not terminate on out-of-memory. See AR-62 for the full change and review notes.

Next steps

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern Error paths that skip cleanup or completion: Fallible work in destructor

  • Prevented by a design rule
  • Earliest detector code review (did not exist then)
  • Diagnosis easy
  • Fix design moderate
  • Debugging aids a clear assertion message
  • Rust partly

Would Rust have prevented it? Partly. Std::thread::spawn panics when the OS refuses a thread, and a panic in Drop during normal execution unwinds rather than aborts, but Rust still offers no way to report the error from Drop, so the same rule applies: create the thread up front with Builder::spawn, which returns a Result. More on Rust

Could it have been prevented?

Yes, with a rule that destructors only release resources and never acquire them. The server destructor in the initial commit hands its object to a cleanup queue and starts the cleanup thread on first use, so the first destructor to need it is the one that pays for creating a thread, at a point where failure cannot be reported. Lazy creation saved one idle thread per event loop. The proposed fix shows the rule: create the thread in the EventLoop constructor, where an error is an ordinary exception.

Could it have been found sooner?

Only by review or fault injection. A test would have to make thread creation fail at the first server object released without destroy(), which no ordinary test does, and allocation failures in the same path are invisible in Bitcoin Core because its processes already terminate on out-of-memory. An independent security review found it while injecting failures along IPC error paths.

What made it hard to debug?

Little, if it ever happened: the runtime prints "terminate called after throwing an instance of 'std::system_error'" with the pthread_create error before aborting. The confusing part would be the timing, an abort when a client disconnects, long after the thread limit was approached by something else.

Lesson

Anything that can fail, such as creating a thread or allocating a queue entry, belongs in a constructor or an explicit start step, not in a destructor that runs on cleanup paths where the error has nowhere to go.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom crash
Worst effect observed crash
Seen in unit-test
Scope production
Trigger Client disconnects abruptly (client-unclean-disconnect)
Phase steady-state
Side server
Component eventloop
Root cause kind unhandled-exception
Fix kind none
Detection review
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released binary
Confidence low