IPC stability bugs

Bug 102

Process aborts when two thread proxies post to one thread at once

Status
Open
Symptom
Assertion failure; worst effect seen: test failure only
Where seen
Libmultiprocess tests (mptest)
Trigger
Several calls running at the same time
First reported
Reported privately in an independent security review, 2026-09-28
Fixed
Not yet
In released binaries
Yes (v31.0 onward)
Defect ever on master
Yes

Summary

Each thread that runs IPC calls has a Waiter that holds one pending call. Since mp#240, each thread proxy (ProxyServer<Thread>) waits for its previous call to start before posting the next, and then asserts that the post succeeded. But one thread can have several thread proxies, one per connection it makes calls over, so two of them can post before the thread picks up the first, and the assertion aborts the process. It was reported privately in an independent security review. The libmultiprocess C++ client cannot trigger it, and no Bitcoin Core call sequence that does is known; a client that overlaps calls across proxies could. No fix is merged yet.

What happened

Nothing has been seen in a running process. An independent security review reported on 2026-09-28 that two connections can post work to the same thread at the same time, and that the second post is dropped. Its test posts twice to a Waiter directly and shows that only one function runs. The path through two connections was argued from the source and not reproduced.

Root cause

Waiter::post() in include/mp/proxy-io.h stores the function to run in a single slot, m_fn, and returns false if the slot is already full. ProxyServer<Thread>::post() in the same file chains every post on that proxy's m_thread_ready promise, which is fulfilled when the previous posted function starts running, and then runs assert(posted). The code comment states the assumption: "new functions are only posted when m_thread_ready is signaled, so this should never happen".

The gate belongs to one proxy, but the Waiter belongs to the thread. CustomBuildField in include/mp/type-context.h creates a callback-thread proxy for every connection a thread makes calls over, and a server worker thread created by makeThread also has its own request-thread proxy. Each of these proxies has its own gate. If two of them post before the thread moves the first function out of m_fn, the second post fails. The assertion runs on the event loop thread, so the whole process aborts. Bitcoin Core always builds with assertions enabled.

In a standalone libmultiprocess build with assertions disabled, reading the code, the dropped function destroys its kj promise fulfillers. That rejects the call with an error and also rejects that proxy's m_thread_ready, so every later call through the proxy fails. This was not run.

A C++ client's calls on one thread are synchronous and nested: the thread waits for each call, and the peer posts back to it only while it is waiting, so it takes each function before the next one can arrive. Reaching the assertion needs a peer that sends a call for a thread through one proxy while another call for that thread is pending through a different proxy, the same overlapping-call behaviour that caused the earlier single-slot crash, now across proxies.

Fix

No fix merged yet. The fix proposed with the report replaces the single slot with a queue, so every post is kept and run in arrival order, and removes the assertion:

-    std::optional<kj::Function<void()>> m_fn MP_GUARDED_BY(m_mutex);
+    std::deque<kj::Function<void()>> m_fns MP_GUARDED_BY(m_mutex);

post() then always succeeds, the waiting loop drains the queue, and the per-proxy gates stay as they are.

Next steps

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern An in-process interface trusting what now comes from another process: An assertion on what is now input from a peer

  • Prevented by a design rule
  • Earliest detector a unit test (did not exist then)
  • Diagnosis moderate
  • Fix design easy
  • Debugging aids a clear assertion message
  • Rust no

Would Rust have prevented it? No. The gate guards one proxy while the slot belongs to the thread; that is a protocol and ownership mismatch, not a memory or data-race error, and a Rust assert! on the same assumption would panic the same way. More on Rust

Could it have been prevented?

Yes, by putting the guard on the thing it protects. The Waiter slot belongs to a thread, but the m_thread_ready gate added in mp#240 belongs to one ProxyServer<Thread>, and a thread can have several. The assertion after the post restated the gate's guarantee as if it covered the thread. Either a queue in the Waiter (what the proposed fix does) or a gate per thread would have made the case impossible to write.

Could it have been found sooner?

By a test that posts through two thread proxies for one thread. mp#240 added tests for overlapping calls through one proxy, the case it was fixing, and nothing exercised a second proxy, which the C++ client never uses this way. An independent security review found it by reading the gate and the Waiter together; its test shows the dropped post at the Waiter level only.

What made it hard to debug?

Nothing has failed yet. If it did, the abort would name assert(posted) and the comment next to it explains the assumption, so the cause would be quick to find. Builds with assertions disabled would instead show a rejected call followed by failures of every later call through that proxy, which would be much harder to trace back.

Lesson

When a guard serializes access to a shared resource, attach it to the resource, not to one of the paths that reach it; an assertion that restates the guard's promise will be wrong as soon as a second path appears.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom assert
Worst effect observed test-failure-only
Seen in mptest
Scope production
Trigger Concurrent calls (concurrent-calls)
Phase steady-state
Side both
Component proxy-io
Root cause kind assumption-violated
Fix kind none
Detection review
Needs a peer that is not libmultiprocess C++ yes
Needed a sanitizer to see no
Defect ever on master yes
Released binary
Confidence medium