IPC stability bugs

Bug 89

On Windows, a thread exiting after IPC calls can deadlock the process

Status
Open, fix under review
Symptom
Deadlock; worst effect seen: deadlock
Where seen
Bitcoin Core unit tests, Bitcoin Core functional tests, libmultiprocess tests (mptest)
Trigger
Shutdown
First reported
2026-04-20 (first report)
Fixed
Not yet
Fix
mp#368
In released binaries
No
Defect ever on master
Yes

Summary

On Windows, a program using libmultiprocess can hang when one of its threads that made IPC calls exits. The thread's per-thread IPC state is cleaned up by a thread_local destructor, which Windows runs while holding its global loader lock, and that cleanup waits for the event loop. If the event loop is at that moment waiting for a worker thread to start or exit, the worker needs the loader lock, and all three threads wait on each other forever. It was found with the Windows port of the Stratum v2 template provider (sv2-tp) and reproduced with native MSVC and MinGW builds. libmultiprocess does not support Windows on master yet, so no release is affected. The fix is mp#368, a draft.

What happened

Root cause

libmultiprocess gives every application thread that makes IPC calls a dedicated worker thread on the server side, and keeps the handles for those workers in a thread_local ThreadContext. When the application thread exits, the ThreadContext destructor releases the handles, and releasing each one waits for the event loop with sync().

On Windows, the loader lock is held while threads start and exit and while thread_local destructors run. The event loop sometimes waits for a thread to start (when it creates a worker) or to finish exiting (when it destroys one, after a client releases it or a connection closes). If an application thread is exiting at the same moment, it holds the loader lock and waits for the event loop, the event loop waits for the worker, and the worker waits for the loader lock. On Linux and macOS, thread exit does not take a shared lock, so the same sequence completes.

Fix

No fix is merged yet. ryanofsky suggested fixing it on the side that holds the lock: the exiting thread should not wait for anything, which covers every place the event loop waits on a thread, including future ones. mp#368 does this. The ThreadContext destructor now hands the handles to the event loop's async cleanup thread and returns without waiting. To make that safe, the handle maps move to a heap object that disconnect handlers can still use after the thread is gone, and cleanup can be queued from any thread. The PR also adds portable tests for these paths.

A last commit also joins workers on the async cleanup thread instead of the event loop. ryanofsky noted that, once the exiting thread no longer waits, this is a separate improvement that keeps the event loop responsive, not part of the deadlock fix. On 2026-10-13 ryanofsky asked for the PR to be rebased on mp#361, which reworks the same code to fix a thread map teardown race, and Sjors marked it a draft until then.

Next steps

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern CI, toolchain and platform: A thread-local variable with a destructor

  • Prevented by a design rule
  • Earliest detector a unit test (did not exist then)
  • Diagnosis hard
  • Fix design hard
  • Debugging aids symbolized stack traces, thread names
  • Rust no

Would Rust have prevented it? No. A destructor that blocks on another thread while the OS holds a lock that thread needs is a deadlock, which Rust does not prevent; Rust's thread_local! destructors on Windows run in the same loader-locked TLS callback, so the same design would hang the same way. More on Rust

Could it have been prevented?

Yes, with a rule Microsoft documents for code that runs under the loader lock: do not wait for other threads there. libmultiprocess's per-thread cleanup is a thread_local destructor that releases remote thread handles and waits for the event loop to do it, and Bitcoin Core's own tidy check forbids nontrivial thread_local destructors; the library silenced it as harmless "on modern platforms". The fix in mp#368 follows the rule: the destructor queues the cleanup on another thread and returns. As ryanofsky put it, avoiding waits in the few places that run under the lock is easier to keep correct than checking every place the event loop might wait on a thread.

Could it have been found sooner?

Only by running on Windows, which libmultiprocess did not support before the port in mp#231. A test in which a thread makes an IPC call and exits while the event loop joins a worker hangs reliably on MSVC; such a test now exists in mp#368, but on Linux it passes with or without the fix, so only a native Windows CI job would catch a regression. The first sighting, in sv2-tp's Windows tests in April 2026, was worked around in the test instead of investigated, because no stack trace was captured.

What made it hard to debug?

A hang in teardown, intermittent on MinGW and mixed with heap-corruption crashes from another cause, with no trace at first. The loader lock is invisible in application code; the cycle only became clear from Windows stacks showing the exiting thread owning LdrpLoaderLock while waiting in EventLoop::sync(), the event loop joining a worker, and the worker in LdrShutdownThread. Thread names that encode which client thread a worker serves ("(from ...)") made the stack readable.

Lesson

Code that runs at thread exit, in a thread_local destructor or a DLL detach hook, must not wait for other threads: on Windows it runs under a process-wide lock that thread creation and exit also need, so hand the work to another thread instead.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom deadlock
Worst effect observed deadlock
Seen in unit-test, functional-test, mptest
Scope production
Trigger Shutdown (shutdown)
Phase any
Side both
Component proxy-io
Root cause kind lock-order
Fix kind new-code-path
Detection ci-intermittent
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released no
Confidence medium