IPC stability bugs

Bug 76

Windows IPC processes corrupt the heap at thread exit with older mingw-w64

Status
Fixed
Symptom
Crash; worst effect seen: crash
Where seen
Bitcoin Core functional tests
Trigger
Shutdown
First reported
2026-07-22 (first report)
Fixed
2026-08-20
Fix
bitcoin#35877
In released binaries
No
Defect ever on master
No

Summary

On Windows builds made with mingw-w64 older than 13.0.0, bitcoin-node and bitcoin-cli could abort with STATUS_HEAP_CORRUPTION (0xC0000374) when a thread that had made IPC calls exited, because the mingw-w64 runtime could free thread_local storage before running libmultiprocess's ~ThreadContext destructor on it. The defect is in the mingw-w64 runtime, reached through libmultiprocess's thread_local ThreadContext. It was seen only in the Windows cross-compiled CI jobs of the Windows IPC port (bitcoin#32387); no released binary was affected, since Bitcoin Core does not build IPC for Windows and its release toolchain already used mingw-w64 13. It went away when bitcoin#35877 moved the CI jobs to a mingw-w64 13 toolchain; the libmultiprocess workaround, mp#318, was closed unmerged.

What happened

Root cause

ThreadContext (include/mp/proxy-io.h) is a thread_local with a nontrivial destructor, marked NOLINT(bitcoin-nontrivial-threadlocal) against Bitcoin Core's tidy check with the comment that this "should not be a problem on modern platforms". Its destructor releases the thread's remote Thread capabilities so the server can free that thread's dedicated server threads.

On mingw-w64 before 13.0.0 with gcc's emutls, both the storage cleanup (emutls_destroy) and the C++ destructor calls (__cxa_thread_atexit registrations) run as winpthreads key destructors, in unspecified order. When storage is freed first, ~ThreadContext walks its request_threads / callback_threads maps through freed memory and double-frees their nodes. The corruption was often detected later, in kj::AsyncIoContext teardown, which is why it first looked like a hang.

The Bitcoin Core CI images used Debian/Ubuntu mingw-w64 12.x packages, which lack the upstream fix, while the release toolchain had already moved to mingw-w64 13, so only CI saw it. libmultiprocess had never run on Windows before the port.

Fix

No code changed in either repository. mingw-w64 commit 8e06daa36dfc, in 13.0.0, fixes the ordering. Bitcoin Core's guix builds already used mingw-w64 13 (31eb46f054f8, merged with bitcoin#34550 on 2026-03-19), and fa8762da626a (maflcko, bitcoin#35877) switched the Windows cross CI jobs to a nix-provided mingw-w64 13 toolchain.

mp#318 had worked around the bug by making CurrentThread() return a deliberately leaked heap object on MinGW, so no destructor was registered at thread exit, at the cost of not freeing server threads until the connection closed. ryanofsky closed it on 2026-08-25, noting that someone building with an older mingw-w64, or with a natively compiled libstdc++ that uses the CRT's __cxa_thread_atexit, could still hit it, and suggesting a #warning or #error on __MINGW64_VERSION_MAJOR < 13. GCC 16 (native thread-local storage on Windows) and Clang also avoid it.

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern CI, toolchain and platform: A thread-local variable with a destructor

  • Prevented by a design rule
  • Earliest detector a unit test (did not exist then)
  • Diagnosis hard
  • Fix design moderate
  • Debugging aids sanitizer output, CI artifacts, logging
  • Rust n/a

Would Rust have prevented it? Not applicable. A mingw-w64 runtime bug in thread_local destructor ordering is a toolchain problem; Rust's thread_local! has its own Windows destructor registration, but the question does not apply to the C++ code. More on Rust

Could it have been prevented?

In principle, by a rule that already existed and was waived. ThreadContext is a thread_local with a nontrivial destructor, which Bitcoin Core's own tidy check bitcoin-nontrivial-threadlocal forbids; libmultiprocess silenced it in 2025 with a comment saying it "should not be a problem on modern platforms, and could lead to a small memory leak at worst on older ones". The mingw-w64 runtime proved the second half wrong: with gcc's emutls, storage teardown and C++ destructor invocation are both pthread-key destructors run in unspecified order, so the destructor can walk freed memory. The design that avoids the class of bug is the one the unmerged workaround (608d09a8fec7) used: a trivially destructible thread-local pointer to a heap object whose cleanup is registered explicitly, so no runtime has to order two teardown mechanisms. The library does need per-thread cleanup (releasing remote Thread capabilities so the server can free that thread's server threads), so "no nontrivial thread_local" is a rule about how to get that cleanup, not whether to have it. The defect itself is in the runtime and was already fixed upstream in mingw-w64 13.0.0 (8e06daa36dfc).

Could it have been found sooner?

Only by building for Windows with the affected toolchain, which nobody did before the Windows port (bitcoin#32387, mp#231). Any mptest run on a mingw-w64 12 build would have crashed the same way, since every thread that made an IPC call exits through ~ThreadContext. The Bitcoin Core CI images pinned Debian and Ubuntu mingw-w64 12 packages while the guix release toolchain had moved to 13 with bitcoin#34550 in March 2026, so the bug was visible in CI only, and would not have reached a release. A version guard (#error on __MINGW64_VERSION_MAJOR < 13), which ryanofsky suggested when closing mp#318, is the check that would still protect someone building with an older runtime.

What made it hard to debug?

Almost everything. The first symptom was a hang, because the corruption was often detected later in kj::AsyncIoContext teardown; only after more diagnostics did the jobs report STATUS_HEAP_CORRUPTION, an exit code with no trace. It hit the msvcrt job consistently and the ucrt job sometimes, which suggested a CRT difference that turned out not to matter. Sjors could not reproduce under Wine or on Windows 11. The mechanism was found by ryanofsky and Claude in an overnight session from ASan runs and CI logs, and needed knowledge of libgcc's emutls.c and winpthreads' key-destructor loop, not of the library. The commit comment in 608d09a8fec7 that lays out the mechanism with six upstream bug links is the diagnosis artefact; the confirming evidence was the revert test showing both cross jobs fail without the workaround and pass with the newer toolchain. What would have helped is a Windows CI job with a heap checker or ASan enabled from the start, and a fatal error message that says which allocation was double-freed rather than a bare exit code.

Lesson

A thread_local with a destructor depends on the platform runtime ordering two teardown mechanisms correctly; if a project rule forbids the construct, the waiver needs a stronger argument than "modern platforms", and per-thread cleanup is safer as a heap object with an explicitly registered destructor.

Notes

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom crash
Worst effect observed crash
Seen in functional-test
Scope production
Trigger Shutdown (shutdown)
Phase any
Side both
Component toolchain
Root cause kind assumption-violated
Fix kind config
Detection ci-intermittent
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master no
Released no
Confidence high