IPC stability bugs

Bug 68

Windows child processes hang at shutdown because closing a socket dropped queued messages

Status
Fixed
Symptom
Hang; worst effect seen: hang
Where seen
Branch-build
Trigger
The process closes its own connection while the peer is still there
First reported
2025-05-02 (first report)
Fixed
2026-07-17
Fix
mp#274
In released binaries
Yes
Defect ever on master
Yes

Summary

On Windows, when a process destroyed a Connection over a socketpair, the Cap'n Proto Release messages still queued for the peer were discarded. Cap'n Proto sends a Release message when a process drops its last reference to a remote object; without it, the peer never freed its ProxyServer objects, and child processes spawned over IPC hung at shutdown. It was found while porting libmultiprocess to Windows and never affected a released or supported platform. Fixed in mp#274, which prepared the library for non-Unix platforms, by flushing pending writes before closing the stream; not yet in Bitcoin Core's subtree.

What happened

Root cause

Connection::~Connection() (src/mp/proxy.cpp) reset m_rpc_system and let m_stream be destroyed, relying on the socket close to deliver whatever the RPC system had queued. Two things break that on a Windows socketpair: closing a socket there discards unsent data, and the peer of a socketpair receives no disconnect event when the other end closes. The peer therefore depends entirely on the Release messages generated when local ProxyClient capabilities are destroyed to drop the matching server objects. With those messages lost, the peer's ProxyServers and the objects they wrap stay alive, "preventing the remote side from freeing ProxyServer resources and shutting down cleanly" (commit message). On unix the fix's commit message says the flush "doesn't seem to be needed because data is sent more reliably".

The code has closed the stream this way since the library's initial commit; nothing needed the flush until the Windows port.

The code

Connection::~Connection() in src/mp/proxy.cpp, as of 0012411ccc6f^:

Connection::~Connection()
{
    // ...
    // Try to cancel any calls that may be executing.
    m_canceler.cancel("Interrupted by disconnect");

    // Shut down RPC system first, since this will garbage collect any
    // ProxyServer objects that were not freed before the connection was closed.
    // ...
    m_rpc_system.reset();

    // ProxyClient cleanup handlers are in sync list, and ProxyServer cleanup
    // handlers are in the async list.

Resetting m_rpc_system releases the local capabilities, which sends Cap'n Proto Release messages to the peer. The socket is then closed when m_stream is destroyed with the Connection, with no step that makes sure pending data goes out first. On Windows, closing a socket discards unsent data, and the other end of a socketpair gets no disconnect event, so the Release messages were the only thing that could tell the peer to free its ProxyServer objects; without them the peer cannot shut down cleanly.

The fix shuts down the write side after resetting the RPC system, so data pending in the socket is sent instead of discarded when it closes (0012411ccc6f; comments and the noexcept(false) declaration omitted):

@@ -121,6 +121,31 @@ Connection::~Connection()
     m_rpc_system.reset();

+    try {
+        m_stream->shutdownWrite();
+    } catch (const kj::Exception& e) {
+        if (e.getType() != kj::Exception::Type::DISCONNECTED &&
+            e.getType() != kj::Exception::Type::UNIMPLEMENTED) throw;
+    }

A peer that already closed makes shutdownWrite() throw DISCONNECTED on macOS and FreeBSD, hence the filter. With libcapnp as a dynamic library on macOS the catch clause did not match kj::Exception at all, so b883fe1e5276 replaced it with KJ's own mechanism:

@@ -134,16 +134,23 @@ Connection::~Connection() noexcept(false)
-    try {
+    KJ_IF_MAYBE(exception, kj::runCatchingExceptions([&]() {
         m_stream->shutdownWrite();
-    } catch (const kj::Exception& e) {
+    })) {

A shut-down write side ends the stream gracefully, so the pending Release messages reach the peer before the socket is closed, and the peer frees its ProxyServer objects even though it never sees a disconnect.

Fix

0012411ccc6f calls m_stream->shutdownWrite() after resetting the RPC system, so pending writes are flushed before the stream is destroyed, and declares ~Connection() noexcept(false) to follow KJ's pattern for exceptions in destructors.

The new call caused two failures of its own, both fixed before merge:

Both failures showed up in the CI of the Windows support PR mp#231 (notes, 2026-04-15/16).

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern Process spawning, file descriptors and signals: Relying on a platform behavior nobody stated

  • Prevented by a design rule
  • Earliest detector other (did not exist then)
  • Diagnosis moderate
  • Fix design moderate
  • Debugging aids logging, CI artifacts
  • Rust no

Would Rust have prevented it? No. Dropping a socket in Rust closes it the same way; whether queued bytes survive the close is an operating-system property the language does not model. More on Rust

Could it have been prevented?

Partly. ~Connection has ended a connection since 2019 by resetting the RPC system and letting the stream be destroyed, which on unix delivers the queued Cap'n Proto Release messages and an EOF as a side effect of close(). The rule the fix applies is that a protocol-level goodbye is sent explicitly, with shutdownWrite() before the stream goes away, rather than left to what the operating system does with a closing socket's buffer (0012411ccc6f). That rule was not written down anywhere in the library, and there was no reason to look for it while every supported platform flushed on close; it became visible only when the Windows port found that a socketpair there discards unsent data and gives the peer no disconnect event, so the lost Release messages were the peer's only signal to free its ProxyServer objects. A reviewer in 2019 could reasonably have asked "what guarantees the release messages are sent?", but the honest answer is that a teardown written for one operating system's semantics was ported to another's.

Could it have been found sooner?

Only by running the existing tests on the platform where the assumption fails, which is what happened: the port was being tested by hand on Windows and the child processes hung at shutdown, noted in ryanofsky's push comment on bitcoin#32387 and reported working four days later. No unix test could have seen it, and no Windows CI existed for the library until the port itself. The one-line fix then had to be found wrong twice by CI on mp#231: macOS and FreeBSD return ENOTCONN from shutdown(SHUT_WR) when the peer already closed, and Bitcoin Core's macOS job with a dynamic libcapnp and REDUCE_EXPORTS did not match catch (const kj::Exception&) at all, because the exception's typeinfo had different addresses in the library and the binary (b883fe1e5276). Each of those was caught by the platform matrix on the first run, which is the detector this class of bug has.

What made it hard to debug?

The symptom was a child process that never exited on a platform the library had never run on, in the middle of a port that was fixing many things at once; no issue was filed and the fix cannot be dated closer than the two pushes. The mechanism needed two facts about Windows sockets that unix experience argues against, and the IPC debug log ("IPC server destroy ..." lines that never appeared for the peer's objects) is the kind of evidence that points at unsent releases rather than a hang in the peer's own code. The follow-on failure is the one worth remembering: a catch clause that silently fails to match across a dynamic-library boundary is invisible in the source and shows up only as an uncaught exception on one platform's CI, which is why the fix moved to KJ's own runCatchingExceptions, the mechanism the library already used elsewhere for the same reason.

Lesson

Say goodbye explicitly: a protocol that depends on the last messages before a close must send and flush them itself, because what an operating system does with a closing socket's buffer is not part of any portable contract; and when a fix depends on exception matching, remember that RTTI-based catch can fail across shared-library boundaries.

Notes

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom hang
Worst effect observed hang
Seen in branch-build
Scope production
Trigger Process closes its own connection (local-disconnect)
Phase shutdown
Side both
Component connection
Root cause kind assumption-violated
Fix kind local-fix
Detection manual-test
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released binary
Confidence low