IPC stability bugs

Bug 46

Python IPC test aborts when libmultiprocess is also installed system-wide

Status
Fixed
Symptom
Crash; worst effect seen: crash
Where seen
Bitcoin Core functional tests
Trigger
Normal use; no special event is needed
First reported
2025-09-17 (first report)
Fixed
2025-09-18
Fix
bitcoin#33420

Summary

The interface_ipc.py functional test aborted at startup with mp/proxy.capnp:0: failed: Duplicate ID @0xcc316e3f71a040fb on machines where libmultiprocess was installed system-wide in the same prefix as Cap'n Proto, because the test's schema import path listed the system include directory before the in-tree one. The test then loaded two copies of the same schema file. The defect was in test code: only the Python test process aborted, no node was involved, and users were not affected. Fixed by bitcoin#33420 ("test: Avoid interface_ipc.py Duplicate ID errors").

What happened

@zaidmstrr ran the new interface_ipc.py on a machine with libmultiprocess installed system-wide, and the test process died right after "Initializing test directory" with terminate called after throwing an instance of 'kj::ExceptionImpl' what(): mp/proxy.capnp:0: failed: Duplicate ID @0xcc316e3f71a040fb. Aborted (core dumped) (bitcoin#33417, 2025-09-17). Removing the system installation made the test pass. ryanofsky reproduced it by pointing the test at an external installation. CI never hit it, since CI machines have no system libmultiprocess.

Root cause

load_capnp_modules in test/functional/interface_ipc.py loads the in-tree mp/proxy.capnp directly with capnp.load(...) and then loads init.capnp and the other Bitcoin Core schemas, whose import "/mp/proxy.capnp" lines are resolved through an imports list. That list was [capnp_dir, src_dir, mp_dir], where capnp_dir is the include directory next to the capnp binary on PATH (for example /usr/local/bin/capnp gives /usr/local/include). If libmultiprocess was installed in that same prefix, the absolute import resolved to the installed proxy.capnp while the direct load had used the in-tree copy. pycapnp's schema loader saw two different files declaring the same file ID and threw, and the uncaught kj exception terminated the interpreter.

Both packages have to share an install prefix, as ryanofsky explained on the fix PR after reviewers who installed only libmultiprocess system-wide could not reproduce it.

The code

Quoted from e9c52272ebd7^. load_capnp_modules in test/functional/interface_ipc.py loads the in-tree mp/proxy.capnp by path, then loads the Bitcoin Core schemas, which import "/mp/proxy.capnp" through the imports search list:

if capnp_bin := shutil.which("capnp"):
    # Add the system cap'nproto path so include/capnp/c++.capnp can be found.
    capnp_dir = Path(capnp_bin).parent.parent / "include"
# ...
src_dir = Path(self.config['environment']['SRCDIR']) / "src"
mp_dir = src_dir / "ipc" / "libmultiprocess" / "include"
imports = [str(capnp_dir), str(src_dir), str(mp_dir)]
return {
    "proxy": capnp.load(str(mp_dir / "mp" / "proxy.capnp"), imports=imports),
    "init": capnp.load(str(src_dir / "ipc" / "capnp" / "init.capnp"), imports=imports),

The search list puts the system include directory first. When libmultiprocess is installed under the same prefix as the capnp binary (for example /usr/local/include/mp/proxy.capnp), the import in init.capnp finds that copy, while the "proxy" entry has already loaded the in-tree copy. pycapnp then has two different files declaring file ID @0xcc316e3f71a040fb, throws "Duplicate ID", and the uncaught kj exception aborts the Python test process.

The fix searches the in-tree directory first (e9c52272ebd7):

@@ -35,7 +35,12 @@ class IPCInterfaceTest(BitcoinTestFramework):
         mp_dir = src_dir / "ipc" / "libmultiprocess" / "include"
-        imports = [str(capnp_dir), str(src_dir), str(mp_dir)]
+        # List of import directories. Note: it is important for mp_dir to be
+        # listed first, in case there are other libmultiprocess installations on
+        # the system, to ensure that `import "/mp/proxy.capnp"` lines load the
+        # same file as capnp.load() loads directly below, and there are not
+        # "failed: Duplicate ID @0xcc316e3f71a040fb" errors.
+        imports = [str(mp_dir), str(capnp_dir), str(src_dir)]

With mp_dir first, import "/mp/proxy.capnp" resolves to the same in-tree file that was loaded directly, so the schema loader sees one file and one ID.

Fix

Commit e9c52272ebd7 reorders the list to [mp_dir, capnp_dir, src_dir], with a comment explaining that the in-tree directory must come first so the imports load the same file as the direct capnp.load() (+6/-1).

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern CI, toolchain and platform: An installed copy found before the tree under test

  • Prevented by a design rule
  • Earliest detector code review (existed then)
  • Diagnosis easy
  • Fix design easy
  • Debugging aids a clear assertion message, a reproduction script
  • Rust no

Would Rust have prevented it? No. Schema import search-list ordering is the same in capnpc-rust or any other loader; the shadowed copy is a configuration bug the language does not see. More on Rust

Could it have been prevented?

Yes, by one rule the fix comment now states: when a file is loaded by explicit path and then reached again through a search list, the directory of that file goes first in the list, so both routes resolve to the same copy. The test loaded the in-tree proxy.capnp directly but put the system include directory ahead of the tree in imports, so the import "/mp/proxy.capnp" lines in the other schemas could find a second copy. Stronger still would be one resolution route only (load every schema through the same search list), so there is nothing to keep consistent. This was knowable when the test was written; it is the same shadowing problem as -I order in a C++ build.

Could it have been found sooner?

By review, which is the only check that sees a search list's order; a reviewer who knows that -I-style lists are searched first to last would ask why the system directory came before the tree. No runtime check in CI could have: Bitcoin Core's CI builds the subtree and never installs libmultiprocess system-wide, and the condition is narrower than that, since Cap'n Proto and libmultiprocess must share an install prefix, which is why Sjors and fanquake could not reproduce it with split prefixes. A CI job that installs both into one prefix, the way a distribution package would, would have failed on the first run, but no such job exists. The test was two weeks old when the first developer with such a machine ran it, so the gap was short anyway.

What made it hard to debug?

Nothing. The exception names the duplicated file ID and the schema, the reporter's own issue already pointed at the import list, ryanofsky reproduced it the same day by pointing the test at an external installation, and the fix was merged the next day. The only obstacle was reviewers failing to reproduce until the shared-prefix condition was spelled out.

Lesson

When the same file can be reached by two routes, an explicit path and a search list, order the search list so it finds the copy you loaded explicitly, or use one route for everything; a search list that starts with system directories silently prefers an installed copy over the tree being tested.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom crash
Worst effect observed crash
Seen in functional-test
Scope test-code
Trigger Normal use (ordinary-path)
Phase startup
Side n/a
Component test-harness
Root cause kind assumption-violated
Fix kind test-only
Detection user-report
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Confidence low