IPC stability bugs

Bug 93

IPC processes abort when the monotonic clock steps backwards, with Cap'n Proto older than 1.2

Status
Open, fix under review
Symptom
Crash; worst effect seen: crash
Where seen
Bitcoin Core unit tests
Trigger
Normal use; no special event is needed
First reported
2026-09-25 (first report)
Fixed
Not yet
Fix
mp#369
In released binaries
No; the code was in source releases only
Defect ever on master
Yes

Summary

A program using libmultiprocess, such as bitcoin-node with IPC or the ipc_tests unit tests, aborts with kj/timer.c++:120: failed: expected newTime >= time ... can't advance backwards in time if the Linux CLOCK_MONOTONIC clock steps back, even by microseconds, while the event loop is waiting. The check that throws is in Cap'n Proto's kj library before version 1.2.0. It was seen once in Bitcoin Core's ASan CI job, which uses Ubuntu's Cap'n Proto 1.0.1 package. Release binaries are built with Cap'n Proto 1.2.0 or later from depends and are not affected; self-built binaries linked against a distribution's Cap'n Proto (every Ubuntu release through 26.04 ships 1.1 or older) are. A workaround in libmultiprocess, mp#369, is in review.

What happened

Root cause

Each time the kj event loop wakes up, it reads CLOCK_MONOTONIC and calls TimerImpl::advanceTo(). In Cap'n Proto before 1.2.0 that function checks, on Linux, that time did not go backwards: KJ_REQUIRE(newTime >= time, "can't advance backwards in time") { return; }. The check is recoverable, but kj's default exception handler turns it into a thrown exception, which escapes libmultiprocess's EventLoop::loop() and terminates the process.

The monotonic clock is supposed never to go backwards, but in practice it occasionally steps back slightly, especially in virtual machines; capnproto#2261 shows the same error from a QEMU VM. Cap'n Proto 1.2.0 replaced the check with time = std::max(time, newTime), which keeps the previous time.

Fix

No fix is merged yet. mp#369 first caught the exception around the event loop's wait and retried it. Following ryanofsky's review suggestion (2026-10-01, written with Claude), it now installs a kj::ExceptionCallback for the duration of EventLoop::loop() that logs a warning and returns for this one error, so kj's own { return; } path keeps the previous time, as 1.2.0 does. The callback is compiled only for Cap'n Proto versions before 1.2.0. A new mpclocktest, suggested by @Sjors, overrides clock_gettime() to make the clock go back 10 ms while the loop waits; the PR reports that it fails on Cap'n Proto 0.9.2 without the callback and passes with it.

That version then failed libmultiprocess's Bitcoin Core ASan CI job. kj requires an ExceptionCallback object to live on the stack and checks this in its constructor, but ASan's detect_stack_use_after_return option moves local variables to a heap-allocated "fake stack", and kj only skips the check when kj itself is built with ASan. ryanofsky posted a fix (2026-10-14): exclude loop() from ASan instrumentation and move its body into a separate function, and outlined an upstream kj change that would make the check work in such builds.

Next steps

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern CI, toolchain and platform: Stale dependency bug

  • Prevented by nothing practical
  • Earliest detector a unit test (did not exist then)
  • Diagnosis easy
  • Fix design moderate
  • Debugging aids a clear assertion message
  • Rust no

Would Rust have prevented it? No. A dependency treating a backward monotonic clock reading as fatal is a wrong assumption about the platform, not memory or thread safety; Rust's std::time::Instant had the same problem and now saturates instead of panicking, which is the same fix Cap'n Proto made. More on Rust

Could it have been prevented?

Not practically from libmultiprocess. The check that throws is inside Cap'n Proto's kj timer, which assumed CLOCK_MONOTONIC never decreases on Linux; it was fixed upstream (capnproto#2296, Cap'n Proto 1.2.0) more than a year before Bitcoin Core CI hit it. The only lever was the minimum supported Cap'n Proto version, which has to stay low for distribution packages: every Ubuntu release through 26.04 ships 1.1 or older, as @maflcko pointed out. The general rule is that a monotonic clock can still step back slightly, and code that compares readings should clamp rather than assert.

Could it have been found sooner?

Only by chance. The failure needs the host clock to step back while an event loop is waiting, which a CI job sees rarely; nothing in Bitcoin Core or libmultiprocess CI forced it. Knowing the upstream fix, a check of which supported Cap'n Proto versions still contain the old behaviour would have flagged the exposure, since the upstream issue and fix were public. The test added with the workaround (mp#369), which overrides clock_gettime() so the clock goes back on demand, is the kind of fault injection that makes it deterministic.

What made it hard to debug?

Little. The message names the check, the file and the two times, and a search for it finds the upstream issue; ryanofsky identified it two days after the report. The fix was harder: the first design caught the exception after kj had already consumed epoll events, and the cleaner design, a kj::ExceptionCallback on the loop thread, ran into kj's requirement that the callback live on the stack, which ASan's detect_stack_use_after_return breaks when kj itself is not built with ASan.

Lesson

Treat a monotonic clock as "almost" monotonic: clamp backward steps instead of asserting, and when a dependency gets that wrong, check which versions your users actually build against before relying on the upstream fix.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom crash
Worst effect observed crash
Seen in unit-test
Scope production
Trigger Normal use (ordinary-path)
Phase any
Side eventloop
Component capnproto
Root cause kind assumption-violated
Fix kind workaround
Detection ci-intermittent
Needs a peer that is not libmultiprocess C++ no
Needed a sanitizer to see no
Defect ever on master yes
Released source
Confidence high