IPC stability bugs

Bug 55

Node memory grows while Stratum v2 mining clients are connected

Status
Open
Symptom
Memory leak; worst effect seen: crash
Where seen
A production node
Trigger
Normal use; no special event is needed
First reported
2025-11-24 (first report)
Fixed
Not yet
In released binaries
Yes (v30.0)
Defect ever on master
Unknown

Summary

bitcoin-node v30.0 memory grew steadily while Stratum v2 mining clients used the IPC mining interface on mainnet, until a node on a 2 GB machine was killed by the kernel's out-of-memory killer after about a day. The cause is not known, and it is still undetermined whether the growth is a Bitcoin Core bug, a consequence of how many block templates clients keep alive, or a client-side problem. Seen with a released v30.0 binary. No fix yet; tracked in bitcoin#33940 ("Memory leak when using IPC mining interface").

What happened

Root cause

Unknown. What has been checked:

Fix

No fix yet. The next step asked for in the issue is a heap profile or a reproducer that grows memory quickly.

Next steps

Timeline

Lessons

What would have prevented this bug, found it sooner, or made it easier to debug. The patterns page groups these lessons across all bugs.

Pattern An in-process interface trusting what now comes from another process: No limit on what a peer can make the server hold

  • Prevented by a design rule
  • Earliest detector a stress test (did not exist then)
  • Diagnosis hard
  • Fix design hard
  • Debugging aids logging, a reproduction script, other
  • Rust no

Would Rust have prevented it? No. Server-side objects whose lifetime a peer controls stay alive under Rust ownership too; bounding what a remote party may hold is a design decision, and the cause here is still undiagnosed. More on Rust

Could it have been prevented?

Unknown, because the cause is unknown; the honest answer is conditional. The one mechanism everyone agrees exists is that a block template handed to a client stays alive on the server, with its transactions pinned, until the client destroys it or disconnects, and nothing bounds how many a client may hold. That is a design rule that was stated only after the fact, as the design issue bitcoin#33899: server-side state whose lifetime a remote peer controls needs a cap or an eviction policy, the same as connection slots or mempool size. If the growth is templates, the rule would have prevented it; if it is something else (allocator retention, a client bug, an unrelated leak), it would not, and ryanofsky's tests on 2025-12-02 and 2025-12-08 showed the server releasing templates promptly on destroy, on dropped references and on disconnect, which points away from the simplest version of the template theory.

Could it have been found sooner?

Only by a soak test that did not exist: a node on mainnet with a real Stratum v2 client attached for hours, with RSS recorded. Nothing in CI runs an IPC client for more than seconds, regtest does not produce the mainnet template churn (a 4 MB template every few seconds through tip changes and mempool updates), and the v30.0 release shipped bitcoin-node before any operator had run it that way. The report came about six weeks after release, from the first people to leave it running on a small machine, which is roughly the earliest the existing process could have produced it.

What made it hard to debug?

It is still open. The reports are RSS plots and an OOM-kill log from mainnet runs; RSS not falling after a disconnect is not evidence of a leak since the allocator keeps freed pages, and no one has produced a heap profile or a reproduction that grows memory in minutes, which is what ryanofsky asked for. Two things would have helped: a heap profiler run (heaptrack) on the affected node, and accounting the node could report itself, such as a count of live templates and the bytes they pin, logged or exposed over RPC, so a reporter can say whether memory tracks templates without a debugger. Without either, the investigation has had to guess at mechanisms and test them one at a time on regtest.

Lesson

When a remote peer decides how long a server-side object lives, bound it, and make the count and size of such objects observable; a memory report from the field is only actionable if the process can say what it is holding on whose behalf.

Classification

The survey's classification of this bug. The explorer charts these fields across all bugs; the method page explains them.

Field Value
Symptom leak
Worst effect observed crash
Seen in production-node
Scope production
Trigger Normal use (ordinary-path)
Phase steady-state
Side server
Component bitcoin-mining
Root cause kind unknown
Fix kind none
Detection user-report
Needs a peer that is not libmultiprocess C++ unknown
Needed a sanitizer to see no
Defect ever on master unknown
Released binary
Confidence low