Skip to content

fix(hooks): consolidate_background never exits after a successful cycle — unclosed psycopg pool spins a core at 98% CPU indefinitely #398

Description

@cdeust

Context

Observed 2026-08-08/09 on macOS 25.6 (Darwin), Python 3.14.4, plugin
hypermnesia-mcp/4.17.2: the background consolidate worker completed its
cycle successfully and then never exited
, spinning one core at ~98% CPU for
9 hours with PPID=1 (orphaned when its spawning session ended).

Measured on the live process before it was terminated:

PID 28135  PPID 1  ELAPSED 09:08:23  %CPU 98,5
  UTIME 528:38.78   STIME 4:42.00        ← burnt in user space, not I/O

Last three meaningful lines of ~/.claude/methodology/consolidate.log
(21 MB; last write 2026-08-08 21:15:56, i.e. ~9 h before the process was
killed — it produced nothing after this point):

[bg-consolidate] finished status=ok duration_ms=1966750      ← 32.8 min, ok
[bg-consolidate] wiki: stub purged=0 classifier purged=451 pending_total=3656
Exception ignored while calling deallocator <function ConnectionPool.__del__ ...>:
  File ".../psycopg_pool/pool.py", line 126, in __del__
  File ".../psycopg_pool/_acompat.py", line 152, in gather
  File ".../python3.14/threading.py", line 1133, in join
PythonFinalizationError: cannot join thread at interpreter shutdown

lsof on the live process showed all three PostgreSQL sockets in state
CLOSED — the work was done and written; only the pool's worker thread was
still alive.

Root cause

The store's connection pools are never closed explicitly; teardown is left to
the garbage collector, and on Python 3.14 __del__ cannot join a thread during
interpreter finalization.

Chain, verified in the tree at 1ba020a6:

  1. mcp_server/hooks/consolidate_background.py:88asyncio.run(handler(args))
  2. mcp_server/handlers/consolidate.py:240store = _get_store(); the store
    is never closed, in the handler or in the hook.
  3. mcp_server/hooks/consolidate_background.py:120sys.exit(0 if status == "ok" else 1)
  4. The psycopg pools own non-daemon worker threads, so the interpreter waits
    for them instead of exiting. ConnectionPool.__del__ fires, calls
    gatherthread.join(), and raises PythonFinalizationError. The pool
    worker keeps running; the process spins forever.

PgStore.close() already exists (mcp_server/infrastructure/pg_store.py:1371)
and does exactly the right thing — closes _interactive_pool, _batch_pool,
then self._conn. No hook calls it.

This is the failure this repo's own rule predicts: an implicit teardown path
that no test exercises.

Scope of the pattern (to verify, not asserted)

Five of seven hooks call asyncio.run with no close/aclose anywhere in the
file:

compaction_checkpoint.py        asyncio.run=1  close=0
consolidate_background.py       asyncio.run=1  close=0   ← reproduced above
ingest_codebase_background.py   asyncio.run=1  close=0
post_tool_capture.py            asyncio.run=1  close=0
session_lifecycle.py            asyncio.run=1  close=0
pipeline_impact_bump.py         asyncio.run=1  close=1
session_start.py                asyncio.run=1  close=4   ← raw sqlite3/pg conns, not the store

Only consolidate_background is proven to leak (a live process was
observed). Whether the other four construct a pooled store on their path must be
checked per hook before any claim is made about them; session_start's four
close() calls are on raw sqlite3/pg connections, not on PgStore.

Note this is backend-dependent: the leak is in the psycopg pool path, so
SQLite-backed installs (the plugin default per PRIVACY.md) are not expected to
reproduce it. That expectation needs a test, not an assumption.

Acceptance criteria

  1. Every hook that constructs a store closes it on every exit path —
    success, handler exception, and sys.exit — via try/finally or a context
    manager. Asserted per hook, including the exception arm.
  2. A test asserts the worker process actually exits: spawn the hook against
    a temporary store, assert the process is reaped within a bounded wait and
    that its exit status is the expected one. A test that only asserts the return
    value would have passed throughout this incident.
  3. No non-daemon thread survives main(); asserted at the end of the hook run,
    not by inspection.
  4. Defence in depth, independent of (1): the worker enforces a wall-clock
    deadline and exits non-zero on expiry, and detects a dead parent (PPID == 1)
    and stops. Each arm has its own test — neither may be the sole mechanism, so
    that a future leak degrades into a logged timeout rather than a 9-hour spin.
  5. The behaviour is verified against the backend that reproduces it (psycopg
    pool). If SQLite genuinely cannot reproduce it, that is asserted by a test,
    not stated in prose.
  6. Scoped mutation run on the changed files: zero surviving non-equivalent
    mutants, or each survivor documented as equivalent.
  7. Completion Ledger in the PR.

Out of scope

  • Reducing the consolidate cycle's 32.8 min duration — it completed with
    status=ok; this issue is about the process not dying afterwards.
  • Log rotation for consolidate.log (21 MB) — separate concern, worth its own
    issue.

Reproduce

Run the worker against a PostgreSQL-backed store and watch the process after it
logs finished status=ok:

python -m mcp_server.hooks.consolidate_background
# then, from another shell:
ps -o pid,ppid,etime,%cpu -p <pid>

Expected today: the process stays alive at ~100% CPU after the final log line.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions