Skip to content

fix(mcp): bound repeated tool rediscovery - #2989

Merged
Astro-Han merged 4 commits into
apache:mainfrom
XonkelX:codex/issue-2981-bound-discovery
Aug 17, 2026
Merged

fix(mcp): bound repeated tool rediscovery#2989
Astro-Han merged 4 commits into
apache:mainfrom
XonkelX:codex/issue-2981-bound-discovery

Conversation

@XonkelX

@XonkelX XonkelX commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Bound consecutive MCP Tool rediscovery attempts when a server emits tools/list_changed for every tools/list. The manager now stops after three superseded attempts, keeps the previous callable snapshot, reports the hostile notification pattern, and remains removable through config sync.

Fixes #2981

Verification

  • npm --workspace @maka/mcp test — 58 passed
  • npm run lint
  • npm run format:check
  • npm run check:third-party-notices
  • npx biome check packages/mcp/src/index.ts packages/mcp/src/__tests__/manager.test.ts
  • Root npm run build reaches an unrelated existing packages/runtime-host/src/server/session-transcript-pager.ts:249 TS7006 error on main; the affected MCP workspace builds and typechecks in its test command.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — repeated hostile discovery notifications now produce a bounded refresh error instead of an infinite loop
  • No

Generated with Codex; reviewed and submitted by the human contributor of record.

@XonkelX
XonkelX marked this pull request as ready for review August 14, 2026 04:48

@M4n5ter M4n5ter left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this bounds only notifications that arrive while the same tools/list request is still in flight; it does not bound the response-followed-by-notification loop described in #2981.

supersededAttempts is local to refreshToolLoop(). Once a refresh completes, refreshState is cleared and the next notification creates a new loop with the counter reset to zero. A server that sends list_changed just after every tools/list response can therefore continue rediscovery indefinitely.

The regression fixture currently sends and awaits the notification before returning the response:

https://github.com/maka-agent/maka-agent/blob/909a2e86778c0da5fb7964906205eea2bcc7c797/packages/mcp/src/__tests__/manager.test.ts#L947-L954

That guarantees the favorable state.pending === true ordering and does not reproduce the issue's "after every response" timing. I changed only the fixture to schedule sendToolListChanged() after returning the response; the test then observed 127 tools/list calls instead of 3 in about 675 ms.

The test also enables the hostile behavior only after the initial sync() has settled, and tests removal only after the retry-limit error has already occurred. It therefore does not verify the two other contracts requested in #2981: initial sync settling and configuration removal preempting an active rediscovery storm.

Could we keep the retry/suppression state at the connection-generation level instead of inside one refresh promise? After reaching the limit, further notifications for that generation should remain suppressed until a defined recovery boundary, such as a quiet period, explicit refresh, or reconnect. The regression test should schedule the notification after each response and attempt removal while rediscovery is still active.

Generated-by: Codex
Signed-off-by: Oniel Alejo Feliz <197416079+XonkelX@users.noreply.github.com>
Generated-by: Codex
Signed-off-by: Oniel Alejo Feliz <197416079+XonkelX@users.noreply.github.com>
@XonkelX
XonkelX force-pushed the codex/issue-2981-bound-discovery branch from 909a2e8 to f6d9cdd Compare August 14, 2026 16:03

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for moving the suppression state to the connection-generation lifetime. That addresses the reset identified in the previous review, but the bound still uses notification timing and count rather than actual rediscovery passes.

A slow server can still bypass the bound: if each tools/list takes longer than the one-second quiet period and then emits list_changed, every notification resets consecutiveSignals. With a 1.2-second response delay, rediscovery continued indefinitely without producing the frequency error.

The opposite timing also causes a regression. MCP notification handlers are dispatched concurrently, so three notifications arriving during one in-flight refresh increment the counter three times. The third marks the generation as suppressed, and refreshToolLoop() then discards the successful definitions it already received. In a reproduction where the server changed from echo to replacement, the manager reported changed too frequently and permanently retained the stale echo snapshot.

Could we count actual rediscovery passes at the per-generation scheduler instead? Notifications received during an active request should coalesce into one pending pass, and the quiet period should begin after a refresh completes. A delayed-response regression and a three-notification burst regression would cover both sides of the invariant.

AI-assisted review disclosure: Codex coordinated the source and lifecycle analysis, and Claude Opus performed an independent adversarial review. The reported behavior was reproduced against the exact PR head; the human reviewer retains responsibility for the final review decision.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed against #2981.

The 5ms notify-after-list flood does stop at three tools/list calls, and disconnect does not wait on an in-flight refresh. Removal can still settle. Keeping the previous snapshot is the right call; disconnecting would open a reconnect churn path.

Two holes remain, so I am not approving yet.

A third list_changed inside the 1s window is discarded without starting a refresh (refreshToolsAfterNotification, consecutiveSignals >= 3). A legitimate server that notifies three times during startup batch registration loses that last change and stays on changed too frequently until a >1s quiet gap or a manual refreshTools. refreshTools has no production caller.

The quiet window is measured from the last notification, not from a finished rediscovery. If tools/list takes longer than 1s (the list timeout is 15s) and then emits list_changed, consecutiveSignals resets every time. That is the original loop with a longer period. The new tests only pin the 5ms path.

Also: this branch currently reports no CI checks.

AI-assisted review: Grok 4.6, opencode-go/deepseek-v4-flash:max, and ark-coding-plan/glm-5.3 each reviewed the PR independently. I walked refreshToolsAfterNotification against a 5ms flood, a 100ms legitimate triple, and a 1100ms slow loop. Unverified by me: I did not run the MCP suite.

@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for tackling the rediscovery loop — the direction is right and test 1 is a genuine regression test. I ran the suite and probed two edge cases the tests don't cover; both surfaced issues worth resolving before merge.

Conclusion: PASS with two mandatory-handling P2s (both verified by reproduction).

P2-1 — the bound can be bypassed by a slow hostile server; the rediscovery loop continues indefinitely.
The counter only accrues when consecutive notifications are ≤1s apart (now - lastSignalAt > 1_000 resets it). I reproduced: with a 1.2s delay on tools/list and notify-per-list enabled, tools/list grew to 6 calls in 6s with no suppression and no error — every notification resets the counter to 1, so it never reaches 3. The issue's pattern ("notify on every tools/list") is only bounded for fast notifiers; at one-list-per-RTT the loop persists. Consider bounding the refresh rate independently of notification spacing (e.g., trailing debounce), or at least documenting this as a known limit.

P2-2 — three legitimate notifications within 1s are misclassified as hostile: spurious error and discarded refresh results.
I reproduced against a fully normal server (no notify-per-list): three rapid notifyToolListChanged() calls produced status.error = 'tool list changed too frequently during discovery', and none of the three valid changes was applied to the snapshot — the refreshed result is discarded by the suppressed check, and the error doesn't clear until the next notification. A legit server doing 3 quick changes (progressive tool registration, batched config load) freezes the tool list in its pre-change state with a misleading error. The count (3) and window (1s) have no protocol or engineering basis.

Optional nits (P3): the suppression error sticks until the next notification (no auto-recovery while the server is quiet); test 2 ("config removal preempts in-flight rediscovery") already passes on main, so the checklist claim "tests fail without the change" holds only for test 1; suppressed notifications still fire status update + emit per notification; the public refreshTools reset has no production callers.


AI-assisted review disclosure: this review was produced with AI assistance (pi review subagent on ollama-cloud/deepseek-v4-flash, read-only). The subagent ran the PR suite (59 pass) in a throwaway worktree, ran test 1 against main's manager code (confirmed red), and reproduced both P2s with probe fixtures. The debounce alternative is a suggestion, not a requirement. Please weigh these findings with your own judgment.

中文摘要(AI 辅助审查)

结论:PASS(两个 P2 必须处理或明确延后)。两个 P2 均已实证复现:① 慢速 hostile server 可绕过 bound——计数仅当相邻通知 ≤1s 才累计,1.2s 延迟 + notify-per-list 时计数每次重置为 1、永远达不到 3,rediscovery 循环以每 RTT 一次的速率无限持续,PR 声称修复的核心场景只对快速通知成立;② 误伤——1 秒内 3 次合法通知被判 hostile,产生虚假 error 且三次有效变更全部被丢弃(工具列表冻结在变更前状态),计数 3 与窗口 1s 无任何协议或工程依据。建议改用 trailing debounce(限速而非判定 hostile,约 15 行 vs 当前 160+ 行)。P3 可选:suppression error 粘滞不自愈、test 2 在 main 上本已通过(checklist 声明不实)、suppressed 后仍逐通知 emit。

@coderabbitai

coderabbitai Bot commented Aug 16, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 4d1e7535-0cbd-4ee0-a1e1-96e010a2160c

📥 Commits

Reviewing files that changed from the base of the PR and between 342caf4 and e6c0b1f.

📒 Files selected for processing (2)
  • packages/mcp/src/__tests__/manager.test.ts
  • packages/mcp/src/index.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/mcp/src/tests/manager.test.ts

Included review availability: Your plan includes up to 3 reviews per rolling hour; 2 remain after this review.


📝 Walkthrough

What this PR solves

MCP servers can emit tools/list_changed after every tools/list. This caused unbounded rediscovery, prevented sync() from settling, and blocked server removal.

The manager now:

  • Bounds notification-driven rediscovery after three superseded attempts.
  • Coalesces refresh requests through shared refresh state.
  • Retains the previous callable tool snapshot when refresh fails or is suppressed.
  • Reports the repeated-notification pattern.
  • Clears refresh state during disconnect and transport failure.
  • Preserves configuration synchronization and server removal.

Design assessment

This change extends the existing McpClientManager refresh and notification path. It does not create a parallel public API or change exported declarations.

The added state coordinates concurrent refreshes, counts superseded attempts, retains the last valid snapshot, and resets state during connection lifecycle changes. This complexity is necessary for the requested behavior. The solution remains coherent, but the state and race handling increase review cost.

The expanded tests cover remote and stdio transports, refresh coalescing, cancellation, stale bindings, reconnect and close races, schema validation, error handling, and configuration reconciliation. No safe deletion or simplification is evident without weakening regression coverage.

Validation and risks

Reported validation includes MCP tests, lint, formatting, notices, and targeted Biome checks. The affected MCP workspace passes its test command. The root build remains blocked by an unrelated existing TS7006 error. Required check status is unverified here.

The review identified two concrete behavioral risks:

  • The counter resets after one second. A hostile server that takes about 1.2 seconds per list request can continue rediscovery indefinitely.
  • Three legitimate notifications within one second can trigger suppression, discard refreshed results, and retain the previous snapshot.

Additional risks include persistent suppression errors, status updates and emissions from suppressed notifications, and a public refreshTools reset with no production callers.

Review-relevant risks

The current diff changes user-visible tool availability and diagnostic behavior by suppressing refreshes and retaining the previous callable snapshot. Material changes in this area require independent human review under repository policy.

The current diff changes MCP synchronization, reconnection, transport-closure handling, and server removal behavior. Material changes in integration and operational behavior require independent human review under repository policy.

No exported declaration changes are reported. No licensing, release, or governance effect was identified in the current diff.

Required checks remain unverified unless confirmed directly. The person performing the merge must review the final diff. A maintainer makes the final determination.

Walkthrough

MCP tool discovery now bounds repeated tools/list_changed refreshes, tracks refresh state per connection, clears state during lifecycle transitions, and reports excessive list changes. Tests expand remote and stdio transport coverage.

Changes

MCP tool discovery refresh

Layer / File(s) Summary
Refresh state and notification entry points
packages/mcp/src/index.ts
The manager tracks notification-triggered work and pass limits. Tool-list notifications use the bounded refresh flow and share active refresh work.
Bounded refresh execution and lifecycle cleanup
packages/mcp/src/index.ts
Refresh loops enforce the three-pass limit, preserve successful snapshots, report frequent list changes, and clear state during disconnect, close, failure, and transport closure.
Notification fixtures and end-to-end coverage
packages/mcp/src/__tests__/manager.test.ts
Remote fixtures support repeated notifications and controlled timing. Tests cover remote and stdio transport behavior, refresh concurrency, cancellation, validation, errors, and configuration reconciliation.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🔵 Low · up to e6c0b

Concurrent refresh requests may reset the safeguard against repeated tool rediscovery, allowing a hostile server notification pattern to continue longer than intended and causing excess refresh work. The PR is mergeable with explicit owner awareness or follow-up on preserving the bound for concurrent callers.

Suggested reviewers: jackwener, m4n5ter, me2seeks

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Ai Use Disclosure ⚠️ Warning The description says “Generated with Codex” but selects neither required declaration and gives no scope; 3 of 4 PR commits have valid trailers, while 342caf4 has none. In “Human ownership and AI attribution” in CONTRIBUTING.md, select the generative-tooling declaration, name Codex and its scope, add Generated-by: Codex to 342caf4, and preserve it through squash or amend.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main change: bounding repeated MCP tool rediscovery.
Description check ✅ Passed The description covers the problem, linked issue, verification, checklist, behavior change, and AI contribution.
Linked Issues check ✅ Passed The changes bound repeated rediscovery, preserve the tool snapshot, and test configuration removal, meeting [#2981].
Out of Scope Changes check ✅ Passed The implementation and expanded tests directly support the linked issue objectives without unrelated changes.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
packages/mcp/src/index.ts (1)

287-294: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

refreshTools clears the suppression bound unconditionally.

Every public refreshTools call replaces refreshNotificationState with refreshPasses: 0, suppressed: false. A refresh that is already in flight for the same client and generation keeps running through refreshToolLoop, which then re-reads the new state object and restarts the budget from zero. A caller that polls refreshTools while a hostile server notifies removes the bound.

If the reset is intended only as an operator escape hatch, restrict it to the case where no refresh is active:

♻️ Restrict the reset to idle connections
-    entry.refreshNotificationState = {
-      client,
-      connectionGeneration,
-      refreshPasses: 0,
-      suppressed: false,
-    };
+    const active =
+      entry.refreshState?.client === client &&
+      entry.refreshState.connectionGeneration === connectionGeneration;
+    if (!active) {
+      entry.refreshNotificationState = {
+        client,
+        connectionGeneration,
+        refreshPasses: 0,
+        suppressed: false,
+      };
+    }
     return this.startToolRefresh(serverId, entry, client, connectionGeneration);
packages/mcp/src/__tests__/manager.test.ts (1)

917-925: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Both transports repeat the same beforeToolList closure.

Lines 917-924 and Lines 940-947 are identical. Hoist one closure above createServer and pass it to both createProtocolServer calls.

♻️ Share one closure
   let toolListClockAdvance = () => {};
+  const beforeToolList = async () => {
+    toolListClockAdvance();
+    const gate = nextToolListGate;
+    if (!gate) return;
+    nextToolListGate = undefined;
+    gate.markStarted();
+    await gate.waitForRelease;
+  };

Then use beforeToolList, at both call sites.

As per path instructions: "Flag concrete cases where code can be deleted or simplified."

Also applies to: 940-948

Source: Path instructions


ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 2c83fdb9-2fa2-4a3c-8f72-5fe4332142e8

📥 Commits

Reviewing files that changed from the base of the PR and between 2632b85 and 342caf4.

📒 Files selected for processing (2)
  • packages/mcp/src/__tests__/manager.test.ts
  • packages/mcp/src/index.ts

Included review availability: Your plan includes up to 3 reviews per rolling hour; 2 remain after this review.

Comment thread packages/mcp/src/index.ts Outdated
Comment thread packages/mcp/src/index.ts Outdated
Generated-by: Codex
Signed-off-by: Oniel Alejo Feliz <197416079+XonkelX@users.noreply.github.com>
@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for the follow-up — the two P2s from the earlier review are addressed, and the new tests are solid. I re-verified on the updated head.

Conclusion: the blockers are resolved. Ready to merge once CI runs green (it currently needs approval to run).

Verified on the updated head (e6c0b1f5):

  • P2-1 (slow-hostile bypass) — fixed. The quiet-period reset (TOOL_REFRESH_QUIET_PERIOD_MS / lastRefreshCompletedAt) is gone; refreshPasses now only accrues and resets on connection change. The updated test advances the clock after the tools/list response (1.1s spacing) and still asserts the bound — the slow-notifier variant is now covered.
  • P2-2 (discarded valid results) — fixed. refreshToolLoop now applies the snapshot (replaceToolSnapshot + update) before the suppressed check, so the latest valid tool list is published even when rediscovery is suppressed. The new "publishes the latest valid snapshot before suppressing rediscovery" test pins exactly that.
  • Burst coalescing — covered. "coalesces a burst of legitimate list-changed notifications" confirms notifications arriving during an in-flight refresh are merged without error.
  • Ran the workspace suite locally on the updated head: 62/62 pass (up from 59).

Remaining observations (non-blocking):

  • P3: a legitimate server doing 4 independently completed refreshes (each notification arriving after the previous refresh finished) still hits the pass cap and surfaces the frequency error. The harm is now limited — the latest snapshot is applied, so the tool list stays fresh — but the error is spurious. If you want to close it fully, a trailing-debounce (rate-limit refreshes per unit time instead of counting passes) would distinguish "hostile notify-per-list" from "legit rapid changes" without a counter. Fine to defer.
  • P3: the PR branch's CI is in action_required state — it needs approval before the checks run. Worth getting that approved so the suite (including the new e2e tests) is verified before merge.

Nice addition: the error-sanitization tests (no secret leakage, control-character stripping in status errors) are a good hardening beyond the original scope.


AI-assisted review disclosure: this follow-up was produced with AI assistance (pi review subagent on ollama-cloud/deepseek-v4-flash, read-only). I verified the fixes against the updated diff, traced the pass-accounting logic, and ran the workspace test suite locally (62/62). The remaining P3 is analysis of the current semantics, not a reproduced failure. Please weigh these findings with your own judgment.

中文摘要(AI 辅助审查)

结论:两个 P2 阻塞项已解决,可以合并(需先批准并跑通 CI)。已修复:① 慢速 hostile 绕过——删除了 quiet-period 重置逻辑,refreshPasses 只增不减,测试改为在 tools/list 响应后推进时钟(1.1s 间隔)仍断言 bound;② 有效结果被丢弃——refreshToolLoop 现在先应用 snapshot 再抛 frequency error,新测试验证 suppressed 时最新有效工具列表仍被发布;③ burst 合并——新增测试验证 in-flight 刷新期间到达的通知被合并、无 error。本地跑工作区测试 62/62 通过(比之前多 3 个)。残余(不阻塞):合法 server 做 4 次独立完成的刷新仍会触发频率 error(后果已减轻——最新快照被应用,工具列表保持最新),如需彻底关闭可改用 trailing debounce 限速而非计数;另外分支 CI 处于 action_required 状态,需批准运行。加分项:新增的错误消息脱敏测试(不泄露 secret、控制字符清理)。

@Astro-Han

Copy link
Copy Markdown
Contributor

LGTM. Both P2s from review are resolved, the new tests cover the slow-notifier and snapshot-preservation cases, and all CI checks are green (test_workspaces, typecheck, test, windows_recovery, windows_baseline). Merging.

AI-assisted review disclosure: the review was AI-assisted (pi subagent on ollama-cloud/deepseek-v4-flash); the merge decision is the human maintainer's.

@Astro-Han
Astro-Han merged commit ec2d286 into apache:main Aug 17, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mcp: tool discovery livelocks against a server that notifies list_changed after every tools/list

3 participants