Skip to content

fix(mcp): let an eval cell use a searchable tool that is not active yet - #408

Merged
code-yeongyu merged 5 commits into
mainfrom
fix/eval-auto-activate-tools
Jul 27, 2026
Merged

fix(mcp): let an eval cell use a searchable tool that is not active yet#408
code-yeongyu merged 5 commits into
mainfrom
fix/eval-auto-activate-tools

Conversation

@code-yeongyu

@code-yeongyu code-yeongyu commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Problem

In session 019fa397-dbbc-7a05-baa2-6a68852ff384 an eval cell ran:

tool.mcp_computer_use_click({'x':826,'y':371,'button':'left','clickCount':1})

and got back:

Tool mcp_computer_use_click is registered but inactive.
Active tools: read, bash, apply_patch, todo, ...

The assistant then emitted empty text and stalled (msg 22). The user had to intervene — "no use eval just pure tool call plz" (msg 25) — and eval was abandoned for the rest of the session.

The tool was registered, just not active. Moments later the model called tool_search, which activated those exact same tools, and the identical work succeeded. Search-mode MCP exposure registers the whole catalog but keeps only directTools active, and only tool_search could promote from it. That asymmetry — tool_search may promote, a code-mode cell may not — is the bug.

Change

executeTool() resolved a name against the active set and, on a miss, consulted the definition registry only to choose between inactive_tool and unknown_tool before throwing.

Add registerLazyToolActivator(activator). When a name resolves to a registered-but-inactive tool, registered activators run before the throw; one returning true means it actually activated the tool, and execution proceeds.

Core deliberately does not decide eligibility. builtin/mcp registers an activator whose eligible set is the tier-B searchable catalog and nothing else.

Why not blanket activation

A one-line "activate anything registered" change is unsafe. _toolDefinitions also holds:

Kind Why it must stay inactive Anchor
Permission-denied tools deny * rules are applied by deactivating permission-system/index.ts:96-100, config.ts:71-84
MCP list_changed additions documented rug-pull defense mcp/notifications.ts:1-9
Removed-tool tombstones re-registered specifically to stay inactive mcp/service.ts:436-439
look_at / read_video capability gates (model lacks vision/video) look-at/index.ts:37-48

None are in the searchable catalog, so all stay inactive. Activation also routes through the tier-B activate() path rather than the generic active-set setter, so stubSwap still swaps the registered stub for the full definition — marking a stub active would have returned stub output while the active-set assertion looked correct.

An activator returning false preserves inactive_tool byte for byte; unknown_tool is untouched. execute-tool.test.ts and look-at-extension.test.ts pass unmodified.

Evidence

Real AgentSession through pi.executeTool, same tool and coordinates as the failing session:

BEFORE active: read
RESULT: [{"type":"text","text":"clicked 826,371"}]
AFTER active: read,mcp_computer_use_click
look_at REFUSED: inactive_tool
unknown REFUSED: unknown_tool
FINAL active: read,mcp_computer_use_click

The capability-gated tool is still refused and never enters the active set.

Gates

  • test/mcp/ + execute-tool + look-at-extension + extensions-runner + permission/: 849 passed (63 files)
  • packages/senpi-codemode: 403 passed, 6 skipped (pre-existing interpreter skips)
  • npx tsgo --noEmit: exit 0
  • npm run check (root): exit 0

Both criteria were captured RED first: the policy test failed on the absent module, and the session-wiring test failed with pi.registerLazyToolActivator is not a function.

Review

Reviewed by momus, which returned REJECT / SAFE-ONLY-IF on the original blanket-activation design. Every blocker was applied: catalog-scoped eligibility, tier-B activate() for stub-swap, opt-in registration, and tests covering the refusal paths rather than only the happy path.

Plan: .omo/plans/eval-auto-activate-tools.md


Summary by cubic

Fixes stalled eval cells by lazily activating eligible searchable MCP tools just in time for eval. Activation is opt-in via activateInactiveTool (enabled only for eval), and gated, tombstoned, and permission‑denied tools remain inactive.

  • Bug Fixes

    • executeTool() now, when activateInactiveTool is true, runs registered lazy activators for registered‑but‑inactive tools and executes on success.
    • Restores the setActiveToolsByName docblock after introducing _activateLazyTool.
  • New Features

    • Adds registerLazyToolActivator(activator) to the Extension API and ExecuteToolOptions.activateInactiveTool. The codemode wrapper enables it for eval; other callers keep inactive_tool/unknown_tool unchanged.
    • builtin/mcp registers an activator that promotes only tier‑B searchable tools via its activate() path, preserving stub‑swap and catalog rules. Extension loader/runner persist and replay activators across bindCore().

Written for commit 07ab203. Summary will update on new commits.

Review in cubic

…on demand

executeTool() resolved a tool name against the ACTIVE set and, on a miss,
consulted the definition registry only to pick between `inactive_tool` and
`unknown_tool` before throwing. An eval cell naming a registered-but-inactive
tool was therefore stuck, even when the host could trivially activate it.

Add registerLazyToolActivator(activator). When a name resolves to a registered
but inactive tool, registered activators run before the throw; one returning
true means it has actually activated the tool and execution proceeds.

Core deliberately does NOT decide eligibility. The definition registry also
contains permission-denied tools, MCP list_changed additions held inactive as
rug-pull defense, removed-tool tombstones, and capability-gated tools such as
look_at and read_video. Ownership stays with the registering extension, so an
activator returning false preserves `inactive_tool` byte for byte and
`unknown_tool` is untouched.

Activators are stashed on the extension and replayed after bindCore(), the same
way registerRemovedToolHint already handles factories running before core is
bound.

Plan: .omo/plans/eval-auto-activate-tools.md
Session 019fa397 ran `tool.mcp_computer_use_click({'x':826,'y':371})` in an eval
cell and got "Tool mcp_computer_use_click is registered but inactive". The
assistant then emitted empty text and stalled, and the user had to abandon eval
for the rest of the session. Moments later tool_search activated those exact
tools and the same work succeeded: search-mode exposure registers the whole
catalog but keeps only directTools active, and only tool_search could promote.

Register a lazy activator whose eligibility is the tier-B searchable catalog and
nothing else, so a cell may reach any tool tool_search could already promote.
Everything else stays inactive: permission-denied tools, list_changed additions
(rug-pull defense), removed-tool tombstones, and capability-gated look_at /
read_video are all absent from that catalog.

Activation goes through the tier-B activate() path rather than the generic
active-set setter, so stubSwap still replaces the registered stub with the full
definition; marking a stub active would have returned stub output while looking
correct in the active set.

Plan: .omo/plans/eval-auto-activate-tools.md
Lazy activation ran for every executeTool caller, so an unrelated extension
calling a searchable MCP tool would activate it exactly like an eval cell did.
Only code-mode needs this: a cell names tools directly and cannot run
tool_search first.

Gate it behind ExecuteToolOptions.activateInactiveTool, off by default. The
codemode executeTool wrapper is the only caller that sets it, so every other
extension keeps the `inactive_tool` contract for the very same tool.

Plan: .omo/plans/eval-auto-activate-tools.md
The new _activateLazyTool landed between setActiveToolsByName's docblock and its
declaration, leaving the docblock attached to the wrong member.
@code-yeongyu
code-yeongyu merged commit 1d4a80e into main Jul 27, 2026
6 checks passed
@code-yeongyu
code-yeongyu deleted the fix/eval-auto-activate-tools branch July 27, 2026 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant