feat: evaluate openadapt-flow on WAA (demonstrate-then-replay + hybrid-as-agent) with cost-guarded dry-run#265
Merged
Merged
Conversation
…d-as-agent) with cost-guarded dry-run Wire openadapt-flow (the demonstration compiler) into the WAA benchmark harness with two eval modes, both scored by WAA's own task verifier: - demonstrate-then-replay (openadapt_evals/flow/replay_runner.py): compile ONE demo into a bundle and replay it via openadapt-flow's WindowsBackend against the WAA in-guest /screenshot + /execute_windows server (~0 model calls). Emits per-task WAA-verified success, structural rung fire rate, model calls, wall-clock, and halt/heal events. - hybrid-as-agent (openadapt_evals/flow/hybrid_agent.py): HybridFlowAgent implements BenchmarkAgent so the existing runner can drive it -- compiled replay first, computer-use agent fallback only on a detected halt, gated by a SpendLedger. Directly comparable to a pure agent baseline on the same tasks. Cost model + hard guardrails (openadapt_evals/flow/cost.py, stdlib-only): per-run $ cap, total $ ceiling, per-task token cap, abort-on-repeated-billing -error (mirrors openadapt-flow's SpendLedger; the prior $40-70 uncapped-run incident is why these are mandatory). scripts/eval_flow_on_waa.py is dry-run by default -- prints the plan + cost for N tasks and never provisions Azure, starts a VM, or spends money. waa_cost_estimate.py gains a flow-aware --tasks/--mode/--dry-run path. openadapt-flow is an optional [flow] extra; all replay/hybrid code lazy-imports it, so the cost model and dry-run work with zero flow dependency and the whole integration is mock/dry-run testable locally (30 new tests, no Azure, no VM). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
…he hybrid agent's paid fallback The observation handed to the first fallback act() predates the compiled replay's VM mutations (replay drove the WAA server directly, bypassing the adapter). A no-op WAIT forces the runner to fetch current state before the paid computer-use agent takes its first real step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
…low harness Adds a LOCAL Parallels environment so the demonstrate-then-replay eval mode runs at $0 against the Parallels Win11 VM (native Apple-Silicon virt) with the SAME runner + metrics as WAA -- for de-risking before spending Azure $. - openadapt_evals/flow/parallels_env.py: ParallelsSession (snapshot -> run -> revert, never stops/deletes the VM), reuses openadapt-flow's ParallelsVM (prlctl wrapper) + win_agent server (PR #95) via launch_agent; once up it exposes the SAME /screenshot + /execute_windows contract WAA does, so the identical WindowsBackend replay path works unchanged. Parallels supplies the ENVIRONMENT but not WAA's tasks/verifiers -- built-in trivial Notepad/ Calculator tasks + ground-truth in-guest verifiers are supplied here. Opt-in gated (OPENADAPT_PARALLELS=1), skipped by default. - scripts/eval_flow_on_waa.py: --env {waa,parallels}. Parallels dry-run shows $0 (local), the opt-in gate status, and the snapshot-safe note; --live is gated on the opt-in var and refuses otherwise. Local-$0 (Parallels) vs cloud-$ (WAA/Azure) is labelled everywhere. Heavy imports (openadapt_flow) stay lazy and the VM + replay are injectable, so the whole path is mock/dry-run testable (10 new tests) with no prlctl, no VM, no mutation. Note: launch_agent/win_agent ship in openadapt-flow PR #95 (still open) -- a live local run depends on that; the dry-run + tests do not. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Wires openadapt-flow (the demonstration compiler) into the existing WAA benchmark harness so we can evaluate it on WindowsAgentArena. Two eval modes, both scored by WAA's own task verifier (not self-report). Ships the code + a cost model. No Azure is provisioned and no money is spent — everything here is mock/dry-run testable locally; the live run is gated on the maintainer's explicit go.
Two eval modes
openadapt_evals/flow/replay_runner.py) — the paradigm-correct eval for a demonstration compiler. Compile ONE demo into a bundle and replay it via openadapt-flow'sWindowsBackendagainst the WAA in-guest/screenshot+/execute_windowsserver (the SAME contract WAA exposes — no adapter layer). Per-task metrics: WAA-verified success, replay self-report, structural rung fire rate, model calls (~0 on healthy replay), wall-clock, halt/heal events.openadapt_evals/flow/hybrid_agent.py) —HybridFlowAgentimplementsBenchmarkAgent, so the existingevaluate_agent_on_benchmarkrunner drives it exactly like any agent. Compiled replay first; computer-use agent fallback only on a detected halt, gated by a sharedSpendLedger. Directly comparable to a pure agent baseline on the same tasks / same verifier.Cost model + hard guardrails
openadapt_evals/flow/cost.py(stdlib-only, no flow/VM/key needed): estimates $ for N tasks (Azure VM-hours + agent-arm token cost) and enforces a per-run $ cap, total $ ceiling, per-task token cap, and abort-on-repeated-billing-error — mirroring openadapt-flow's benchmarkSpendLedger(the prior $40-70 uncapped-run incident is why these are mandatory).scripts/eval_flow_on_waa.pyis dry-run by default — prints the plan + cost and never provisions Azure/starts a VM/spends money.scripts/waa_cost_estimate.pygains a flow-aware--tasks/--mode/--dry-runpath (legacy table unchanged with no args).Estimated cost (Claude Sonnet 4, $0.19/hr VM)
(The $6.15 hybrid@154 case trips the default $5 total cap — the guard prints a WARNING and a
--liverun is refused until the cap is raised.)Runnable now vs. needs Azure + $
--dry-runplan/estimate. 30 new tests, zero Azure, zero VM, zero flow dependency.waa-pool: the actual replay/hybrid execution —oa-vm pool-create/pool-wait, compiled bundles per task, then--live. Gated behind--live+ a reachable server; refuses on missing bundles before any network call.Dependency note
openadapt-flowis added as an optional[flow]extra (on PyPI, 0.20.0). All replay/hybrid code lazy-imports it.openadapt_flow.backends.win_agent(PR #95, still open) is not required for the WAA path — WAA already exposes the same in-guest server contract thatWindowsBackendtargets. Verified at API level against the real flow package (WindowsBackendserver_url,RunReportfields) without a VM.Tests
tests/test_flow_waa_cost.py,tests/test_flow_replay_runner.py,tests/test_flow_hybrid_agent.py,tests/test_eval_flow_on_waa_dryrun.py— 30 passing, fully mocked. Existing WAA/mock/cost suites (113 tests) still green; ruff clean on all new/modified files.🤖 Generated with Claude Code
https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ