test(e2e): add daily secret-free Playwright suite [murmur:agentic-data-stack/agentic-data-stack-issue-12]#18
Merged
dustinhealy merged 15 commits intoJul 4, 2026
Conversation
Adds a browser-level E2E suite that verifies the stack works (not just boots): login, an agent chat that queries ClickHouse via the real MCP server, Langfuse trace creation, and the feedback->Langfuse score path. Runs daily + manual only (never on PRs) so the smoke test stays the PR-time launch gate while this heavier suite catches :latest image drift. A local OpenAI-compatible mock (e2e/mock-llm/server.js) fakes only the inference with adaptive tool-calling (finds the ClickHouse query tool by name pattern, runs SELECT 1, echoes the result). The MCP server, ClickHouse, and Langfuse stay real, so no ANTHROPIC_API_KEY and no token spend while still producing real traces + scores. - docker-compose.e2e.yml: mock-llm service + CONFIG_PATH override - e2e/librechat.e2e.yaml: prod config + MockLLM endpoint - e2e/: Playwright project (config, fail-fast env, Langfuse polling client, auth setup, and librechat/langfuse/roundtrip/scoring specs) - .github/workflows/e2e.yml: schedule + workflow_dispatch, de-duped daily-failure tracking issue Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The auth setup lives in e2e/setup/, outside the top-level testDir of ./specs. Playwright only matches testMatch against files under a project's testDir, so the setup project discovered zero tests, never wrote the storage state, and every spec failed to open the missing playwright/.auth/user.json (ENOENT) on a local run. Set an explicit testDir per project (setup -> ./setup, chromium -> ./specs). `playwright test --list` now shows the [setup] test ahead of the specs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Top-level test.use({ baseURL }) throws "did not expect test.use() to be
called here" under some Playwright loader setups. The Langfuse spec
targets a different origin than the configured baseURL anyway, so use
absolute Langfuse URLs directly and remove the override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a Troubleshooting note: the "did not expect test.describe()/test.use() to be called here" error is a stale/mismatched @playwright/test install, resolved by `rm -rf node_modules && npm ci` (and avoiding a global playwright shadowing the local runner). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docker compose up --wait now blocks until Langfuse passes its health probe, preventing ERR_CONNECTION_REFUSED when the E2E suite starts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Langfuse v3 redirects to / after sign-in rather than /project/.... Check that we left /auth/ instead of asserting a specific destination; each test navigates to its own URL immediately after. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
The API-only login (POST /api/auth/login) captured the refresh cookie but not the localStorage the LibreChat SPA also needs, so browsers restored from that storage state redirected to /login and every authed spec failed at the text-input composer. Drive the real login form in auth.setup.ts and snapshot the session after the authenticated app shell renders (nav-new-chat-button) — the canonical Playwright auth recipe, capturing cookies + localStorage. Also correct the login selectors to the real markup: the submit control is a "Continue" button (no login-button testid), and the login fields are role=textbox named Email/Password. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Langfuse web runs DB migrations on cold start and takes 2-3 minutes to answer /api/public/health, but start_period: 30s + 10x5s retries marked it unhealthy after ~80s, failing `docker compose up --wait` (and the smoke test) on a normal cold start. Bump start_period to 180s. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
180s was still short: CI logs show langfuse-web was mid-Prisma-migration at ~227s when Docker marked it unhealthy, failing `up --wait`. Raise the start_period to 300s (with 12x10s retries, ~420s total headroom under the 600s wait-timeout). A passing check flips healthy immediately, so the larger grace doesn't slow faster starts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The langfuse/langfuse:3 image ships neither wget nor curl, so the wget-based healthcheck reported unhealthy for the full grace window even though the app logged "All migrations applied" and "Ready" — failing `up --wait`. Switch to a node http probe (node is guaranteed present), mirroring the python3 probe used for clickhouse-mcp. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…host Root cause of the persistent 'langfuse-web is unhealthy': Next.js standalone binds to process.env.HOSTNAME (Docker sets it to the container id), so Langfuse listens on the container IP — a loopback probe is refused. External access via the published port worked, so the app looked 'Ready' while the healthcheck failed for the entire grace window, failing `docker compose up --wait`. Probe http://$HOSTNAME:3000/api/public/health with node (the image has no wget/curl). Verified locally: 200 -> healthy, 503 -> unhealthy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
Author
| Re: the langfuse-web healthcheck comment (thread resolved) — the healthcheck was the cause of the red smoke-test, and the root cause was deeper than `start_period`: the `langfuse/langfuse:3` image ships no wget/curl, and Next.js standalone binds to `$HOSTNAME` (the container id), so a `localhost` probe was refused even after the app was Ready. Fixed in a8777fb by probing `http://$HOSTNAME:3000/api/public/health` with node. smoke-test now passes in ~3m. (Replied here rather than in-thread because an unrelated pending review is blocking review-reply creation.) |
LibreChat rotates the refresh-token cookie on every /api/auth/refresh call, which the SPA fires unconditionally on every page mount. The shared storage-state snapshot from auth.setup.ts was captured once, so the first spec to load an authenticated page burned that snapshot's refresh token; every later spec in the (serial) run replayed the stale cookie, got a 401 on refresh, and was bounced to /login. Add a page fixture that re-persists storage state after each test so the rotated cookie carries forward. Langfuse's own NextAuth cookies are stripped from the snapshot: LibreChat and Langfuse share the "localhost" cookie jar (cookies aren't port-scoped), so without this a signed-in Langfuse session would leak into later specs and break langfuse.spec.ts's assumption that each test starts unauthenticated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three of the picker/button selectors in lib/librechat.ts had drifted
from the running LibreChat UI, silently no-oping (per the file's own
"best-effort" selector strategy) rather than throwing:
- selectMockEndpoint only clicked the MockLLM endpoint entry, which
merely opens its model submenu; the model itself ("mock-model")
still needed a click to finalize the selection, so chats silently
fell back to the default ChatGPT endpoint and failed with
"No key found".
- enableClickHouseMcp's loose /mcp/i button match also matched the
sidebar's persistent "MCP Settings" panel toggle, which sits earlier
in the DOM and won tryClick's `.first()` over the composer's actual
per-message "MCP Servers" toggle, so the tool was never attached to
the message.
- giveThumbsUp looked for "thumbs up"/"good response"/"helpful" button
text, but the current UI's positive-feedback button is titled "Love
this" and only opens a reason submenu; feedback isn't submitted to
the backend until a reason chip is picked.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The traces-view assertion looked for a heading matching /traces/i, but the page's actual heading text is "Tracing" — "Traces" is a nav link underneath it, not a heading, so the assertion never matched even though the page loaded correctly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
|
Ran the suite locally against a freshly-pulled current librechat:latest (v0.8.7) plus real MCP/ClickHouse/Langfuse via the e2e override: 8/8 pass, including roundtrip (MockLLM → real MCP → ClickHouse → Langfuse trace with generation + tool spans) and scoring (a user-feedback=1 score confirmed on the trace via the public API). The suite is functionally sound against the deployed image. Approved 🚀 |
dustinhealy
approved these changes
Jul 4, 2026
Member
Author
| Thanks for the thorough local run against `librechat:latest` (v0.8.7), @dustinhealy — that closes the one gap I flagged earlier: the round-trip and scoring flows (and the endpoint/MCP-picker + 👍 selectors) could only be validated against a live UI, which I couldn't drive from CI. 8/8 green against the deployed image with real MCP/ClickHouse/Langfuse is the confirmation I wanted. Appreciate the review 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #12.
What
Adds a browser-level Playwright E2E suite that verifies the stack actually works — not just that it boots (that's the smoke test's job). It covers the real user flows: login, an agent chat that queries ClickHouse via the real MCP server, Langfuse trace creation, and the feedback → Langfuse score path.
Runs daily + manual only (never on PRs), so
smoke-test.ymlstays the PR-time launch gate while this heavier suite catches:latestimage drift on a schedule.Secret-free by design
e2e/mock-llm/server.jsis a ~200-line OpenAI-compatible mock (Node built-ins only) that fakes only the inference. The MCP server, ClickHouse, and Langfuse stay real — so noANTHROPIC_API_KEY, no token spend, no frontier-model flakiness, while a chat still produces a real Langfuse generation observation + real MCP→ClickHouse tool spans and exercises scoring.Adaptive tool-calling: the mock inspects the request's
tools[], finds the ClickHouse query tool by name pattern (never a hardcoded name — LibreChat rewrites MCP tool names), emits atool_callsdelta runningSELECT 1, then echoes the tool result in a final answer.Components
docker-compose.e2e.yml— override addingmock-llmand repointing LibreChat viaCONFIG_PATH=/app/librechat.e2e.yaml(new mount path, prod config untouched).e2e/librechat.e2e.yaml— prod config + aMockLLMcustom endpoint (fetch: false), retainingendpoints.agents, MCP access, and theClickHouse-LocalMCP server.e2e/— Playwright project: chromium,baseURL:3080,setupauth project (API login → storage state), fail-fast.env-backed config, a Langfuse public-API client that polls (ingestion is async), and thelibrechat/langfuse/roundtrip/scoringspecs..github/workflows/e2e.yml—schedule+workflow_dispatch; brings the stack up with--wait, installs Playwright, runs the suite, uploads the report on failure, always tears down, and opens/updates a de-duped tracking issue on scheduled failure (mirrorssmoke-test.yml).Verification done here
tsc --noEmitclean;playwright test --listloads all 7 tests; all YAML validated;npm cilockfile committed.The endpoint/MCP-picker and 👍/👎 selectors are best-effort (LibreChat markup drifts) with strict outcome assertions, and are flagged in-code + in
e2e/README.mdforplaywright codegenconfirmation against the live UI — as called out in the issue's unknowns.Note
The issue's advertising comment for a third-party "QA Boutique" tool was disregarded — it's unsolicited promotion from a non-collaborator, not part of the task.
Created via Murmur
🤖 Generated with Claude Code