Skip to content

test(rest): add fake server for REST scan planning#1494

Merged
zeroshade merged 2 commits into
apache:mainfrom
Revanth14:test/rest-scan-planning-fake-server
Jul 20, 2026
Merged

test(rest): add fake server for REST scan planning#1494
zeroshade merged 2 commits into
apache:mainfrom
Revanth14:test/rest-scan-planning-fake-server

Conversation

@Revanth14

Copy link
Copy Markdown
Contributor

Part of #1178.

Summary

  • Add a concurrency-safe in-process fake REST scan-planning server.
  • Model config discovery, plan submission, polling, cancellation, and task fetching.
  • Support deterministic response sequences, nested fan-out, duplicate task handles, request capture, storage credentials, and configurable failures.
  • Exercise the existing low-level REST scan-planning client against independently authored wire payloads.

Scope

This is test infrastructure only. It does not change Catalog.PlanFiles, scan-task decoding, planning modes, Java integration, or documentation.

Testing

  • go test ./...
  • go test -race ./catalog/rest/...
  • go vet ./catalog/rest/...
  • golangci-lint run --timeout=10m
  • git diff --check

@Revanth14
Revanth14 requested a review from zeroshade as a code owner July 18, 2026 14:06

@tanmayrauth tanmayrauth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mechanics are solid — this drives the real planning client, not a stub. The gap is coverage: the fake already models the fetch-path 404s but skips a few production error paths it's built to exercise. Details inline.

Comment thread catalog/rest/scan_planning_fake_test.go
Comment thread catalog/rest/scan_planning_fake_test.go
@Revanth14

Copy link
Copy Markdown
Contributor Author

Mechanics are solid — this drives the real planning client, not a stub. The gap is coverage: the fake already models the fetch-path 404s but skips a few production error paths it's built to exercise. Details inline.

@tanmayrauth
Thanks for the review. I’ve addressed the inline coverage gaps by adding tests for terminal planning statuses, plan-submission 404 mappings, and config-provided prefix routing.

Let me know if any changes needed.

@Revanth14
Revanth14 requested a review from tanmayrauth July 18, 2026 19:38

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — fake REST scan-planning server

Test-only change (4 files, +2349/-0). Drives the real rest.Catalog scan-planning client (phases 0–5) against an in-process fake. I pulled the branch and ran the full gate locally in addition to CI.

Verification

Check Result
go build ./... exit 0
go vet ./catalog/rest/... clean
gofmt -l (4 files) clean
golangci-lint v2.8.0 (pinned) on planfake + rest 0 issues
go test -race internal/planfake PASS
go test -race -run TestPlanFake ./catalog/rest/ 11/11 groups PASS
go test -race ./catalog/rest/... PASS (no regressions)
git diff --check clean

Wire compatibility checks out: endpoint.String() renders "METHOD PATH" byte-identical to the fake's endpoint constants, and endpointFromString round-trips them, so the capability-discovery tests exercise the real parse path (scenario.go:34-37endpoints.go:108-111).

What's strong

  • Independent wire fixtures. The fake uses hand-authored JSON rather than the production wire structs (scenario.go:18-20, :30-32), so a struct-tag regression can't silently update both sides of a test. TestPlanFakeIndependentWireFixtureContract asserts the fixture shape on its own. This directly targets the epic's highest-risk concern (silent wire drift).
  • Opaque-token routing. parseRoute/pathSegments split EscapedPath() before per-segment url.PathUnescape, so plan IDs with encoded slashes (a%2Fb) and unit-separator namespaces (db%1Fanalytics) survive as single decoded segments — matching real server semantics.
  • Shutdown semantics. panic(http.ErrAbortHandler) avoids synthesizing phantom 200s on close, and Close() shuts the underlying http.Server before httptest.Server.Close() so a handler blocked in io.ReadAll can't hang cleanup. Both are covered (TestServerGatedResponseAbortsOnServerClose, TestServerCloseAbortsHandlerBlockedReadingRequestBody).
  • Concurrency. Single mutex + close/recreate broadcast channel for WaitForPollCount, with the count incremented before a gated wait so observers can synchronize. A 50-goroutine race test confirms it.
  • Defensive copying of scenario/requests/headers/bodies throughout, with mutation tests.
  • Fixtures document Java ContentFileParser conventions (typed partition values, hex-encoded binary bounds).

One scope point (non-blocking)

The epic's Phase 6 sketch (#1178) described the fake as "implementing the four endpoints over an in-memory table, running the same local planning code server-side (so parity tests are meaningful)", plus a RUN_INTEGRATION_TESTS=1 suite against the Java iceberg-rest-fixture image.

This PR instead ships a scenario-driven mock (canned JSON), which is arguably a better fit for client conformance — it gives precise control over retries/503 backoff/terminal statuses/expiry/fanout/idempotency headers. Two consequences worth tracking as follow-ups so the epic's acceptance criterion isn't lost:

  1. The local-vs-remote parity test isn't enabled by this fake (it doesn't actually plan).
  2. Without the Java-fixture integration suite, nothing automatically guarantees the hand-authored fixtures match real Java output (mitigated here by authoring to Java conventions).

Both are consistent with the PR's stated scope, and #1178 explicitly invites refining the phase breakdown — just want to confirm they're captured as follow-up issues.

Nits (optional)

  • server_test.go:514-518: the wg.Add(1)/go func(){ defer wg.Done() … } loop could use Go 1.25 wg.Go(...) (gopls modernize hint; not flagged by golangci-lint).
  • nextResponse increments the per-key count even for missing/exhausted keys — intentional and documented ("requests received"), noting only so it isn't mistaken for a bug.

Nicely done — thorough, idiomatic, and well-tested.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Test-only change; all CI green (rat/license, build, ubuntu+macos on go1.25/1.26, Spark 3.5 and 4.0 integration). Verified locally with -race tests, golangci-lint v2.8.0 (0 issues), go vet, gofmt, and git diff --check — all clean. Detailed notes in the prior review comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants