Internal RDAP data API + privileged-grant store (backend)#2937
Open
OlegPhenomenon wants to merge 15 commits into
Open
Internal RDAP data API + privileged-grant store (backend)#2937OlegPhenomenon wants to merge 15 commits into
OlegPhenomenon wants to merge 15 commits into
Conversation
The .ee RDAP service must stop reading the registry DB directly and instead consume a registry-side API (lead decision 2026-06-25). This adds the contract + implementation guide for that internal API: - GET /api/v1/internal/rdap/domains/:name (full privileged domain) - GET /api/v1/internal/rdap/registrars/:code (code,name,phone,website) - GET /api/v1/internal/rdap/nameservers/:host (hostname,hostname_puny) - GET /api/v1/internal/rdap/grants/active?subject= (net-new grant store) Authoritative spec lives in the RDAP repo (specs/specs/pending/08-registry-side-rdap-api). No code yet; doc only. Branch off master for review; do not merge to master.
Auth = pre-shared key + IP-allowlist; backend-only (no admin UI this spec); categories/states confirmed; defaults for namespace/touch/delete-date/disclosure.
New machine-to-machine JSON API consumed by the public RDAP service, so RDAP
no longer reads the registry's normalized DB directly. Namespace
/api/v1/internal/rdap/*, auth = pre-shared key + IP-allowlist.
Endpoints:
- GET domains/:name full privileged domain (PII + glue + DNSSEC)
- GET registrars/:code narrow entity {code,name,phone,website}
- GET nameservers/:host thin {hostname,hostname_puny}, DISTINCT-collapsed
- GET grants/active the single authoritative-active privilege grant
- POST grants/:id/touch best-effort last_used_at (204)
Net-new rdap_privilege_grants table + model (active_for_subject resolution,
fail-closed). Secrets-free Serializers::Rdap::Domain (never emits transfer_code/
auth_info/registrant_verification_token). minitest integration + model tests.
Backend only: no admin UI (grants seeded via fixtures/console per the spec).
Without this, a blank/unset shared key makes expected = 'Basic ' and any caller sending an empty Basic credential would authenticate. Reject when the key is not configured.
…rar test structure.sql: only the new rdap_privilege_grants table (CREATE TABLE + sequence + pkey + 3 indexes) and its schema_migrations version — no unrelated churn (the noisy full pg_dump regen was discarded). Validated: db:test:prepare reloads it from scratch and the suite is green (36 runs, 101 assertions, 0F/0E). registrar happy-path test now asserts full-hash equality (proves the exact narrow 4-key shape, drops the assert_nil deprecation).
Companion to the RDAP spec 10-rdap-issued-api-token. RDAP owns no database,
so the opaque API tokens it mints persist here and are reached over the
internal RDAP data API — the same seam that already serves domains,
registrars, nameservers and privilege grants.
Implements the six pinned token operations RDAP's registry client calls,
all keyed by the token_hash (RDAP's keyed HMAC-SHA-256 digest of the raw
token; the raw token and the HMAC secret never reach the registry):
POST /api/v1/internal/rdap/tokens create (store hash + metadata)
GET /api/v1/internal/rdap/tokens/active find active by hash (404 if
revoked/expired/unknown — no
caller-visible distinction)
GET /api/v1/internal/rdap/tokens?subject= list a subject's tokens
POST /api/v1/internal/rdap/tokens/revoke revoke one (idempotent, 204)
POST /api/v1/internal/rdap/tokens/revoke_all kill-all-by-subject (-> count)
POST /api/v1/internal/rdap/tokens/touch best-effort last_used_at only
- rdap_api_tokens table (token_hash unique, subject+revoked_at index); no
privilege field — authorization stays 100% grant-driven. Migration +
hand-spliced structure.sql (schema_format :sql).
- Model RdapApiToken: active_by_hash / for_subject scopes, idempotent revoke!,
touch_last_used! that never moves expires_at (no sliding renewal).
- Fix (existing): add subject + token_hash to filter_parameters — the grants
controller already said 'do not log the subject' (a national id) but nothing
enforced it; now both are filtered out of logs.
- Tests: model + integration (24 new; full internal/rdap suite 60 runs / 158
assertions / 0 failures via docker-images-registry-1).
Contract mirrors the RDAP-side interface (Rdap::Registry::Client TOKEN_OPERATIONS
+ Token value object) so the RDAP RealClient token methods drop straight onto it.
Admin CRUD, inside the registry app, for managing RdapPrivilegeGrant (spec 09-registry-privileged-admin), sitting next to the data it manages. - Migration: add full_name (NOT NULL), legal_basis_ref (NOT NULL) and optional personal_id_code to rdap_privilege_grants; new per-model log_rdap_privilege_grants audit table + creator_str/updator_str columns. - Model: include the Versions concern (immutable paper_trail audit), presence validations for full_name/legal_basis_ref, and a read-time derived expired?/display_status (STATUSES stays active/revoked/suspended). - Controller/views/routes: top-level admin CRUD mirroring admin_users, with distinct suspend/revoke member actions and no destroy (grants end via suspend/revoke only; audit entries have no edit/delete route). - Ability: can :manage, RdapPrivilegeGrant for the admin role only. - personal_id_code is capture-only: never in the index, never in the frozen internal grants serializer, filtered from logs (config.filter_parameters). - The frozen api/v1/internal/rdap/grants_controller#serialize is untouched; active_for_subject and grant_id are unchanged. - Fresh i18n keys + menu link; all 6 fixtures populated for the new columns. - Tests: model (validations, derived expired, revoke-takes-effect), admin integration (CRUD, suspend/revoke, auth gating, audit history, no-delete route, no PII/no missing-translation), serializer no-leak.
…it history
Closes the AC23 gap the tester found: the show page's paper_trail audit
table is built from object_changes, a potential PII vector. The view already
does object_changes.except('personal_id_code'); this test proves it — creates
a grant with the code, makes an audit-generating change, asserts the value
never appears on the rendered show page.
43 runs / 140 assertions / 0 failures via docker exec docker-images-registry-1.
…int (spec 11) Add an insert-only rdap_access_events table plus the internal write endpoint POST /api/v1/internal/rdap/access-events for recording privileged (result_code 200) RDAP disclosures. - Migration: bigint PK, timestamptz requested_at, snapshot columns (organization_name/accessor_name/category/grant_ref), nullable request_id, explicit created_at (no updated_at); indexes on [domain_name, requested_at], [grant_ref, requested_at], [requested_at]; no FK, no request_id index. - Model RdapAccessEvent: presence validations, create-only immutability (readonly? => !new_record? + before_destroy raise); NOT a paper_trail model. - Controller: three-way outcome (404 unresolved grant / 422 non-200 result_code or unparseable requested_at or validation-false / 204 success) plus an explicit in-action rescue for the 500 path that logs + increments a New Relic failure metric. Grant resolved server-side by uuid-or-id; snapshots taken at write time; eeid_subject/personal_id_code never read or stored. - Route: single explicit post line inside namespace :internal / :rdap. - Tests: model (presence, immutability, no Versions, nullable organization) and integration (AC5-AC17) following grants_test.rb; new fixtures.
Add GET /api/v1/registrant/domains/:uuid/access_events, the registrant-facing
"who accessed my domain data" read endpoint.
- Thin AccessEventsController < Api::V1::Registrant::BaseController: inherits the
Bearer auth (401 on missing/invalid), resolves the addressed domain via
current_registrant_user.domains.find_by(uuid: params[:domain_uuid]) (404 in the
registrant-API error shape, byte-identical for unknown vs other-registrant), and
renders EXACTLY {accessed_at, organization, category} per event.
- RegistrantAccessEventsQuery PORO (app/services): reconstructs the caller's
HALF-OPEN tenure intervals on THIS item_id from the ordered log_domains timeline
(object_changes/object registrant_id), keeps only intervals whose effective
registrant is one of the caller's own direct contact ids, and selects
RdapAccessEvent rows for the domain name whose requested_at falls inside a tenure
interval AND is older than the disclosure cutoff; DESC, capped 100. The per-interval
lower bound excludes prior/later owners and same-name deleted-namesake events.
- Read-time disclosure delay from Setting.rdap_access_transparency_disclosure_delay
with a safe non-zero 5-minute fallback (never 0/real-time); seed + idempotent data
migration create the SettingEntry (integer, group rdap, value 5).
- Route nested under the authenticated registrant domains resource.
- Integration + fixture coverage for historical per-tenure ownership (in-tenure,
prior-owner, later-owner/transfer-instant, two non-contiguous intervals, namesake),
delay filter incl. missing-Setting fallback, DESC+100 cap, PII value/log gates,
read-only, 401, 404 no-leak, empty->200 [].
Commit 9dae4a3 added rdap_api_token_hmac_secret without a trailing newline. The CI test job appends config via 'echo >> config/application.yml' after copying the sample, so the first appended line was concatenated onto rdap_api_token_hmac_secret, producing invalid YAML and a figaro Psych parse error at boot (config/application.rb:18) that aborted the entire test suite before any test loaded. Adding the trailing newline restores CI.
The spec-13 access-transparency fixtures (mytenure/capdomain domains + their log_domains timeline) grew the shared fixture corpus by two john-owned domains, which broke four unrelated shared assertions: - Market-share stats (ReppV1StatsMarketShareTest): moved mytenure/capdomain and their create log events onto a dedicated test_registrar (spec13_registrar). serialize_distribution/growth_rate skip any registrar with test_registrar:true, so the two domains no longer shift the Best Names distribution/growth numbers. No stats assertions changed. - Registrant domain counts (RegistrantApiV1DomainsTest, RegistrantApiDomainsTest): the corpus legitimately gained two john-owned domains, so bumped the hardcoded expected totals/counts by +2. - Admin domain-history CSV export (DomainVersionsTest#test_download_domain_history): regenerated the golden CSV to include the eight new log_domains rows (registrar now reads Spec13 Registrar for the two create rows). No production code changed; only fixtures and the specific assertions the fixture growth broke. The spec-13 access-events privacy tests are untouched.
The spec-13 mytenure/capdomain test_registrar domains are excluded from the market-share domains breakdown (serialize_growth_rate_result filters test registrars), but calculate_market_share divides by the total across ALL registrars, so the two extra domains enlarge the denominator. Update the growth-rate percentage expectations accordingly (prev Good 60.0; today Good 25.0 / Best 50.0). Raw domain counts are unchanged.
…r owner's accesses M1 (HIGH, found by security + code review): RegistrantAccessEventsQuery#build_intervals opened the caller's live tenure interval with `current_start ||= rows.first&.first` whenever the LIVE domain.registrant_id was one of the caller's contacts. When the reconstructed walk had NOT itself established the caller's current tenure (current_start nil — the acquiring version row is missing, or its object_changes was empty so carry-forward kept the prior owner), that fabricated current_start = the domain's FIRST-EVER version and emitted [first_version, now), spanning every prior owner's tenure and disclosing their rdap_access_events (police/CERT lookups) to the current caller — the cross-registrant leak R5c/R7/N3 forbid. Fail closed instead: when the caller currently owns the domain but the walk did not establish their current tenure start, anchor the open interval to the most recent version the timeline actually shows the caller as effective registrant (backward scan), and if the caller never appears as registrant in the recorded history, emit NO open interval — under- disclose rather than risk exposing a prior owner's accesses. The normal case (walk established current tenure) is unchanged. Regression coverage (this path was untested — all prior fixtures had well-formed timelines): domains(:orphan) — john owns it live but the timeline only records prior owner jane; asserts jane's prior-tenure event is excluded (fail closed, no interval). domains(:partial) — john appears earlier (2021) then it's handed back to jane (2022) with no john re-acquire row; asserts only events at/after john's last effective-registrant version (2021) are disclosed, jane's pre-2021 event excluded. S2: order the timeline query by (:created_at, :id) for a deterministic tie-break so two same-instant registrant changes cannot order nondeterministically and invert an interval. S3: add item_type: 'Domain' to the timeline where so the (item_type, item_id) index's leading column is used and the scope is explicit (correctness-neutral). Fixture bookkeeping: the two new john-owned spec13_registrar domains add +2 to john's registrant domain counts and +2 spec-13 domains to the market-share denominator; updated the specific count/percentage assertions with comments, and regenerated the golden domain_versions.csv for the added log_domains rows.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Internal RDAP data API + privileged-grant store (backend only)
The public .ee RDAP service must stop reading the registry DB directly and instead consume a registry-side API (lead decision 2026-06-25). RDAP already shipped the consumer side (a client interface + an in-memory mock). This PR is the real API behind that mock.
Spec (authoritative): lives in the RDAP repo —
specs/specs/completed/08-registry-side-rdap-api/(proposal, requirements,api-contract.md,registry-grounding.md). In-repo copy:doc/api/v1/rdap.md.Endpoints (new namespace
/api/v1/internal/rdap/*)GET …/domains/:nameGET …/registrars/:code{code,name,phone,website}onlyGET …/nameservers/:host{hostname,hostname_puny}(DISTINCT-collapsed)GET …/grants/active?subject=POST …/grants/:id/touchlast_used_at(204)What's in it
Api::V1::Internal::BaseController+Api::V1::Internal::Rdap::{Domains,Registrars,Nameservers,Grants}Controller.authenticate_shared_keypattern;secure_compare; fail-closed when the key is unset). mTLS is a later prod-hardening step. ENV:rdap_internal_api_shared_key,rdap_internal_api_allowed_ips.RdapPrivilegeGrantmodel + migration (categoriespolice/cert/ria/eis_internal, statusesactive/revoked/suspended, server-authoritativeactive_for_subject). The registry had no authority-access concept before.Serializers::Rdap::Domain— never emitstransfer_code/auth_info/registrant_verification_token(does NOT reuseSerializers::Repp::Domain(sponsored: true), which leakstransfer_code). Returns raw disclosure flags (union) — RDAP applies disclosure policy, not the registry.Scope
Backend only — no admin UI this PR (grants seeded via fixtures/console; admin CRUD is a later spec).
Tests
minitest integration + model tests. Verified locally inside the
docker-imagesstack (docker exec docker-images-registry-1): 36 runs, 101 assertions, 0 failures, 0 errors.structure.sqlupdated with only the new table and validated by reloading from scratch (db:test:prepare).doc/api/v1/rdap.mdwere already resolved with the RDAP lead; the namespace/URL is the one thing easiest to change if you prefer a different home.