fix(pg_catalog): stop pg_attribute from evicting schema/catalog OIDs (#389)#393
Merged
sunng87 merged 1 commit intoJul 21, 2026
Conversation
…atafusion-contrib#389) The dynamic catalog tables share one OID counter + cache, and each builder reconciles that cache against the live catalog. `pg_attribute` reconciled by wholesale-replacing the cache: let mut swap_cache = HashMap::new(); ... swap_cache.insert(Table(..), oid) ... *oid_cache = swap_cache; `pg_attribute` enumerates only tables, so `swap_cache` only ever held `Table` keys. The replacement therefore silently deleted every `Catalog`/`Schema` OID -- not because those objects were gone, but because pg_attribute never looked at them. The next `pg_namespace`/`pg_class` query then found a cache miss for the schema and re-allocated it a *different* OID via the counter. A pooled client (DBeaver, JDBC) that resolved a schema OID earlier -- e.g. `pg_class WHERE relnamespace = <oid>` -- now matched nothing, so no tables or columns were listed. Reproduces (before this change) as a schema OID drifting across a single column-metadata query: pg_namespace.oid(public): 16384 -> 16457 (unstable) pg_class rows joining on relnamespace=16384: 0 Fix: reconcile surgically, like `pg_namespace` already does -- drop only stale `Table` entries and leave `Catalog`/`Schema` entries untouched, then refresh the live table OIDs: oid_cache.retain(|key, _| match key { OidCacheKey::Catalog(..) => true, OidCacheKey::Schema(..) => true, OidCacheKey::Table(..) => table_oid_cache.contains_key(key), }); oid_cache.extend(table_oid_cache); pg_attribute already preserved OIDs for the tables it visits (via the `oid_cache.get` memo hit); now it also stops clobbering the OIDs it doesn't. After the fix the schema OID is stable and the join resolves. Adds two regression tests covering the invariant within a single shared SessionContext (the deployment model of the shipped server). Note: this addresses datafusion-contrib#389 only. datafusion-contrib#390 (pg_catalog must keep the fixed namespace OID 11 that the static catalog rows reference) is orthogonal and not changed here.
sunng87
added a commit
to GreptimeTeam/datafusion-postgres
that referenced
this pull request
Jul 21, 2026
…tafusion-contrib#390) The static catalog rows shipped by this crate (the `pg_catalog_arrow_exports/*.feather`, produced by `export_pg_catalog_arrow.sh` from a fresh PostgreSQL) reference the canonical `pg_catalog` namespace OID 11 -- e.g. `pg_type.typnamespace = 11`, `pg_proc.pronamespace = 11`. But `pg_namespace`/`pg_class` assigned the `pg_catalog` schema a dynamically generated OID from the counter (16384+), so no namespace with OID 11 existed. Any client joining a system object to its namespace (`pg_type.typnamespace -> pg_namespace.oid`) got nothing -- concretely, DBeaver could not resolve any column's data type (every column showed as unknown). Resolve OIDs through a single helper so a fixed canonical OID is always honored: fn resolve_oid(key, oid_cache, oid_counter) -> Oid { fixed_oid(key) // 1. fixed canonical OID, if any .or_else(|| oid_cache.get(key).copied()) // 2. previously assigned .unwrap_or_else(|| oid_counter.fetch_add(1, ...)) // 3. new } `fixed_oid` is keyed by `&OidCacheKey` (not just the schema name) and is now consulted at *every* OID-assignment site -- catalogs and tables, not only schemas -- so adding a fixed OID for any object kind (other namespaces, specific catalogs/tables, columns later) is a new match arm in `fixed_oid` and is picked up everywhere automatically. Today only `pg_catalog` (-> 11, below the user range, no collision) is pinned. All eight assignment sites in `pg_namespace`/`pg_class`/`pg_database`/ `pg_attribute` now call `resolve_oid`, removing the duplicated `fixed -> cache -> counter` ladder. Adds a regression test asserting `pg_catalog` keeps OID 11 and that a built-in type (`int4`) joins back to its namespace. The test fails on `master` (`16384 != 11`) and passes with this change. Scope: addresses the pg_catalog OID from datafusion-contrib#390 only. The static rows also reference `information_schema`'s bootstrap OID (13283 in the PG-17 export); that namespace is intentionally left dynamic here -- its composite view-types rarely need to be resolved to a namespace, and 13283 is a non-canonical export artifact. Documented for a possible follow-up. Builds on datafusion-contrib#393.
sunng87
added a commit
that referenced
this pull request
Jul 21, 2026
…) (#394) The static catalog rows shipped by this crate (the `pg_catalog_arrow_exports/*.feather`, produced by `export_pg_catalog_arrow.sh` from a fresh PostgreSQL) reference the canonical `pg_catalog` namespace OID 11 -- e.g. `pg_type.typnamespace = 11`, `pg_proc.pronamespace = 11`. But `pg_namespace`/`pg_class` assigned the `pg_catalog` schema a dynamically generated OID from the counter (16384+), so no namespace with OID 11 existed. Any client joining a system object to its namespace (`pg_type.typnamespace -> pg_namespace.oid`) got nothing -- concretely, DBeaver could not resolve any column's data type (every column showed as unknown). Resolve OIDs through a single helper so a fixed canonical OID is always honored: fn resolve_oid(key, oid_cache, oid_counter) -> Oid { fixed_oid(key) // 1. fixed canonical OID, if any .or_else(|| oid_cache.get(key).copied()) // 2. previously assigned .unwrap_or_else(|| oid_counter.fetch_add(1, ...)) // 3. new } `fixed_oid` is keyed by `&OidCacheKey` (not just the schema name) and is now consulted at *every* OID-assignment site -- catalogs and tables, not only schemas -- so adding a fixed OID for any object kind (other namespaces, specific catalogs/tables, columns later) is a new match arm in `fixed_oid` and is picked up everywhere automatically. Today only `pg_catalog` (-> 11, below the user range, no collision) is pinned. All eight assignment sites in `pg_namespace`/`pg_class`/`pg_database`/ `pg_attribute` now call `resolve_oid`, removing the duplicated `fixed -> cache -> counter` ladder. Adds a regression test asserting `pg_catalog` keeps OID 11 and that a built-in type (`int4`) joins back to its namespace. The test fails on `master` (`16384 != 11`) and passes with this change. Scope: addresses the pg_catalog OID from #390 only. The static rows also reference `information_schema`'s bootstrap OID (13283 in the PG-17 export); that namespace is intentionally left dynamic here -- its composite view-types rarely need to be resolved to a namespace, and 13283 is a non-canonical export artifact. Documented for a possible follow-up. Builds on #393.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A surgical fix for the
pg_catalogOID instability in #389: stoppg_attributefrom evictingCatalog/SchemaOIDs out of the shared OID cache. Plus two regression tests.This is a narrower, more targeted alternative to #391 (which derives OIDs deterministically via hashing). Both fix the same symptom; this one keeps the existing counter + cache design and just removes the over-broad cache rewrite that caused the drift.
Root cause
The dynamic catalog tables (
pg_namespace,pg_class,pg_attribute,pg_database) share one OID counter + cache, and each builder reconciles that cache against the live catalog.pg_attributereconciled by wholesale-replacing the cache:pg_attributeenumerates only tables, soswap_cacheonly ever heldTablekeys. The replacement therefore silently deleted everyCatalog/SchemaOID — not because those objects were gone, but becausepg_attributenever looked at them. The nextpg_namespace/pg_classquery then hit a cache miss for the schema and re-allocated it a different OID via the counter. A pooled client (DBeaver, JDBC) that resolved a schema OID earlier — e.g.pg_class WHERE relnamespace = <oid>— now matched nothing, so no tables/columns were listed.Reproduces within a single shared
SessionContext(the deployment model of the shipped server, which shares one context across all connections):Key nuance:
pg_attributedoes preserve OIDs for the tables it visits (it readsoid_cache.getand carries the value intoswap_cache). The bug is purely that its wholesale*oid_cache = swap_cachediscards the keys it doesn't visit. It conflates "I didn't enumerate it" with "it doesn't exist."Fix
Reconcile surgically, the way
pg_namespacealready does — drop only staleTableentries, leaveCatalog/Schemaentries untouched, then refresh the live table OIDs:The other three builders don't need changes:
pg_namespace— already surgical (retain+extend).pg_database— already surgical (retain+extend).pg_class— wholesale-replaces, but its view is complete (catalog+schema+table), so replace == upsert + correct GC. No drift.After the fix the schema OID is stable across a
pg_attributequery and the join resolves:Tests
Two regression tests added (both fail on
master, pass with this change):schema_oid_is_stable_across_pg_attribute_query— a schema's OID must not change after a column-metadata query.pg_class_resolves_table_via_namespace_oid— the DBeaver symptom: filteringpg_classby an earlier-resolvedrelnamespacemust still resolve the table.cargo fmt --check,cargo clippy --all-features -- -D warnings, the fulldatafusion-pg-catalogsuite (48), and thedbeaver/psql/pgcli/metabaseintegration tests all pass.Scope
Addresses #389 only. #390 (
pg_catalogmust keep the fixed namespace OID 11 that the static catalog rows reference) is orthogonal and intentionally left untouched here. (A test for #390 lives on the separate test-only branch/PR #392.)Closes #389.
vs. #391
#391 replaces the counter entirely with deterministic
stable_oid(key) = hash(key). That is the more robust design (also survives process restarts and makes the cache redundant), but it's a larger change and, as written, leaves dead code that fails this repo's-D warningsgate (unusedOrderingimports + unreadoid_counterfields + a clippycollapsible_if). This PR is the minimal fix that removes the actual drift while preserving the existing design; either approach resolves the symptom.