Skip to content

fix(pg_catalog): stop pg_attribute from evicting schema/catalog OIDs (#389)#393

Merged
sunng87 merged 1 commit into
datafusion-contrib:masterfrom
GreptimeTeam:fix/pg-attribute-preserve-namespace-oids
Jul 21, 2026
Merged

fix(pg_catalog): stop pg_attribute from evicting schema/catalog OIDs (#389)#393
sunng87 merged 1 commit into
datafusion-contrib:masterfrom
GreptimeTeam:fix/pg-attribute-preserve-namespace-oids

Conversation

@sunng87

@sunng87 sunng87 commented Jul 21, 2026

Copy link
Copy Markdown
Member

What

A surgical fix for the pg_catalog OID instability in #389: stop pg_attribute from evicting Catalog/Schema OIDs out of the shared OID cache. Plus two regression tests.

This is a narrower, more targeted alternative to #391 (which derives OIDs deterministically via hashing). Both fix the same symptom; this one keeps the existing counter + cache design and just removes the over-broad cache rewrite that caused the drift.

Root cause

The dynamic catalog tables (pg_namespace, pg_class, pg_attribute, pg_database) share one OID counter + cache, and each builder reconciles that cache against the live catalog. pg_attribute reconciled by wholesale-replacing the cache:

let mut swap_cache = HashMap::new();
... swap_cache.insert(Table(..), oid) ...   // only Table keys ever inserted
*oid_cache = swap_cache;                     // cache becomes *exactly* the tables I visited

pg_attribute enumerates only tables, so swap_cache only ever held Table keys. The replacement therefore silently deleted every Catalog/Schema OID — not because those objects were gone, but because pg_attribute never looked at them. The next pg_namespace/pg_class query then hit a cache miss for the schema and re-allocated it a different OID via the counter. A pooled client (DBeaver, JDBC) that resolved a schema OID earlier — e.g. pg_class WHERE relnamespace = <oid> — now matched nothing, so no tables/columns were listed.

Reproduces within a single shared SessionContext (the deployment model of the shipped server, which shares one context across all connections):

pg_namespace.oid(public): 16384 -> 16457   (unstable, after a pg_attribute query)
pg_class rows joining on relnamespace=16384: 0

Key nuance: pg_attribute does preserve OIDs for the tables it visits (it reads oid_cache.get and carries the value into swap_cache). The bug is purely that its wholesale *oid_cache = swap_cache discards the keys it doesn't visit. It conflates "I didn't enumerate it" with "it doesn't exist."

Fix

Reconcile surgically, the way pg_namespace already does — drop only stale Table entries, leave Catalog/Schema entries untouched, then refresh the live table OIDs:

oid_cache.retain(|key, _| match key {
    OidCacheKey::Catalog(..) => true,
    OidCacheKey::Schema(..) => true,
    OidCacheKey::Table(..) => table_oid_cache.contains_key(key),
});
oid_cache.extend(table_oid_cache);

The other three builders don't need changes:

  • pg_namespace — already surgical (retain + extend).
  • pg_database — already surgical (retain + extend).
  • pg_class — wholesale-replaces, but its view is complete (catalog+schema+table), so replace == upsert + correct GC. No drift.

After the fix the schema OID is stable across a pg_attribute query and the join resolves:

pg_namespace.oid(public): 16384 -> 16384   (stable)
pg_class rows joining on relnamespace=16384: 1

Tests

Two regression tests added (both fail on master, pass with this change):

  • schema_oid_is_stable_across_pg_attribute_query — a schema's OID must not change after a column-metadata query.
  • pg_class_resolves_table_via_namespace_oid — the DBeaver symptom: filtering pg_class by an earlier-resolved relnamespace must still resolve the table.

cargo fmt --check, cargo clippy --all-features -- -D warnings, the full datafusion-pg-catalog suite (48), and the dbeaver/psql/pgcli/metabase integration tests all pass.

Scope

Addresses #389 only. #390 (pg_catalog must keep the fixed namespace OID 11 that the static catalog rows reference) is orthogonal and intentionally left untouched here. (A test for #390 lives on the separate test-only branch/PR #392.)

Closes #389.

vs. #391

#391 replaces the counter entirely with deterministic stable_oid(key) = hash(key). That is the more robust design (also survives process restarts and makes the cache redundant), but it's a larger change and, as written, leaves dead code that fails this repo's -D warnings gate (unused Ordering imports + unread oid_counter fields + a clippy collapsible_if). This PR is the minimal fix that removes the actual drift while preserving the existing design; either approach resolves the symptom.

…atafusion-contrib#389)

The dynamic catalog tables share one OID counter + cache, and each builder
reconciles that cache against the live catalog. `pg_attribute` reconciled by
wholesale-replacing the cache:

    let mut swap_cache = HashMap::new();
    ... swap_cache.insert(Table(..), oid) ...
    *oid_cache = swap_cache;

`pg_attribute` enumerates only tables, so `swap_cache` only ever held `Table`
keys. The replacement therefore silently deleted every `Catalog`/`Schema` OID
-- not because those objects were gone, but because pg_attribute never looked
at them. The next `pg_namespace`/`pg_class` query then found a cache miss for
the schema and re-allocated it a *different* OID via the counter. A pooled
client (DBeaver, JDBC) that resolved a schema OID earlier -- e.g.
`pg_class WHERE relnamespace = <oid>` -- now matched nothing, so no tables or
columns were listed.

Reproduces (before this change) as a schema OID drifting across a single
column-metadata query:

    pg_namespace.oid(public): 16384 -> 16457   (unstable)
    pg_class rows joining on relnamespace=16384: 0

Fix: reconcile surgically, like `pg_namespace` already does -- drop only stale
`Table` entries and leave `Catalog`/`Schema` entries untouched, then refresh
the live table OIDs:

    oid_cache.retain(|key, _| match key {
        OidCacheKey::Catalog(..) => true,
        OidCacheKey::Schema(..) => true,
        OidCacheKey::Table(..) => table_oid_cache.contains_key(key),
    });
    oid_cache.extend(table_oid_cache);

pg_attribute already preserved OIDs for the tables it visits (via the
`oid_cache.get` memo hit); now it also stops clobbering the OIDs it doesn't.
After the fix the schema OID is stable and the join resolves.

Adds two regression tests covering the invariant within a single shared
SessionContext (the deployment model of the shipped server).

Note: this addresses datafusion-contrib#389 only. datafusion-contrib#390 (pg_catalog must keep the fixed
namespace OID 11 that the static catalog rows reference) is orthogonal and not
changed here.
@sunng87
sunng87 merged commit e093d1f into datafusion-contrib:master Jul 21, 2026
7 checks passed
@sunng87
sunng87 deleted the fix/pg-attribute-preserve-namespace-oids branch July 21, 2026 14:21
sunng87 added a commit to GreptimeTeam/datafusion-postgres that referenced this pull request Jul 21, 2026
…tafusion-contrib#390)

The static catalog rows shipped by this crate (the
`pg_catalog_arrow_exports/*.feather`, produced by
`export_pg_catalog_arrow.sh` from a fresh PostgreSQL) reference the canonical
`pg_catalog` namespace OID 11 -- e.g. `pg_type.typnamespace = 11`,
`pg_proc.pronamespace = 11`. But `pg_namespace`/`pg_class` assigned the
`pg_catalog` schema a dynamically generated OID from the counter (16384+), so
no namespace with OID 11 existed. Any client joining a system object to its
namespace (`pg_type.typnamespace -> pg_namespace.oid`) got nothing --
concretely, DBeaver could not resolve any column's data type (every column
showed as unknown).

Resolve OIDs through a single helper so a fixed canonical OID is always
honored:

    fn resolve_oid(key, oid_cache, oid_counter) -> Oid {
        fixed_oid(key)            // 1. fixed canonical OID, if any
            .or_else(|| oid_cache.get(key).copied())   // 2. previously assigned
            .unwrap_or_else(|| oid_counter.fetch_add(1, ...))  // 3. new
    }

`fixed_oid` is keyed by `&OidCacheKey` (not just the schema name) and is now
consulted at *every* OID-assignment site -- catalogs and tables, not only
schemas -- so adding a fixed OID for any object kind (other namespaces,
specific catalogs/tables, columns later) is a new match arm in `fixed_oid`
and is picked up everywhere automatically. Today only `pg_catalog` (-> 11,
below the user range, no collision) is pinned.

All eight assignment sites in `pg_namespace`/`pg_class`/`pg_database`/
`pg_attribute` now call `resolve_oid`, removing the duplicated
`fixed -> cache -> counter` ladder.

Adds a regression test asserting `pg_catalog` keeps OID 11 and that a built-in
type (`int4`) joins back to its namespace. The test fails on `master`
(`16384 != 11`) and passes with this change.

Scope: addresses the pg_catalog OID from datafusion-contrib#390 only. The static rows also
reference `information_schema`'s bootstrap OID (13283 in the PG-17 export);
that namespace is intentionally left dynamic here -- its composite view-types
rarely need to be resolved to a namespace, and 13283 is a non-canonical
export artifact. Documented for a possible follow-up. Builds on datafusion-contrib#393.
sunng87 added a commit that referenced this pull request Jul 21, 2026
…) (#394)

The static catalog rows shipped by this crate (the
`pg_catalog_arrow_exports/*.feather`, produced by
`export_pg_catalog_arrow.sh` from a fresh PostgreSQL) reference the canonical
`pg_catalog` namespace OID 11 -- e.g. `pg_type.typnamespace = 11`,
`pg_proc.pronamespace = 11`. But `pg_namespace`/`pg_class` assigned the
`pg_catalog` schema a dynamically generated OID from the counter (16384+), so
no namespace with OID 11 existed. Any client joining a system object to its
namespace (`pg_type.typnamespace -> pg_namespace.oid`) got nothing --
concretely, DBeaver could not resolve any column's data type (every column
showed as unknown).

Resolve OIDs through a single helper so a fixed canonical OID is always
honored:

    fn resolve_oid(key, oid_cache, oid_counter) -> Oid {
        fixed_oid(key)            // 1. fixed canonical OID, if any
            .or_else(|| oid_cache.get(key).copied())   // 2. previously assigned
            .unwrap_or_else(|| oid_counter.fetch_add(1, ...))  // 3. new
    }

`fixed_oid` is keyed by `&OidCacheKey` (not just the schema name) and is now
consulted at *every* OID-assignment site -- catalogs and tables, not only
schemas -- so adding a fixed OID for any object kind (other namespaces,
specific catalogs/tables, columns later) is a new match arm in `fixed_oid`
and is picked up everywhere automatically. Today only `pg_catalog` (-> 11,
below the user range, no collision) is pinned.

All eight assignment sites in `pg_namespace`/`pg_class`/`pg_database`/
`pg_attribute` now call `resolve_oid`, removing the duplicated
`fixed -> cache -> counter` ladder.

Adds a regression test asserting `pg_catalog` keeps OID 11 and that a built-in
type (`int4`) joins back to its namespace. The test fails on `master`
(`16384 != 11`) and passes with this change.

Scope: addresses the pg_catalog OID from #390 only. The static rows also
reference `information_schema`'s bootstrap OID (13283 in the PG-17 export);
that namespace is intentionally left dynamic here -- its composite view-types
rarely need to be resolved to a namespace, and 13283 is a non-canonical
export artifact. Documented for a possible follow-up. Builds on #393.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pg_catalog OIDs are assigned per-connection and unstable — breaks pooled clients (DBeaver, etc.)

1 participant