Skip to content

reflect: adopt message values from another descriptor pool on set#308

Draft
iainmcgin wants to merge 1 commit into
mainfrom
iain/fix-297-reflect-pool
Draft

reflect: adopt message values from another descriptor pool on set#308
iainmcgin wants to merge 1 commit into
mainfrom
iain/fix-297-reflect-pool

Conversation

@iainmcgin

Copy link
Copy Markdown
Collaborator

Fixes #297.

The bug

A generated type reflects against its defining crate's DescriptorPool. A Duration view always reports buffa-types' pool — never the consumer pool that references it. So a nested cross-crate message value can never be pointer-identical to its parent's pool, and #272's Arc::ptr_eq check on message values rejected every one of them, with an error that says the quiet part out loud:

field "timeout" (#2) on verify.Job
  expects message google.protobuf.Duration,
  got     message google.protobuf.Duration

That makes the vtable rebuild walk — for_each_set + set(fd, vr.to_owned()) — panic for any message with a well-known-type field.

It is broader than the issue title suggests. It is not WKT-specific (any extern_path-mapped proto hits it, and so do two packages generated in a single codegen run, which get distinct pool Arcs), and it is not view-specific (owned types and the bridge path are affected identically; only lazy views escape, having no reflect impls at all).

The fix

try_set adopts rather than rejects. A message of the same full name from a foreign pool is re-homed into the target pool by a wire round-trip — which re-homes the whole subtree in one decode — recursively covering singular, list, and map values.

Descriptors stay strict: you still pass field descriptors resolved from the message's own pool. Values get adopted, because the codegen makes pool-identical values impossible for a cross-crate type. Full name is the adoption key — the same criterion protobuf itself uses across compilations, and what Any's type URL carries — so a genuinely wrong type is still WrongValueKind whatever pool it came from.

Two details worth a reviewer's eye:

  • The pre-existing strict check doubles as the "already homed and well-shaped" predicate, so a value needing nothing is returned untouched. The common path — every value the decoder and the JSON parser produce — allocates nothing, not even for repeated and map fields.
  • Re-homing goes through try_encode_to_vec, not encode_to_vec: after encode: enforce the protobuf 2 GiB message-size limit #271 the latter panics above 2 GiB, and a value handed to set must never take the process down. An over-limit foreign message is rejected like any other value the pool cannot hold.

Why CI didn't catch it

The conformance job ran only three of the seven modes. via-lazy, view-json, via-reflect and via-vtable existed solely in the local Docker script, and via-vtable is the one that walks this exact rebuild path. All four are now wired into CI. The via-vtable run goes from 20 panicking failures to 1246 successes, 0 failures; the suite is 14/14.

Behavior change

A cross-pool value that previously returned Err(WrongValueKind) now succeeds. Callers that leaned on that rejection as a provenance check must compare ReflectMessage::pool themselves — called out in the changelog fragment. #272's own fragment is still unreleased and claimed "nested message descriptor identity included" among the rejections, so it would have shipped contradicting this one in the same release; it is amended here to say what it actually delivers (nested messages of the wrong type).

Also adds MapValue::into_entries, the by-value counterpart to from_entries, so a map can be rebuilt without cloning its values.

Verification

Beyond the unit tests (which reproduce the cross-pool condition with two pools decoded from identical FDS bytes — no buffa-types needed), I drove the real cross-crate scenario from a scratch consumer crate: a Job message with Duration / Timestamp / repeated Duration fields, generated with vtable reflection, walked through for_each_set + set. On main it panics with the error above. With this change the walk completes, the rebuilt DynamicMessage encodes byte-identically to the generated message (49 bytes), the adopted sub-messages report the consumer's pool, the reflective JSON is correct ("timeout":"90.000500s"), and a Timestamp set into a Duration field is still rejected — now with an error that actually names two different types.

Full workspace tests, clippy -D warnings, and conformance 14/14.

A generated type reflects against its *defining* crate's DescriptorPool: a
Duration view always reports buffa-types' pool, never the consumer pool that
references it. So a nested cross-crate message value can never be
pointer-identical to its parent's pool, and validating message values by
Arc::ptr_eq rejected every one of them — with a self-contradictory error:

    field "timeout" (#2) on verify.Job
      expects message google.protobuf.Duration,
      got     message google.protobuf.Duration

That made the vtable rebuild walk (for_each_set + set(fd, vr.to_owned()))
panic for any message with a well-known-type field. It is not WKT-specific:
any extern_path-mapped proto hits it, and so do two packages generated in one
codegen run, which get distinct pool Arcs. Views, owned types, and the bridge
path are all affected; only lazy views escape, having no reflect impls.

try_set now adopts rather than rejects. A message of the same full name from a
foreign pool is re-homed into the target pool by a wire round-trip, which
re-homes the whole subtree in one decode, recursively covering singular, list
and map values. Descriptors stay strict — you still pass field descriptors
resolved from the message's own pool — but values are adopted, because the
codegen makes pool-identical values impossible for a cross-crate type. Full
name is the adoption key, the same criterion Any's type URL uses, so a
genuinely wrong type is still WrongValueKind whatever pool it came from.

The strict check doubles as the "already homed and well-shaped" predicate, so a
value that needs nothing is returned untouched and the common path — every
value the decoder and the JSON parser produce — allocates nothing, not even for
repeated and map fields. Re-homing goes through try_encode_to_vec: a foreign
message over the 2 GiB limit is rejected, never a panic inside set.

CI ran only three of the seven conformance modes, which is why this shipped
green; via-lazy, view-json, via-reflect and via-vtable are now wired in too.
The via-vtable run goes from 20 panicking failures to 1246 successes, 0
failures, and the suite is 14/14.

Adds MapValue::into_entries, the by-value counterpart to from_entries, so a map
can be rebuilt without cloning its values.
@github-actions

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vtable rebuild of WKT sub-messages panics since #272 (pool pointer-identity vs cross-crate to_dynamic); via-vtable conformance not run in CI

1 participant