reflect: adopt message values from another descriptor pool on set#308
Draft
iainmcgin wants to merge 1 commit into
Draft
reflect: adopt message values from another descriptor pool on set#308iainmcgin wants to merge 1 commit into
iainmcgin wants to merge 1 commit into
Conversation
A generated type reflects against its *defining* crate's DescriptorPool: a
Duration view always reports buffa-types' pool, never the consumer pool that
references it. So a nested cross-crate message value can never be
pointer-identical to its parent's pool, and validating message values by
Arc::ptr_eq rejected every one of them — with a self-contradictory error:
field "timeout" (#2) on verify.Job
expects message google.protobuf.Duration,
got message google.protobuf.Duration
That made the vtable rebuild walk (for_each_set + set(fd, vr.to_owned()))
panic for any message with a well-known-type field. It is not WKT-specific:
any extern_path-mapped proto hits it, and so do two packages generated in one
codegen run, which get distinct pool Arcs. Views, owned types, and the bridge
path are all affected; only lazy views escape, having no reflect impls.
try_set now adopts rather than rejects. A message of the same full name from a
foreign pool is re-homed into the target pool by a wire round-trip, which
re-homes the whole subtree in one decode, recursively covering singular, list
and map values. Descriptors stay strict — you still pass field descriptors
resolved from the message's own pool — but values are adopted, because the
codegen makes pool-identical values impossible for a cross-crate type. Full
name is the adoption key, the same criterion Any's type URL uses, so a
genuinely wrong type is still WrongValueKind whatever pool it came from.
The strict check doubles as the "already homed and well-shaped" predicate, so a
value that needs nothing is returned untouched and the common path — every
value the decoder and the JSON parser produce — allocates nothing, not even for
repeated and map fields. Re-homing goes through try_encode_to_vec: a foreign
message over the 2 GiB limit is rejected, never a panic inside set.
CI ran only three of the seven conformance modes, which is why this shipped
green; via-lazy, view-json, via-reflect and via-vtable are now wired in too.
The via-vtable run goes from 20 panicking failures to 1246 successes, 0
failures, and the suite is 14/14.
Adds MapValue::into_entries, the by-value counterpart to from_entries, so a map
can be rebuilt without cloning its values.
|
All contributors have signed the CLA ✍️ ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #297.
The bug
A generated type reflects against its defining crate's
DescriptorPool. ADurationview always reports buffa-types' pool — never the consumer pool that references it. So a nested cross-crate message value can never be pointer-identical to its parent's pool, and #272'sArc::ptr_eqcheck on message values rejected every one of them, with an error that says the quiet part out loud:That makes the vtable rebuild walk —
for_each_set+set(fd, vr.to_owned())— panic for any message with a well-known-type field.It is broader than the issue title suggests. It is not WKT-specific (any
extern_path-mapped proto hits it, and so do two packages generated in a single codegen run, which get distinct poolArcs), and it is not view-specific (owned types and the bridge path are affected identically; only lazy views escape, having no reflect impls at all).The fix
try_setadopts rather than rejects. A message of the same full name from a foreign pool is re-homed into the target pool by a wire round-trip — which re-homes the whole subtree in one decode — recursively covering singular, list, and map values.Descriptors stay strict: you still pass field descriptors resolved from the message's own pool. Values get adopted, because the codegen makes pool-identical values impossible for a cross-crate type. Full name is the adoption key — the same criterion protobuf itself uses across compilations, and what
Any's type URL carries — so a genuinely wrong type is stillWrongValueKindwhatever pool it came from.Two details worth a reviewer's eye:
try_encode_to_vec, notencode_to_vec: after encode: enforce the protobuf 2 GiB message-size limit #271 the latter panics above 2 GiB, and a value handed tosetmust never take the process down. An over-limit foreign message is rejected like any other value the pool cannot hold.Why CI didn't catch it
The conformance job ran only three of the seven modes.
via-lazy,view-json,via-reflectandvia-vtableexisted solely in the local Docker script, and via-vtable is the one that walks this exact rebuild path. All four are now wired into CI. The via-vtable run goes from 20 panicking failures to 1246 successes, 0 failures; the suite is 14/14.Behavior change
A cross-pool value that previously returned
Err(WrongValueKind)now succeeds. Callers that leaned on that rejection as a provenance check must compareReflectMessage::poolthemselves — called out in the changelog fragment. #272's own fragment is still unreleased and claimed "nested message descriptor identity included" among the rejections, so it would have shipped contradicting this one in the same release; it is amended here to say what it actually delivers (nested messages of the wrong type).Also adds
MapValue::into_entries, the by-value counterpart tofrom_entries, so a map can be rebuilt without cloning its values.Verification
Beyond the unit tests (which reproduce the cross-pool condition with two pools decoded from identical FDS bytes — no buffa-types needed), I drove the real cross-crate scenario from a scratch consumer crate: a
Jobmessage withDuration/Timestamp/repeated Durationfields, generated with vtable reflection, walked throughfor_each_set+set. Onmainit panics with the error above. With this change the walk completes, the rebuiltDynamicMessageencodes byte-identically to the generated message (49 bytes), the adopted sub-messages report the consumer's pool, the reflective JSON is correct ("timeout":"90.000500s"), and aTimestampset into aDurationfield is still rejected — now with an error that actually names two different types.Full workspace tests, clippy
-D warnings, and conformance 14/14.