Skip to content

Conversation

@martykulma
Copy link
Contributor

@martykulma martykulma commented Jan 30, 2026

Adds support for Avro schema references with Confluent Schema Registry.

  • Update Avro parser to accept reference schemas and use them during parsing
  • Update SQL syntax to include key and value references in the seed
  • Update purification to populate key and value references
  • Plumb reference schemas through decoder/interchange
  • Update testdrive to add support for reference schemas

Motivation

Implements SS-47: Add support to Kafka source for Avro schema references with CSR

Tips for reviewer

I broke up the diff into commits to allow looking at smaller subsets of changes that make sense. Testdrive needed a lot of love as it needs to handle publishing and then cleanup.

The framework was mainly there, a few extra bits needed to be added.

  • src/avro/src/schema.rs - collect_named_types identifies the named types in the references and creates a single global index of them, after that the parser has all it needs.
  • src/ccsr/src/client.rs - get_subject_and_references_by_version was sorting the references alphabetically, but parsing needs to process dependents before parents. I didn't find any code that relied on an alphabetical sort, so I changed this to a topological sort.

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

@martykulma martykulma marked this pull request as ready for review January 30, 2026 12:24
@martykulma martykulma requested review from a team as code owners January 30, 2026 12:24
@def-
Copy link
Contributor

def- commented Jan 30, 2026

Test failures in your Nightly run are unrelated

@martykulma
Copy link
Contributor Author

Test failures in your Nightly run are unrelated

Thanks @def- - you and I were apparently looking at the same time, I didn't see your comment. 😂

Cancelled racing-car-testdrive-4-replicas since it takes at least an hour, and the failure is unrelated to this PR.

@martykulma martykulma force-pushed the maz-avro-reference-schemas branch from 95d84fe to 6c544ef Compare February 2, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants