Skip to content

codegen: never let a proto comment's code fence become a doctest#309

Merged
iainmcgin merged 2 commits into
mainfrom
fix/comment-fence-doctest-hazard
Jul 16, 2026
Merged

codegen: never let a proto comment's code fence become a doctest#309
iainmcgin merged 2 commits into
mainfrom
fix/comment-fence-doctest-hazard

Conversation

@iainmcgin

Copy link
Copy Markdown
Collaborator

Fixes #307.

A code fence in a .proto comment is emitted into the generated #[doc] attributes as-is, so rustdoc compiles it as a doctest in the consuming crate — where the snippet has no imports and names proto types rather than Rust ones. Proto comments are language-agnostic by nature (the same file feeds Go, Java, TS and Rust codegen), so a fenced JSON payload is completely ordinary, and today every one of them is handed to the Rust compiler. The repro in #307 fails a downstream cargo test --doc on generated code the user cannot edit.

Don't classify — make every fence inert

I first tried deciding "is this fence Rust?" and only then acting. That is a trap, and each of these was verified against rustdoc directly:

  • An explicit rust keeps a block Rust even beside an unknown word, so ```rust,noplayground (an mdBook idiom, very copy-pasteable into a proto) compiles. Error-code tokens (```compile_fail,E0277) and {class=…} attributes keep it Rust too, and the verdict is even order-dependent (```rust,json compiles; ```json,no_run does not).
  • Of rustdoc's attributes, only a bare ignore reliably stops compilation: no_run still type-checks, should_panic runs, and compile_fail merely inverts the verdict.
  • ignore-<target> is read as a target list that replaces a plain ignore — so ```ignore-wasm32,ignore still compiles on every other target.

So this drops any ignore-<target> token and ensures a bare ignore on every fence. The author's language annotation survives, so a ```rust fence is still syntax-highlighted as ```rust,ignore; an unannotated fence becomes ```text, since rustdoc highlights nothing but Rust — a ```json block renders as <pre class="language-json"> with no token markup either way, so guessing a language buys no rendering benefit.

Tilde fences are covered too. rustdoc's markdown parser compiles ~~~ blocks exactly like ``` ones, and the old toggle never looked at them.

CommonMark-correct fence detection

The previous open/close toggle flipped on any line starting with ```. It now requires, per CommonMark: a closer with at least the opener's run length, the same fence character, and only spaces or tabs after it; and it treats a marker indented 4+ spaces (or preceded by any other whitespace, e.g. an NBSP) as not a fence — misreading one of those would "close" a fence rustdoc never opened and leave the run it really does treat as an opener unguarded. An unterminated fence is closed at the end of the comment instead of swallowing the doc text after it, and a field's "Field N" tag is now rendered separately from the comment body so an open fence cannot pull the tag into the code block.

Verification

  • 566 buffa-codegen tests pass, including new cases for each hazard above.
  • The Code fences in proto comments become doctests in the consuming crate #307 repro goes red → green: its doctests are now ignored rather than compiled, and cargo doc -D warnings is clean.
  • No churn in the checked-in bootstrap descriptor types (task gen-bootstrap-types is a no-op), so descriptor.proto's own comments are unaffected.
  • MSRV-safe (no APIs newer than 1.75).

connect-rust had forked this sanitizer for its service/method comments and hit the same hazards; the identical hardening is in connectrpc/connect-rust#225. Once this is released, that fork can be deleted in favor of buffa's.

A code fence in a .proto comment is emitted into the generated #[doc]
attributes as-is, and rustdoc then compiles it as a doctest in the
*consuming* crate — where the snippet has no imports and names proto types
rather than Rust ones. Proto comments are language-agnostic by nature, so a
fenced JSON payload or shell example is completely ordinary, and today
every one of them is handed to the Rust compiler.

Classifying which fences rustdoc considers Rust turns out to be a trap. An
explicit `rust` keeps a block Rust even beside an unknown word
(`rust,noplayground`), error-code tokens (`compile_fail,E0277`) and
`{class=...}` attributes keep it Rust too, and the verdict is
order-dependent. Of the attributes, only a bare `ignore` reliably stops
compilation: `no_run` still type-checks, `should_panic` runs,
`compile_fail` merely inverts the verdict, and `ignore-<target>` is read as
a target list that *replaces* a plain `ignore`, so the block still compiles
everywhere else.

So don't classify. Drop any `ignore-<target>` token and ensure a bare
`ignore` on every fence, tilde fences included — rustdoc's markdown parser
compiles `~~~` blocks just like ``` ones. The author's language annotation
survives, so a `rust` fence is still syntax-highlighted as `rust,ignore`;
an unannotated fence becomes `text`, since rustdoc highlights nothing but
Rust and guessing a language would buy no rendering benefit.

Fence detection now follows CommonMark, which the previous open/close
toggle did not: a closer needs at least the opener's run length, the same
fence character, and only spaces or tabs after it, while a marker indented
4+ spaces (or preceded by other whitespace) is not a fence at all. An
unterminated fence is closed at the end of the comment, and a field's
"Field N" tag is rendered separately from the comment body so an open fence
cannot swallow it.

Fixes #307.

Signed-off-by: Iain McGinniss <309153+iainmcgin@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

An indented code block is four columns of leading whitespace, and CommonMark
expands a tab to the next four-column stop. Detection counted literal leading
spaces instead, so a line indented with one to three spaces followed by a tab
reached column four for rustdoc but not for us: it was emitted as prose and
compiled as a doctest in the consuming crate, which is the failure this whole
path exists to prevent.

Measure the indent by column with tab expansion, and strip one level by the
same measure.

Also fix the doc comment on strip_indent, whose bare backtick runs paired into
an inline code span, and trim the changelog fragment to the user-visible
effect.

No-Verification-Needed: covered by codegen unit tests; no runtime surface
@iainmcgin
iainmcgin marked this pull request as ready for review July 16, 2026 18:28
@iainmcgin

Copy link
Copy Markdown
Collaborator Author

[claude code] Review pass on this branch turned up one real gap in the fix, now addressed in 8a91677.

Indentation was measured in literal leading spaces, not columns. CommonMark expands a tab to the next four-column stop, so a line indented with one to three spaces followed by a tab (" \t") reaches column four and is an indented code block for rustdoc — but is_indented returned false for it, so the line went out as prose and was compiled as a doctest in the consuming crate. That is the same failure this PR exists to eliminate, surviving in the one case the original check couldn't see. Indent is now measured by column with tab expansion, and strip_indent removes one level by the same measure.

Also in that commit: the doc comment on strip_indent contained bare triple-backtick runs that CommonMark paired into an inline code span (the fence PR breaking its own rustdoc), and the changelog fragment now leads with the user-visible effect rather than the parser internals.

Verified end-to-end against a real consumer crate, which is the surface that actually matters here — a scratch crate whose .proto comment carries a bare fence, a tilde fence, a mixed space+tab indented block, and a four-space indented block:

buffa doctests harvested cargo test --doc
main 4 fails to compile
this branch 0 ok

The four failures on main come from the generated x.rs and x.__view.rs — precisely the downstream cargo test breakage reported in #307. Unit-level: 67 comments tests pass, and the new column-measurement test goes red against the old literal-space check. cargo fmt --check, clippy --workspace --all-targets -D warnings, and RUSTDOCFLAGS='-D warnings' cargo doc are all clean — the last one matters because this change adds ignore to non-Rust fences, and I confirmed that emits no rustdoc warning under the deny-warnings job added in a09993cc.

Known limitation, deliberately not fixed here: a bare fence loses author-intended syntax highlighting because an unannotated fence becomes text. A bare fence in a proto comment is genuinely ambiguous, so inert-by-default is the right trade.

Marking ready for review.

@iainmcgin
iainmcgin enabled auto-merge July 16, 2026 18:48
@iainmcgin
iainmcgin requested a review from azdagron July 16, 2026 18:48
@iainmcgin
iainmcgin added this pull request to the merge queue Jul 16, 2026
Merged via the queue into main with commit ea55e7e Jul 16, 2026
9 checks passed
@iainmcgin
iainmcgin deleted the fix/comment-fence-doctest-hazard branch July 16, 2026 19:02
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Code fences in proto comments become doctests in the consuming crate

2 participants