codegen: never let a proto comment's code fence become a doctest#309
Conversation
A code fence in a .proto comment is emitted into the generated #[doc]
attributes as-is, and rustdoc then compiles it as a doctest in the
*consuming* crate — where the snippet has no imports and names proto types
rather than Rust ones. Proto comments are language-agnostic by nature, so a
fenced JSON payload or shell example is completely ordinary, and today
every one of them is handed to the Rust compiler.
Classifying which fences rustdoc considers Rust turns out to be a trap. An
explicit `rust` keeps a block Rust even beside an unknown word
(`rust,noplayground`), error-code tokens (`compile_fail,E0277`) and
`{class=...}` attributes keep it Rust too, and the verdict is
order-dependent. Of the attributes, only a bare `ignore` reliably stops
compilation: `no_run` still type-checks, `should_panic` runs,
`compile_fail` merely inverts the verdict, and `ignore-<target>` is read as
a target list that *replaces* a plain `ignore`, so the block still compiles
everywhere else.
So don't classify. Drop any `ignore-<target>` token and ensure a bare
`ignore` on every fence, tilde fences included — rustdoc's markdown parser
compiles `~~~` blocks just like ``` ones. The author's language annotation
survives, so a `rust` fence is still syntax-highlighted as `rust,ignore`;
an unannotated fence becomes `text`, since rustdoc highlights nothing but
Rust and guessing a language would buy no rendering benefit.
Fence detection now follows CommonMark, which the previous open/close
toggle did not: a closer needs at least the opener's run length, the same
fence character, and only spaces or tabs after it, while a marker indented
4+ spaces (or preceded by other whitespace) is not a fence at all. An
unterminated fence is closed at the end of the comment, and a field's
"Field N" tag is rendered separately from the comment body so an open fence
cannot swallow it.
Fixes #307.
Signed-off-by: Iain McGinniss <309153+iainmcgin@users.noreply.github.com>
|
All contributors have signed the CLA ✍️ ✅ |
An indented code block is four columns of leading whitespace, and CommonMark expands a tab to the next four-column stop. Detection counted literal leading spaces instead, so a line indented with one to three spaces followed by a tab reached column four for rustdoc but not for us: it was emitted as prose and compiled as a doctest in the consuming crate, which is the failure this whole path exists to prevent. Measure the indent by column with tab expansion, and strip one level by the same measure. Also fix the doc comment on strip_indent, whose bare backtick runs paired into an inline code span, and trim the changelog fragment to the user-visible effect. No-Verification-Needed: covered by codegen unit tests; no runtime surface
|
[claude code] Review pass on this branch turned up one real gap in the fix, now addressed in Indentation was measured in literal leading spaces, not columns. CommonMark expands a tab to the next four-column stop, so a line indented with one to three spaces followed by a tab ( Also in that commit: the doc comment on Verified end-to-end against a real consumer crate, which is the surface that actually matters here — a scratch crate whose
The four failures on main come from the generated Known limitation, deliberately not fixed here: a bare fence loses author-intended syntax highlighting because an unannotated fence becomes Marking ready for review. |
Fixes #307.
A code fence in a
.protocomment is emitted into the generated#[doc]attributes as-is, so rustdoc compiles it as a doctest in the consuming crate — where the snippet has no imports and names proto types rather than Rust ones. Proto comments are language-agnostic by nature (the same file feeds Go, Java, TS and Rust codegen), so a fenced JSON payload is completely ordinary, and today every one of them is handed to the Rust compiler. The repro in #307 fails a downstreamcargo test --docon generated code the user cannot edit.Don't classify — make every fence inert
I first tried deciding "is this fence Rust?" and only then acting. That is a trap, and each of these was verified against rustdoc directly:
rustkeeps a block Rust even beside an unknown word, so```rust,noplayground(an mdBook idiom, very copy-pasteable into a proto) compiles. Error-code tokens (```compile_fail,E0277) and{class=…}attributes keep it Rust too, and the verdict is even order-dependent (```rust,jsoncompiles;```json,no_rundoes not).ignorereliably stops compilation:no_runstill type-checks,should_panicruns, andcompile_failmerely inverts the verdict.ignore-<target>is read as a target list that replaces a plainignore— so```ignore-wasm32,ignorestill compiles on every other target.So this drops any
ignore-<target>token and ensures a bareignoreon every fence. The author's language annotation survives, so a```rustfence is still syntax-highlighted as```rust,ignore; an unannotated fence becomes```text, since rustdoc highlights nothing but Rust — a```jsonblock renders as<pre class="language-json">with no token markup either way, so guessing a language buys no rendering benefit.Tilde fences are covered too. rustdoc's markdown parser compiles
~~~blocks exactly like ``` ones, and the old toggle never looked at them.CommonMark-correct fence detection
The previous open/close toggle flipped on any line starting with
```. It now requires, per CommonMark: a closer with at least the opener's run length, the same fence character, and only spaces or tabs after it; and it treats a marker indented 4+ spaces (or preceded by any other whitespace, e.g. an NBSP) as not a fence — misreading one of those would "close" a fence rustdoc never opened and leave the run it really does treat as an opener unguarded. An unterminated fence is closed at the end of the comment instead of swallowing the doc text after it, and a field's "Field N" tag is now rendered separately from the comment body so an open fence cannot pull the tag into the code block.Verification
buffa-codegentests pass, including new cases for each hazard above.cargo doc -D warningsis clean.task gen-bootstrap-typesis a no-op), sodescriptor.proto's own comments are unaffected.connect-rust had forked this sanitizer for its service/method comments and hit the same hazards; the identical hardening is in connectrpc/connect-rust#225. Once this is released, that fork can be deleted in favor of buffa's.