refactor(unparser): centralize aggregate-scope rendering in the SQL unparser#23789
refactor(unparser): centralize aggregate-scope rendering in the SQL unparser#23789naman-modi wants to merge 6 commits into
Conversation
|
Hey @kosiew, can you please review this PR, when you get a chance? One small design note: I went with a small IMO, it reads a bit cleaner at the call sites and keeps the flag from being recomputed per clause. If you'd rather keep it consistent with the existing TIA! |
There was a problem hiding this comment.
@naman-modi
Nice fix overall. I like the direction of consolidating the aggregate scope handling into a small helper. I just have a couple of minor suggestions that could make the abstraction a little cleaner and the test coverage a bit clearer.
| /// ORDER BY resolves against the same scope as every other clause in the | ||
| /// block. Associated fn taking `Option<&Self>` because a sort can appear | ||
| /// with no aggregate below it, in which case only unprojection applies. | ||
| fn prepare_sort_expr( |
There was a problem hiding this comment.
I like UnparserAggScope as a small abstraction because it naturally ties an aggregate to its derived-input scope decision.
One small thing that stood out is prepare_sort_expr(scope: Option<&Self>, ...). The None case doesn't really have an aggregate scope, so it feels like the abstraction is carrying a responsibility outside its model.
Would it make sense to keep aggregate normalization as an instance method and move the optional sort orchestration into an Unparser helper, or simply split the aggregate and non-aggregate paths? I think that would keep the responsibilities a bit more focused. Just a small design suggestion.
There was a problem hiding this comment.
Makes sense, I have updated the prepare_sort_expr to now be an instance method of UnparserAggScope, and created a Unparser::unproject_sort_expr_in_scope to handle both the arms, hence unifying the call sites.
| } | ||
|
|
||
| #[tokio::test] | ||
| async fn optimized_duckdb_unparse_top_level_sort_over_agg_stays_in_scope() -> Result<()> { |
There was a problem hiding this comment.
This test is useful because it exercises the top-level Sort path, but since it orders by the selected alias customers, it doesn't actually demonstrate that the new prepare_sort_expr logic strips an aggregate input qualifier in that branch.
Would it be worth adding a direct top-level Sort regression whose sort expression contains an aggregate over a qualified input column? Alternatively, the test name or comment could be adjusted to make it clear that it's only covering the top-level Sort path. The non-selected aggregate regression already covers the reported bug.
There was a problem hiding this comment.
You're right, this holds. I have updated the method name & comment to be more specific - that it only covers the top-level Sort routing.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #23789 +/- ##
=======================================
Coverage ? 80.71%
=======================================
Files ? 1089
Lines ? 368903
Branches ? 368903
=======================================
Hits ? 297764
Misses ? 53375
Partials ? 17764 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thanks for reviewing @kosiew! I've addressed both review comments. It would be great, if you could trigger the CI. |
Which issue does this PR close?
Rationale for this change
... FROM (SELECT ...)).csonly exist inside that subquery. Any clause outside it (SELECT, GROUP BY, HAVING, QUALIFY, ORDER BY) must use the subquery's output columns, not the aliases, so the unparser drops the alias.ORDER BY round(sum("cs"."total_revenue"), 2), while the SELECT list in the same query correctly printedsum("total_revenue").csis out of scope there.What changes are included in this PR?
I moved the rule into one helper,
UnparserAggScope, that checks once per aggregate whether the input is an inner subquery and then prepares expressions for each clause. SELECT, GROUP BY, HAVING, QUALIFY, and both ORDER BY paths now go through it, which fixes the ORDER BY case. The window-over-aggregate path is left as-is with a comment: it is only reachable from hand-built plans, since a window in SQL always sits inside a SELECT that already drops the alias. Only ORDER BY output changes; every already-correct case stays the same.Are these changes tested?
Yes. New tests in
datafusion/core/tests/sql/unparser.rscover the inner-subquery shape for a window sorting by an aggregate, ORDER BY on an unselected aggregate (the fixed case), and a top-level ORDER BY (the second sort path). Existing unparser tests and the TPC-H/Clickbench roundtrips still pass.Are there any user-facing changes?
ORDER BY over an aggregate is now printed without an out-of-scope table qualifier, so the generated SQL is valid for stricter databases. No API changes.