feat: hint at native pyarrow UDF path when the feature is disabled#4892
Open
andygrove wants to merge 1 commit into
Open
feat: hint at native pyarrow UDF path when the feature is disabled#4892andygrove wants to merge 1 commit into
andygrove wants to merge 1 commit into
Conversation
When spark.comet.exec.pyarrowUdf.enabled is off (the default), mapInArrow and mapInPandas operators silently fall back to Spark. Annotate the operator with a non-fallback [COMET-INFO] hint naming the config to opt into the native CometMapInBatchExec path, reusing the existing NativeOptIn wording. The hint fires only on operators that would actually run natively if the flag were flipped: the eligibility check is unchanged, the conf gate simply moves from the extractor into the rewrite arm.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
There is no dedicated tracking issue for this change.
Rationale for this change
spark.comet.exec.pyarrowUdf.enabledisfalseby default. When it is off,mapInArrow/mapInPandasoperators silently fall back to vanilla Spark execution, and nothing in the plan tells the user that a faster native path (CometMapInBatchExec, which passes Arrow columnar data directly to Python UDFs) is available behind a config flag.Comet already surfaces this kind of guidance for expressions via
NativeOptIn+withInfo, which writes a non-fallback[COMET-INFO: ...]message into extended-explain output pointing the user at the config that would opt into a native path. This PR applies the same idiom at the operator level for the PyArrow UDF feature.What changes are included in this PR?
EliminateRedundantTransitions, theEligibleMapInBatchextractor no longer gates onspark.comet.exec.pyarrowUdf.enabled. It now matches whenever an operator would run natively (version-shimmedmapInArrow/mapInPandasmatcher, strippable Comet columnar child,useLargeVarTypesoff).CometMapInBatchExecas before; when it is disabled the vanilla Spark operator is left in place but annotated with a[COMET-INFO]opt-in hint (reusingNativeOptIn.messageso the wording stays consistent with the expression-level hints).Because the hint reuses the full eligibility check with only the conf inverted, it fires only on operators that would actually go native if the flag were flipped, and stays silent when
useLargeVarTypesor a non-Comet child would prevent the rewrite anyway. This is Spark 4.x-only, matching where the feature exists (the 3.x matchers returnNone).How are these changes tested?
Added two JVM-only cases to the existing
CometMapInBatchSuite(which exercises the rule against a stubbedMapInArrowExecover a Comet leaf without spinning up Python): one asserts the[COMET-INFO]hint and config key appear in extended-explain output when the feature is disabled, and one asserts no hint is emitted when it is enabled. No plan-stability golden files are affected, as no TPCDS query uses these operators.