Skip to content

perf: optimize spark_cast_utf8_to_boolean (>2x faster)#4914

Open
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/spark_cast_utf8_to_boolean-datafusion-comet-20260714-030947
Open

perf: optimize spark_cast_utf8_to_boolean (>2x faster)#4914
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/spark_cast_utf8_to_boolean-datafusion-comet-20260714-030947

Conversation

@andygrove

@andygrove andygrove commented Jul 14, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A

Rationale for this change

Optimize existing expressions.

What changes are included in this PR?

Replaced the per-row to_ascii_lowercase() String allocation in the string-to-boolean cast with an allocation-free length-dispatched eq_ignore_ascii_case match, and dropped the per-row Result for the non-ANSI eval modes.

How are these changes tested?

Existing tests + new unit tests.

Benchmark (criterion):

  • legacy_mixed: 68.325% faster (base 156779ns -> cand 49659ns)
  • try_mixed: 68.699% faster (base 158363ns -> cand 49570ns)
  • legacy_valid: 67.019% faster (base 169509ns -> cand 55905ns)
  • ansi_valid: 68.293% faster (base 171297ns -> cand 54313ns)
  • try_valid: 67.195% faster (base 169282ns -> cand 55533ns)

Full criterion output:

cast_string_to_boolean/legacy/mixed
                        time:   [49.828 µs 50.046 µs 50.365 µs]
                        change: [−68.437% −68.325% −68.202%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe
cast_string_to_boolean/legacy/valid
                        time:   [55.765 µs 55.855 µs 55.965 µs]
                        change: [−67.067% −67.019% −66.955%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 16 outliers among 100 measurements (16.00%)
  6 (6.00%) low severe
  1 (1.00%) low mild
  9 (9.00%) high severe
cast_string_to_boolean/try/mixed
                        time:   [49.542 µs 49.571 µs 49.615 µs]
                        change: [−68.723% −68.699% −68.676%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high severe
cast_string_to_boolean/try/valid
                        time:   [55.479 µs 55.502 µs 55.533 µs]
                        change: [−67.236% −67.195% −67.142%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) high mild
  4 (4.00%) high severe
cast_string_to_boolean/ansi/valid
                        time:   [54.193 µs 54.227 µs 54.275 µs]
                        change: [−68.532% −68.293% −68.070%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

@andygrove andygrove changed the title perf: optimize spark_cast_utf8_to_boolean in datafusion-comet-spark-expr perf: optimize spark_cast_utf8_to_boolean (>2x faster) Jul 14, 2026
@andygrove andygrove marked this pull request as ready for review July 14, 2026 14:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant