Add optional native Lance scan support

[wirybeaver]

What is the problem the feature request solves?

Comet has native scan support for formats such as Iceberg, but Lance tables planned by Spark through Lance Spark currently execute through Spark's Lance reader. A native Lance path would let Comet read ordinary Lance table scans directly in Rust while preserving Spark/Lance Spark as the planning contract.

This should be optional and dependency-free for default Comet builds.

Describe the potential solution

Add Lance as an experimental, opt-in Comet contrib reader:

• Keep Spark planning Lance tables through Lance Spark.
• Let Comet core expose only a generic V2 scan contrib hook: CometScanContrib, discovered with ServiceLoader from optional contrib profiles.
• Move Lance-specific scan detection and nativeScanPlan() reflection into contrib-lance, not core.
• Extract a stable native-read descriptor exposed by Lance Spark.
• Add a build-gated contrib-lance profile / Cargo feature so default builds do not depend on Lance.
• Add a typed lance_scan = 118 native proto payload with common scan invariants and per-partition split descriptors.
• Surface common/per-partition scan payloads through CometScanWithPlanData and inject them through a contrib PlanDataInjector service.
• Execute the assigned Lance fragments through the native Rust Lance API.
• Gate runtime activation behind spark.comet.scan.lanceNative.enabled=false by default.

Minimal v1 should target ordinary Lance table reads only: local/direct object-store storage options, projection, filter SQL, limit/offset, batch size, fragment splits, and Comet-supported Spark types. Search, hybrid search, index-backed planning, namespace-backed credential refresh, metadata/version columns, and aggregation pushdown should be added in later phases after separate semantic review.

Architecture sketch

The key contract is that Lance Spark owns Spark planning and scan semantics. Comet core only provides generic contrib extension points; the optional Lance contrib consumes Lance Spark's explicit native-read descriptor and executes it with Rust Lance when the profile and runtime config are enabled.

Spark SQL / DataFrame read.format("lance")
        |
        v
+--------------------------------------+
| Spark planner + lance-spark           |
|                                      |
| org.lance.spark.read.LanceScan        |
| - resolves dataset version            |
| - applies projection/filter           |
| - computes partition splits           |
| - owns Lance Spark semantics          |
| - exposes nativeScanPlan() descriptor |
+-------------------+------------------+
                    |
                    | Spark physical plan:
                    | BatchScanExec(scan = LanceScan)
                    v
+--------------------------------------+
| Comet core                           |
|                                      |
| CometScanRule                        |
| - handles built-in scans             |
| - calls CometScanContrib SPI         |
|                                      |
| Core Lance footprint:                |
| - config key                         |
| - lance_scan proto                   |
| - generic contrib dispatch           |
| - Rust feature gates                 |
|                                      |
| No Lance Spark classes or reflection |
+-------------------+------------------+
                    |
                    | ServiceLoader, only when
                    | -Pcontrib-lance is on classpath
                    v
+--------------------------------------+
| Comet contrib-lance (Scala)          |
|                                      |
| META-INF/services/...CometScanContrib|
|   -> LanceScanRuleExtension          |
|                                      |
| LanceScanRuleExtension               |
| - recognizes LanceScan by reflection |
| - calls LanceScan.nativeScanPlan()   |
| - checks Comet/Lance configs         |
| - returns original scan on fallback  |
|                                      |
| CometLanceSupport                    |
| CometLanceNativeScan                 |
| CometLanceNativeScanExec             |
| - validates Comet schema             |
| - serializes lance_scan common data  |
| - exposes CometScanWithPlanData      |
|                                      |
| META-INF/services/...PlanDataInjector|
|   -> LancePlanDataInjector           |
| - injects per-task Lance partitions  |
+-------------------+------------------+
                    |
                    | native proto payload:
                    | lance_scan = 118
                    |   LanceScanCommon
                    |   LanceScanPartition(fragment_ids)
                    v
+--------------------------------------+
| Comet native Rust                    |
|                                      |
| planner.rs                           |
| - accepts LanceScan only when        |
|   --features contrib-lance is set    |
|                                      |
| LanceScanExec                        |
| - opens DatasetBuilder               |
| - pins resolved version              |
| - applies storage options            |
| - Scanner::with_fragments            |
| - projection/filter/limit/batch size |
+-------------------+------------------+
                    |
                    v
+--------------------------------------+
| Rust Lance crate + storage           |
| local/object store dataset           |
| Arrow RecordBatch stream             |
+-------------------+------------------+
                    |
                    v
+--------------------------------------+
| Comet columnar execution             |
| returns Spark-compatible batches     |
+--------------------------------------+

Fallback paths:

  - contrib-lance profile absent
  - spark.comet.scan.lanceNative.enabled=false
  - Comet native execution disabled
  - LanceScan.nativeScanPlan() absent/empty
  - descriptor/schema/storage feature unsupported

        -> Comet keeps the original BatchScanExec
        -> Spark executes the scan through lance-spark

Descriptor boundary:

lance-spark PR
  LanceScan.nativeScanPlan(): Optional[LanceNativeScanPlan]
    - dataset URI
    - resolved version
    - Spark/projected schema JSON
    - projection and pushed filter SQL
    - limit/offset and batch size
    - storage options
    - per-partition fragment IDs
    - fallback reasons

Comet PR
  Core CometScanRule delegates to ServiceLoader-backed CometScanContrib
  contrib-lance reflects LanceNativeScanPlan from LanceScan
  Scala serde converts it to lance_scan proto
  PlanDataInjector injects partition payloads per task
  Rust LanceScanExec executes exactly those fragments/version/options

Additional context

The phased roadmap is:

1. Lance Spark native descriptor for ordinary reads.
2. Comet contrib scaffold and generic scan SPI bridge.
3. Minimal native Rust Lance scan v1.
4. V1 hardening and CI parity tests.
5. Advanced table-read parity and metrics.
6. Lance index/search read support.
7. Remote namespace and credential refresh support.

Known prototype blocker to resolve before this can be considered merge-ready: packaged Comet currently contains org.apache.arrow.c classes rewritten against Comet's shaded Arrow allocator, while Lance Spark expects the normal Arrow C Data ABI. An end-to-end packaged smoke test with both jars exposes this classpath conflict. The final design needs an explicit Arrow C Data packaging/classloader strategy and CI coverage for Comet + Lance Spark together.

[wirybeaver]

Draft prototype PR attached: #4633. Companion Lance Spark Issue: lance-format/lance-spark/#623 and descriptor PR: lance-format/lance-spark#624.

[wirybeaver]

Potential solution for the Arrow C Data class conflict seen in the packaged smoke test:

The failure does not look solvable by jar ordering. Today the packaged Comet jar can expose classes named org.apache.arrow.c.*, but their bytecode has been rewritten to use Comet's relocated Arrow allocator/types under org.apache.comet.shaded.arrow.*. That produces a hybrid ABI:

• If Comet's jar wins class loading, Lance Spark loads Comet-mutated org.apache.arrow.c.* classes and normal Lance/Arrow calls fail with NoSuchMethodError.
• If Lance Spark's Arrow C Data jar wins class loading, Comet loads normal upstream org.apache.arrow.c.* classes and Comet calls that pass shaded Arrow allocator/types fail with NoSuchMethodError.

The clean boundary should be that Comet never publishes mutated classes under the upstream Arrow package name:

1. Fully relocate Comet's Arrow C Data classes into Comet's shaded namespace, for example org.apache.comet.shaded.arrow.c.*.
2. Ensure the packaged comet-spark jar does not contain org/apache/arrow/c/** classes rewritten against shaded Arrow types.
3. Update Comet JNI/native class lookups that currently reference org/apache/arrow/c/... to use the shaded class names in packaged Comet. If needed for dev/test compatibility, use a small dual-lookup helper that tries shaded first and then unshaded.
4. Let Lance Spark keep using the normal upstream Arrow namespace: org.apache.arrow.c.* and org.apache.arrow.memory.*.
5. Add a packaging regression test that asserts the packaged Comet jar contains shaded Arrow C Data classes and does not contain Comet-mutated upstream org/apache/arrow/c/** classes.
6. Add an integration smoke test with both packaged Comet and Lance Spark jars on the same Spark classpath.

I would avoid solving this with Spark classloader ordering or Lance Spark changes. The durable rule should be: Comet's shaded Arrow universe uses shaded package names consistently, and Lance Spark's upstream Arrow universe remains untouched.

[wirybeaver]

Hi @andygrove, could you take a look at this proposal when you have a chance?

The issue body includes an ASCII architecture sketch for the proposed Lance native read path. The intended boundary is: Spark/lance-spark remains the planning authority and exposes LanceScan.nativeScanPlan(), while Comet consumes that descriptor through optional contrib-lance support and executes the assigned fragments with Rust Lance.

There is a draft prototype in #4633, with the companion Lance Spark descriptor PR at lance-format/lance-spark#624. I would especially appreciate your feedback on whether this fits Comet's contrib/native scan patterns, and on the open Arrow C Data packaging/classpath blocker before this could move beyond draft.

[coderfender]

Tagging @mbutrovich , @comphead , @comphead , @kazuyukitanimura

[kazuyukitanimura]

Since the draft PR is posted by @wirybeaver assigning to him

[wirybeaver]

Thanks for the triaging!