Consider providing an option for dynamic linking

[sergiimk]

Is your feature request related to a problem or challenge?

datafusion is a "swiss army knife" kind of library - it's large and developing against it means ~13 second link times every time you re-run tests, even with modern linkers like mold.

I recently tried the approach described in this blog post - wrapping datafusion into a dynamic library. This reduced my incremental test build times by ~10 seconds!

Describe the solution you'd like

Bevy project (also a very large library) now offer a feature flag to link it dynamically.

Specifying crate-type = ["rlib", "dylib"] might also be enough to let users easily link datafusion dynamically in their dev builds using RUSTFLAGS="-C prefer-dynamic".

It may iterations speeds for datafusion's own development.

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

Thanks @sergiimk -- this is quite interesting

I think one reason for the long cycle time is not just re-link but also that datafusion-core is large and any change to it requires recompiling the entire crate.

Another approach I have seen taken in the rust community is to break such large crates into smaller ones. Not only does this tend to increase test cycle times it also encourages more encapsulation.

We have been chipping away at doing this (e.g. #1754) but extracting datafusion-physical-plan and datafusion-datasource are especially connected.

Maybe it is time to invest some more time trying to split out those crates 🤔

[sergiimk]

As a user of datafusion I rarely recompile the library itself, but I do pay 13s in link time every time I touch my app or tests.

So while splitting up crates can speed up your dev iterations, think of dynamic linking as primarily user-facing speedup feature.

[sergiimk]

I've prototyped a feature flag that allows users of datafusion to link it as a shared library.

Reusing DF across dozens of tests and example binaries in our large workspace reduced our dev build footprint from 112 to 57 GiB and shaved off 10s from executable linking times. Seems like a huge win for local development.

Would love to get feedback on the approach and see if there's an interest to upstream this. Draft PR: #23767

[alamb]

@timsaucer -- is this related (possibly a better approach) to using datafusion via python bindings (or maybe it isn't related as the different libraries will need to pass data structures anyways 🤔