[SPARK-57791][4.0][INFRA] Backport CI precompile artifact sharing, Coursier cache unification, and zstd in CI images#56913
Closed
gaogaotiantian wants to merge 1 commit into
Conversation
…ursier cache unification, and zstd in CI images This backports the CI build-time optimization series to branch-4.0, combining the work that landed on branch-4.1 as apache#56798 (precompile artifact sharing + unified Coursier cache) and apache#56911 (install zstd in the CI container images). build_and_test.yml: a precompile job builds Spark once with SBT and publishes the compile output (target/ dirs, zstd-compressed) as an artifact. The build, pyspark, sparkr, tpcds-1g, docker-integration-tests and k8s-integration-tests jobs download and extract it, falling back to a local SBT build when the precompile job is absent or fails. The run-tests-driven jobs export SKIP_SCALA_BUILD to skip the local build. Per-job Coursier caches are unified under a single ${{ runner.os }}-coursier key, with the precompile job as the sole writer and all consumers restore-only. maven_test.yml: a precompile-maven job runs `clean install` once and publishes target/ plus ~/.m2/.../spark as an artifact for the matrix entries. Adapted for branch-4.0, which has no SPARK-51628 assembly-wipe: the full target/ is shipped (including assembly), so the reuse path simply skips the local `clean install` and needs no assembly rebuild for connect. No macOS cache guards are added since SPARK-54466 is not on branch-4.0 and the existing build job caches unconditionally. python_macos_test.yml: branch-4.0's analog of branch-4.1's python_hosted_runner_test.yml. A precompile job builds Spark with SBT on macOS and publishes a macOS-distinct artifact that the matrix consumes. dev/run-tests.py: guard build_apache_spark and build_spark_assembly_sbt behind SKIP_SCALA_BUILD so the test run reuses the precompiled output. Dockerfiles: install zstd in dev/infra/Dockerfile and the docs, lint, sparkr, python-311 and pypy-310 images under dev/spark-test-image/, the set the branch-4.0 scheduler builds. Without zstd the precompiled artifact's `zstd -dc` extraction fails in the container jobs, and actions/cache silently falls back to gzip and never restores zstd-saved caches. CI-only; no user-facing change. Co-authored-by: Isaac
zhengruifeng
approved these changes
Jul 1, 2026
Contributor
Author
|
Confirmed from the CIs that pre-compiled artifacts are properly built and used. |
gaogaotiantian
added a commit
that referenced
this pull request
Jul 1, 2026
…ursier cache unification, and zstd in CI images ### What changes were proposed in this pull request? This backports the CI build-time optimization series to `branch-4.0`, combining the work that landed on `branch-4.1` as #56798 (precompile artifact sharing + unified Coursier cache) and #56911 (install `zstd` in the CI container images) into a single PR. - **`build_and_test.yml`**: a `precompile` job builds Spark once with SBT and publishes the compile output (`target/` dirs, zstd-compressed) as an artifact. The `build`, `pyspark`, `sparkr`, `tpcds-1g`, `docker-integration-tests`, and `k8s-integration-tests` jobs download and extract it, falling back to a local SBT build if the precompile job is absent or fails. The `run-tests`-driven jobs export `SKIP_SCALA_BUILD` to skip the local build. The per-job Coursier caches are unified under a single `${{ runner.os }}-coursier` key, with the `precompile` job as the sole writer and all consumers restore-only. - **`maven_test.yml`**: a `precompile-maven` job runs `clean install` once and publishes `target/` plus `~/.m2/.../spark` as an artifact for the matrix entries to consume. - **`python_macos_test.yml`**: `branch-4.0`'s analog of `branch-4.1`'s `python_hosted_runner_test.yml`. A `precompile` job builds Spark with SBT on macOS and publishes a macOS-distinct artifact that the matrix consumes. - **`dev/run-tests.py`**: guard `build_apache_spark` and `build_spark_assembly_sbt` behind `SKIP_SCALA_BUILD` so the test run reuses the precompiled output. - **Dockerfiles**: install `zstd` in `dev/infra/Dockerfile` and the `docs`, `lint`, `sparkr`, `python-311`, and `pypy-310` images under `dev/spark-test-image/`. Adaptations for `branch-4.0`: - `maven_test.yml`: `branch-4.0` has no SPARK-51628 assembly-wipe, so the full `target/` (including `assembly`) is shipped and the reuse path simply skips the local `clean install` with no assembly rebuild needed for the connect tests. No macOS cache guards are added, since SPARK-54466 is not on `branch-4.0` and the existing build job already caches unconditionally. - The `zstd` Dockerfile changes cover only the images the `branch-4.0` scheduler (`branch40_scheduler.yml`) builds: `docs`, `lint`, `sparkr`, `python-311`, and `pypy-310` (and `dev/infra`). `python-314` does not exist on `branch-4.0` and is not scheduled, so it is omitted. ### Why are the changes needed? To cut redundant Scala/Maven compilation and Coursier cache duplication on `branch-4.0` CI, matching the optimization already present on the newer branches. Without `zstd` in the container images, `build_and_test.yml`'s `zstd -dc compile-artifact.tar.zst` extraction fails in the container jobs, and `actions/cache` silently falls back to `gzip` and never restores caches saved by host jobs. ### Does this PR introduce _any_ user-facing change? No. CI-only. ### How was this patch tested? CI on this PR. The three workflow files validate with `python3 -c "import yaml; yaml.safe_load(...)"`, and `dev/run-tests.py` compiles cleanly. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (claude-opus-4-8) Closes #56913 from gaogaotiantian/SPARK-57791-precompile-backport-4.0. Authored-by: Tian Gao <gaogaotiantian@hotmail.com> Signed-off-by: Tian Gao <gaogaotiantian@hotmail.com>
Contributor
|
it seems the merge script cannot close a pr targeting non-master branch, let me take a look seems due to old merge script |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This backports the CI build-time optimization series to
branch-4.0, combining the work that landed onbranch-4.1as #56798 (precompile artifact sharing + unified Coursier cache) and #56911 (installzstdin the CI container images) into a single PR.build_and_test.yml: aprecompilejob builds Spark once with SBT and publishes the compile output (target/dirs, zstd-compressed) as an artifact. Thebuild,pyspark,sparkr,tpcds-1g,docker-integration-tests, andk8s-integration-testsjobs download and extract it, falling back to a local SBT build if the precompile job is absent or fails. Therun-tests-driven jobs exportSKIP_SCALA_BUILDto skip the local build. The per-job Coursier caches are unified under a single${{ runner.os }}-coursierkey, with theprecompilejob as the sole writer and all consumers restore-only.maven_test.yml: aprecompile-mavenjob runsclean installonce and publishestarget/plus~/.m2/.../sparkas an artifact for the matrix entries to consume.python_macos_test.yml:branch-4.0's analog ofbranch-4.1'spython_hosted_runner_test.yml. Aprecompilejob builds Spark with SBT on macOS and publishes a macOS-distinct artifact that the matrix consumes.dev/run-tests.py: guardbuild_apache_sparkandbuild_spark_assembly_sbtbehindSKIP_SCALA_BUILDso the test run reuses the precompiled output.zstdindev/infra/Dockerfileand thedocs,lint,sparkr,python-311, andpypy-310images underdev/spark-test-image/.Adaptations for
branch-4.0:maven_test.yml:branch-4.0has no SPARK-51628 assembly-wipe, so the fulltarget/(includingassembly) is shipped and the reuse path simply skips the localclean installwith no assembly rebuild needed for the connect tests. No macOS cache guards are added, since SPARK-54466 is not onbranch-4.0and the existing build job already caches unconditionally.zstdDockerfile changes cover only the images thebranch-4.0scheduler (branch40_scheduler.yml) builds:docs,lint,sparkr,python-311, andpypy-310(anddev/infra).python-314does not exist onbranch-4.0and is not scheduled, so it is omitted.Why are the changes needed?
To cut redundant Scala/Maven compilation and Coursier cache duplication on
branch-4.0CI, matching the optimization already present on the newer branches. Withoutzstdin the container images,build_and_test.yml'szstd -dc compile-artifact.tar.zstextraction fails in the container jobs, andactions/cachesilently falls back togzipand never restores caches saved by host jobs.Does this PR introduce any user-facing change?
No. CI-only.
How was this patch tested?
CI on this PR. The three workflow files validate with
python3 -c "import yaml; yaml.safe_load(...)", anddev/run-tests.pycompiles cleanly.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-8)