[SPARK-57883][CORE][TESTS] Fix PluginContainerSuite to check executor memory correctly

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to fix PluginContainerSuite to check executor memory correctly.

Why are the changes needed?

Previously, it checks driver accidentally because it didn't filter out driver from the executor list.

spark/core/src/test/scala/org/apache/spark/internal/plugin/PluginContainerSuite.scala

Lines 261 to 263 in e3e6d9e

	// Check executor memory is also updated
	val execInfo = sc.statusTracker.getExecutorInfos.head
	assert(execInfo.totalOffHeapStorageMemory() == MemoryOverridePlugin.offHeapMemory)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Could you review this PR when you have some time, @viirya ? I decided to fix this bug first from PluginContainerSuite before moving forward the original PR.

• [SPARK-57867][CORE] Driver should not reserve off-heap memory in non-local mode #56945

[viirya]

+1, LGTM. Verified the root cause independently: getExecutorInfos goes through AppStatusStore.executorList(true), which reads base.index("active").reverse().first(true).last(true) — within the index, entries sort by executor id, so after reverse() the "driver" entry (letters sort after digits) comes first and .head was indeed always the driver. The old assertion only stayed green because the driver happened to carry the same off-heap value, so the executor side was never actually verified.

The eventually is necessary rather than just defensive: waitUntilExecutorsUp only guarantees the executor entry exists (from onExecutorAdded), while memoryMetrics is populated later when the executor's BlockManager registers, so the first read can legitimately see None.

With this fixed, the test genuinely verifies that plugin-set off-heap settings reach the executors — which also makes it a useful guard for #56945.

[dongjoon-hyun]

Thank you, @viirya !

[dongjoon-hyun]

Since this is a test suite only change and I verified locally like the following, I'm going to merge.

$ build/sbt "core/testOnly *.PluginContainerSuite"
...
[info] PluginContainerSuite:
[info] - plugin initialization and communication (1 second, 156 milliseconds)
[info] - do nothing if plugins are not configured (65 milliseconds)
[info] - merging of config options (33 milliseconds)
[info] - SPARK-33088: executor tasks trigger plugin calls (435 milliseconds)
[info] - SPARK-33088: executor failed tasks trigger plugin calls (57 milliseconds)
[info] - plugin initialization in non-local mode (1 second, 753 milliseconds)
[info] - plugin initialization in non-local mode with resources (1 second, 464 milliseconds)
[info] - memory override in plugin (1 second, 487 milliseconds)
[info] - The plugin should be shutdown before the listener bus is stopped (25 milliseconds)
[info] - SPARK-52548: override shuffle manager in plugin (21 milliseconds)
[info] Run completed in 7 seconds, 490 milliseconds.
[info] Total number of tests run: 10
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 10, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 12 s, completed Jul 2, 2026, 9:43:17 AM

[dongjoon-hyun]

Merge Summary:

• merged into master 17e7e14
• merged into branch-4.x 8cadba0

Posted by merge_spark_pr.py