Skip to content

Allow overriding the vtbackup data volume per tablet pool (#661)#798

Open
mcrauwel wants to merge 5 commits into
mainfrom
mcrauwel/vtbackup-data-volume-override
Open

Allow overriding the vtbackup data volume per tablet pool (#661)#798
mcrauwel wants to merge 5 commits into
mainfrom
mcrauwel/vtbackup-data-volume-override

Conversation

@mcrauwel

Copy link
Copy Markdown
Member

Summary

Fixes #661.

vtbackup Pods — both the initial --initial_backup taken at shard creation and any periodic/scheduled backups — inherit their spec from the tablet pool, including dataVolumeClaimTemplate. That forces a PVC (and underlying disk) to be allocated per shard just to bootstrap an empty backup. In large clusters this is wasteful and sometimes unschedulable, as called out in the issue.

This adds an optional vtbackup block on VitessShardTabletPool with a dataVolumeClaimTemplate override, decoupling the vtbackup Pod's storage from the tablets':

tabletPools:
- cell: cell1
  type: replica
  replicas: 3
  dataVolumeClaimTemplate:        # tablets still get their normal PVC
    accessModes: [ReadWriteOnce]
    resources: {requests: {storage: 100Gi}}
  vtbackup:
    # omit dataVolumeClaimTemplate => vtbackup Pods run with NO PVC (emptyDir)
    # or set one to use a smaller disk / different storageClass:
    dataVolumeClaimTemplate:
      storageClassName: cheap-disk
      accessModes: [ReadWriteOnce]
      resources: {requests: {storage: 10Gi}}

Semantics

vtbackup block vtbackup Pod data volume
omitted inherits the pool's dataVolumeClaimTemplate (unchanged behavior)
present, dataVolumeClaimTemplate omitted no PVC — ephemeral emptyDir scratch space
dataVolumeClaimTemplate set uses that template (smaller disk, different storageClass, etc.)

Both code paths funnel through MakeVtbackupSpec, so the override applies consistently to initial and scheduled backups. Tablet Pods are unaffected. Default behavior is fully backwards compatible.

Scope note

This first pass intentionally covers only the data volume — the concrete pain in the issue. The broader "override the whole vtbackup pod spec" idea (and the comment's request to label the vitessbackupstorage subcontroller) can build on this vtbackup block in follow-ups.

Tests

  • Unit (reconcile_backup_job_test.go): all three states of the data-volume override.
  • Integration: an empty vtbackup: {} override on the first pool yields a vtbackup-init Pod with no PVC-backed volume, while the tablet Pods keep their PVCs. (Verified locally on arm64 envtest.)
  • Regenerated CRDs / deepcopy / API docs.

🤖 Generated with Claude Code

vtbackup Pods (the initial backup taken at shard creation and any
periodic/scheduled backups) inherit their spec from the tablet pool,
including DataVolumeClaimTemplate. That forces a PVC and disk allocation
per shard just to bootstrap an empty backup, which is wasteful and
sometimes unschedulable in large clusters.

Add an optional `vtbackup` block on VitessShardTabletPool with a
`dataVolumeClaimTemplate` override:

  - block omitted        -> inherit the pool's data volume (unchanged).
  - block present, empty -> vtbackup Pods run with no PVC (ephemeral
                            emptyDir scratch space).
  - template set         -> vtbackup Pods use that template (e.g. a
                            smaller disk or a different storageClass).

Both vtbackup code paths funnel through MakeVtbackupSpec, so the override
applies consistently to initial and scheduled backups. Tablet Pods are
unaffected.

Tests: unit coverage of all three states, plus an integration assertion
that an empty override yields a vtbackup-init Pod with no PVC-backed
volume while the tablets keep their PVCs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Matthias Crauwels <matthias.crauwels@planetscale.com>
@mattlord
mattlord requested review from frouioui and mattlord July 18, 2026 19:23
mattlord added 2 commits July 18, 2026 19:24
Signed-off-by: Matt Lord <mattalord@gmail.com>
Comment on lines +687 to +689
if vtbackupSpec.TabletSpec.DataVolumePVCSpec == nil {
return job, nil
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should fail with an error here instead. Otherwise the job we create won't have any corresponding PVC and will fail.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at this more closely and I don't think the Job references a missing PVC. NewBackupPod derives both the volume and mounts from DataVolumePVCSpec; when it is nil, it omits the PVC volume and mounts /vt/vtdataroot from the existing vt-root emptyDir. That no-PVC Job is the behavior requested by #661. I've restored this path locally and strengthened the test to assert that the Job has no PVC-backed volume and that no PVC object is created.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in c2e444c.

Comment on lines +223 to +226
dataVolumeClaimTemplate := pool.DataVolumeClaimTemplate
if vts.Spec.Vtbackup != nil {
dataVolumeClaimTemplate = vts.Spec.Vtbackup.DataVolumeClaimTemplate
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vts.Spec.Vtbackup.DataVolumeClaimTemplate is an optional field, we should only override the dataVolumeClaimTemplate if we the user has defined something in the CR.

@mattlord mattlord Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the nil inner field is intentional here. The two pointers provide the three states this API needs: no vtbackup block inherits the pool PVC; an empty block selects emptyDir; a non-nil dataVolumeClaimTemplate selects a custom PVC. Checking both pointers collapses the first two states and leaves no way to request the no-PVC behavior from #661. I've restored the three-state handling and its tests in c2e444c.

Comment on lines +119 to +122
// Vtbackup can optionally override the data volume used by the initial and
// scheduled vtbackup Pods for this shard. When omitted, vtbackup Pods inherit
// the first tablet pool's DataVolumeClaimTemplate.
Vtbackup *VitessShardVtbackup `json:"vtbackup,omitempty"`

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are modifying the CRD, we need to re-generate the operator yaml files, the test files, and port the related changes to the vitess repository, like so: #629 (comment)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can do one vitessio/vitess PR once this and #797 are both merged.

mattlord added 2 commits July 22, 2026 01:31
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
@mattlord
mattlord requested review from frouioui July 22, 2026 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Allow vtbackup pod spec to be override-able.

3 participants