diff --git a/en/backup-restore-cr.md b/en/backup-restore-cr.md index 2a290f244e..6324ec7907 100644 --- a/en/backup-restore-cr.md +++ b/en/backup-restore-cr.md @@ -258,6 +258,33 @@ This section introduces the fields in the `Backup` CR. * `.spec.local.volume`: the persistent volume configuration. * `.spec.local.volumeMount`: the persistent volume mount configuration. +## CompactBackup CR fields + +For TiDB v9.0.0 and later versions, you can use `CompactBackup` to accelerate PITR (Point-in-time recovery). To compact log backup data into structured SST files, you can create a custom `CompactBackup` CR object to define a backup task. The following introduces the fields in the `CompactBackup` CR: + +* `.spec.startTs`: the start timestamp for log compaction backup. +* `.spec.endTs`: the end timestamp for log compaction backup. +* `.spec.concurrency`: the maximum number of concurrent log compaction tasks. The default value is `4`. +* `.spec.maxRetryTimes`: the maximum number of retries for failed compaction tasks. The default value is `6`. +* `.spec.toolImage`:the tool image used by `CompactBackup`. BR is the only tool image used in `CompactBackup`. When using BR for backup, you can specify the BR version with this field: + + - If not specified or left empty, the `pingcap/br:${tikv_version}` image is used for backup by default. + - If a BR version is specified, such as `.spec.toolImage: pingcap/br:v9.0.0`, the image of the specified version is used for backup. + - If an image is specified without a version, such as `.spec.toolImage: private/registry/br`, the `private/registry/br:${tikv_version}` image is used for backup. + +* `.spec.env`: the environment variables for the Pod that runs the compaction task. +* `.spec.affinity`: the affinity configuration for the Pod that runs the compaction task. For details on affinity, refer to [Affinity and anti-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity). +* `.spec.tolerations`: specifies that the Pod that runs the compaction task can schedule onto nodes with matching [taints](https://kubernetes.io/docs/reference/glossary/?all=true#term-taint). For details on taints and tolerations, refer to [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). +* `.spec.podSecurityContext`: the security context configuration for the Pod that runs the compaction task, which allows the Pod to run as a non-root user. For details on `podSecurityContext`, refer to [Run Containers as a Non-root User](containers-run-as-non-root-user.md). +* `.spec.priorityClassName`: the name of the priority class for the Pod that runs the compaction task, which sets priority for the Pod. For details on priority classes, refer to [Pod Priority and Preemption](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/). +* `.spec.imagePullSecrets`: the [imagePullSecrets](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod) for the Pod that runs the compaction task. +* `.spec.serviceAccount`: the name of the ServiceAccount used for compact. +* `.spec.useKMS`: whether to use AWS-KMS to decrypt the S3 storage key used for the backup. +* `.spec.br`: BR-related configuration. For more information, refer to [BR fields](#br-fields). +* `.spec.s3`: S3-related configuration. For more information, refer to [S3 storage fields](#s3-storage-fields). +* `.spec.gcs`: GCS-related configuration. For more information, refer to [GCS fields](#gcs-fields). +* `.spec.azblob`:Azure Blob Storage-related configuration. For more information, refer to [Azure Blob Storage fields](#azure-blob-storage-fields). + ## Restore CR fields To restore data to a TiDB cluster on Kubernetes, you can create a `Restore` CR object. For detailed restore process, refer to documents listed in [Restore data](backup-restore-overview.md#restore-data). @@ -352,6 +379,7 @@ The `backupSchedule` configuration consists of three parts: the configuration of * `backupTemplate`: the configuration of the snapshot backup. Specifies the configuration related to the cluster and remote storage of the snapshot backup, which is the same as the `spec` configuration of [the `Backup` CR](#backup-cr-fields). * `logBackupTemplate`: the configuration of the log backup. Specifies the configuration related to the cluster and remote storage of the log backup, which is the same as the `spec` configuration of [the `Backup` CR](#backup-cr-fields). The log backup is created and deleted along with `backupSchedule` and recycled according to `.spec.maxReservedTime`. The log backup name is saved in `status.logBackup`. +* `compactBackupTemplate`: the configuration template of the log compaction backup. The fields are the same as those in the `spec` configuration of [the `CompactBackup` CR](#compactbackup-cr-fields). The compaction backup is created and deleted along with `backupSchedule`. The log backup names are stored in `status.logBackup`. The storage settings of the compaction backup should be the same as that of `logBackupTemplate` in the same `backupSchedule`. > **Note:** > @@ -362,4 +390,5 @@ The `backupSchedule` configuration consists of three parts: the configuration of * `.spec.maxBackups`: a backup retention policy, which determines the maximum number of backup files to be retained. When the number of backup files exceeds this value, the outdated backup file will be deleted. If you set this field to `0`, all backup items are retained. * `.spec.maxReservedTime`: a backup retention policy based on time. For example, if you set the value of this field to `24h`, only backup files within the recent 24 hours are retained. All backup files older than this value are deleted. For the time format, refer to [`func ParseDuration`](https://golang.org/pkg/time/#ParseDuration). If you have set `.spec.maxBackups` and `.spec.maxReservedTime` at the same time, the latter takes effect. * `.spec.schedule`: the time scheduling format of Cron. Refer to [Cron](https://en.wikipedia.org/wiki/Cron) for details. + * `.spec.compactInterval`: the time interval used to trigger a new compaction task. * `.spec.pause`: `false` by default. If this field is set to `true`, the scheduled scheduling is paused. In this situation, the backup operation will not be performed even if the scheduling time point is reached. During this pause, the backup garbage collection runs normally. If you change `true` to `false`, the scheduled snapshot backup process is restarted. Because currently, log backup does not support pause, this configuration does not take effect for log backup. diff --git a/en/backup-to-aws-s3-using-br.md b/en/backup-to-aws-s3-using-br.md index 07cc5b8bf7..d106c0c578 100644 --- a/en/backup-to-aws-s3-using-br.md +++ b/en/backup-to-aws-s3-using-br.md @@ -491,8 +491,7 @@ In TiDB Operator v1.5.4, v1.6.0, and earlier versions, you can use the `logStop: backupMode: log logSubcommand: log-start/log-pause/log-stop br: - cluster: demo1 - clusterNamespace: test1 + mespace: test1 sendCredToTikv: true s3: provider: aws @@ -538,6 +537,65 @@ In TiDB Operator v1.5.4, v1.6.0, and earlier versions, you can use the `logStop: demo1-log-backup-s3 log Stopped ... 2022-10-10T15:21:00+08:00 ``` +### Compact log backup + +For TiDB v9.0.0 and later versions, you can use a `CompactBackup` CR to compact log backup data into SST format, accelerating downstream PITR (Point-in-time recovery). + +This section explains how to compact log backup based on the log backup example from previous sections. + +1. In the `backup-test` namespace, create a `CompactBackup` CR named `demo1-compact-backup`. + + ```shell + kubectl apply -f compact-backup-demo1.yaml + ``` + + The content of `compact-backup-demo1.yaml` is as follows: + + ```yaml + --- + apiVersion: pingcap.com/v1alpha1 + kind: CompactBackup + metadata: + name: demo1-compact-backup + namespace: backup-test + spec: + startTs: "***" + endTs: "***" + concurrency: 8 + maxRetryTimes: 2 + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + s3: + provider: aws + secretName: s3-secret + region: us-west-1 + bucket: my-bucket + prefix: my-log-backup-folder + ``` + + The `startTs` and `endTs` fields specify the time range for the logs to be compacted by `demo1-compact-backup`. Any log that contains at least one write within this time range will be included in the compaction process. As a result, the final compacted data might include data written outside this range. + + The `s3` settings should be the same as the storage settings of the log backup to be compacted. `CompactBackup` reads log files from the corresponding location and compact them. + +#### View the status of log backup compaction + +After creating the `CompactBackup` CR, TiDB Operator automatically starts compacting the log backup. You can check the backup status using the following command: + +```shell +kubectl get cpbk -n backup-test +``` + +From the output, you can find the status of the `CompactBackup` CR named `demo1-compact-backup`. An example output is as follows: + +``` +NAME STATUS PROGRESS MESSAGE +demo1-compact-backup Complete [READ_META(17/17),COMPACT_WORK(1291/1291)] +``` + +If the `STATUS` field displays `Complete`, the compact log backup process has finished successfully. + ### Backup CR examples
@@ -917,6 +975,93 @@ The steps to prepare for a scheduled snapshot backup are the same as those of [P log-integrated-backup-schedule-s3 log Running .... ``` +## Integrated management of scheduled snapshot backup, log backup, and compact log backup + +To accelerate downstream recovery, you can enable `CompactBackup` CR in the `BackupSchedule` CR. This feature periodically compacts log backup files in remote storage. You must enable log backup before using log backup compaction. This section extends the configuration from the previous section. + +### Prerequisites: Prepare for a scheduled snapshot backup + +The steps to prepare for a scheduled snapshot backup are the same as that of [Prepare for an ad-hoc backup](#prerequisites-prepare-for-an-ad-hoc-backup). + +### Create `BackupSchedule` + +1. Create a `BackupSchedule` CR named `integrated-backup-schedule-s3` in the `backup-test` namespace. + + ```shell + kubectl apply -f integrated-backup-schedule-s3.yaml + ``` + + The content of `integrated-backup-schedule-s3.yaml` is as follows: + + ```yaml + --- + apiVersion: pingcap.com/v1alpha1 + kind: BackupSchedule + metadata: + name: integrated-backup-schedule-s3 + namespace: backup-test + spec: + maxReservedTime: "3h" + schedule: "* */2 * * *" + compactInterval: "1h" + backupTemplate: + backupType: full + cleanPolicy: Delete + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + s3: + provider: aws + secretName: s3-secret + region: us-west-1 + bucket: my-bucket + prefix: my-folder-snapshot + logBackupTemplate: + backupMode: log + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + s3: + provider: aws + secretName: s3-secret + region: us-west-1 + bucket: my-bucket + prefix: my-folder-log + compactBackupTemplate: + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + s3: + provider: aws + secretName: s3-secret + region: us-west-1 + bucket: my-bucket + prefix: my-folder-log + ``` + + In the preceding example of `integrated-backup-schedule-s3.yaml`, the `backupSchedule` configuration is based on the previous section, with the following additions for `compactBackup`: + + * Added the `BackupSchedule.spec.compactInterval` field to specify the interval for log backup compaction. It is recommended not to exceed the interval of scheduled snapshot backups and to keep it between one-half to one-third of the scheduled snapshot backup interval. + + * Added the `BackupSchedule.spec.compactBackupTemplate` field. Ensure that the `BackupSchedule.spec.compactBackupTemplate.s3` configuration matches the `BackupSchedule.spec.logBackupTemplate.s3` configuration. + + For the field description of `backupSchedule`, refer to [BackupSchedule CR fields](backup-restore-cr.md#backupschedule-cr-fields). + +2. After creating `backupSchedule`, use the following command to check the backup status: + + ```shell + kubectl get bks -n backup-test -o wide + ``` + + A compact log backup task is created together with `backupSchedule`. You can check the `CompactBackup` CR using the following command: + + ```shell + kubectl get cpbk -n backup-test + ``` + ## Delete the backup CR If you no longer need the backup CR, refer to [Delete the Backup CR](backup-restore-overview.md#delete-the-backup-cr). diff --git a/en/backup-to-azblob-using-br.md b/en/backup-to-azblob-using-br.md index f5fce7bd28..2b05398941 100644 --- a/en/backup-to-azblob-using-br.md +++ b/en/backup-to-azblob-using-br.md @@ -449,6 +449,63 @@ In TiDB Operator v1.5.4, v1.6.0, and earlier versions, you can use the `logStop: demo1-log-backup log Complete ... 2022-10-10T15:21:00+08:00 ``` +### Compact log backup + +For TiDB v9.0.0 and later versions, you can use a `CompactBackup` CR to compact log backup data into SST format, accelerating downstream PITR (Point-in-time recovery). + +This section explains how to compact log backup based on the log backup example from previous sections. + +1. In the `backup-test` namespace, create a `CompactBackup` CR named `demo1-compact-backup`. + + ```shell + kubectl apply -f compact-backup-demo1.yaml + ``` + + The content of `compact-backup-demo1.yaml` is as follows: + + ```yaml + --- + apiVersion: pingcap.com/v1alpha1 + kind: CompactBackup + metadata: + name: demo1-compact-backup + namespace: backup-test + spec: + startTs: "***" + endTs: "***" + concurrency: 8 + maxRetryTimes: 2 + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + azblob: + secretName: azblob-secret + container: my-container + prefix: my-log-backup-folder + ``` + + The `startTs` and `endTs` fields specify the time range for the logs to be compacted by `demo1-compact-backup`. Any log that contains at least one write within this time range will be included in the compaction process. As a result, the final compacted data might include data written outside this range. + + The `azblob` settings should be the same as the storage settings of the log backup to be compacted. `CompactBackup` reads log files from the corresponding location and compact them. + +#### View the status of log backup compaction + +After creating the `CompactBackup` CR, TiDB Operator automatically starts compacting the log backup. You can check the backup status using the following command: + +```shell +kubectl get cpbk -n backup-test +``` + +From the output, you can find the status of the `CompactBackup` CR named `demo1-compact-backup`. An example output is as follows: + +``` +NAME STATUS PROGRESS MESSAGE +demo1-compact-backup Complete [READ_META(17/17),COMPACT_WORK(1291/1291)] +``` + +If the `STATUS` field displays `Complete`, the compact log backup process has finished successfully. + ### Backup CR examples
@@ -768,6 +825,89 @@ The steps to prepare for a scheduled snapshot backup are the same as those of [P log-integrated-backup-schedule-azblob log Running .... ``` +## Integrated management of scheduled snapshot backup, log backup, and compact log backup + +To accelerate downstream recovery, you can enable `CompactBackup` CR in the `BackupSchedule` CR. This feature periodically compacts log backup files in remote storage. You must enable log backup before using log backup compaction. This section extends the configuration from the previous section. + +### Prerequisites: Prepare for a scheduled snapshot backup + +The steps to prepare for a scheduled snapshot backup are the same as that of [Prepare for an ad-hoc backup](#prerequisites-prepare-an-ad-hoc-backup-environment). + +### Create `BackupSchedule` + +1. Create a `BackupSchedule` CR named `integrated-backup-schedule-azblob` in the `backup-test` namespace. + + ```shell + kubectl apply -f integrated-backup-schedule-azblob.yaml + ``` + + The content of `integrated-backup-schedule-azblob.yaml` is as follows: + + ```yaml + --- + apiVersion: pingcap.com/v1alpha1 + kind: BackupSchedule + metadata: + name: integrated-backup-schedule-azblob + namespace: backup-test + spec: + maxReservedTime: "3h" + schedule: "* */2 * * *" + backupTemplate: + backupType: full + cleanPolicy: Delete + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + azblob: + secretName: azblob-secret + container: my-container + prefix: schedule-backup-folder-snapshot + #accessTier: Hot + logBackupTemplate: + backupMode: log + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + azblob: + secretName: azblob-secret + container: my-container + prefix: schedule-backup-folder-log + #accessTier: Hot + compactBackupTemplate: + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + azblob: + secretName: azblob-secret + container: my-container + prefix: schedule-backup-folder-log + #accessTier: Hot + ``` + + In the preceding example of `integrated-backup-schedule-azblob.yaml`, the `backupSchedule` configuration is based on the previous section, with the following additions for `compactBackup`: + + * Added the `BackupSchedule.spec.compactInterval` field to specify the time interval for log backup compaction. It is recommended not to exceed the interval of scheduled snapshot backups and to keep it between one-half to one-third of the scheduled snapshot backup interval. + + * Added the `BackupSchedule.spec.compactBackupTemplate` field. Ensure that the `BackupSchedule.spec.compactBackupTemplate.azblob` configuration matches the `BackupSchedule.spec.logBackupTemplate.azblob` configuration. + + For the field description of `backupSchedule`, refer to [BackupSchedule CR fields](backup-restore-cr.md#backupschedule-cr-fields). + +2. After creating `backupSchedule`, use the following command to check the backup status: + + ```shell + kubectl get bks -n backup-test -o wide + ``` + + A compact log backup task is created together with `backupSchedule`. You can check the `CompactBackup` CR using the following command: + + ```shell + kubectl get cpbk -n backup-test + ``` + ## Delete the backup CR If you no longer need the backup CR, you can delete it by referring to [Delete the Backup CR](backup-restore-overview.md#delete-the-backup-cr). diff --git a/en/backup-to-gcs-using-br.md b/en/backup-to-gcs-using-br.md index 514f50823a..03e2f8075e 100644 --- a/en/backup-to-gcs-using-br.md +++ b/en/backup-to-gcs-using-br.md @@ -449,6 +449,64 @@ In TiDB Operator v1.5.4, v1.6.0, and earlier versions, you can use the `logStop: demo1-log-backup-gcs log Stopped ... 2022-10-10T15:21:00+08:00 ``` +### Compact log backup + +For TiDB v9.0.0 and later versions, you can use a `CompactBackup` CR to compact log backup data into SST format, accelerating downstream PITR (Point-in-time recovery). + +This section explains how to compact log backup based on the log backup example from previous sections. + +1. In the `backup-test` namespace, create a `CompactBackup` CR named `demo1-compact-backup`. + + ```shell + kubectl apply -f compact-backup-demo1.yaml + ``` + + The content of `compact-backup-demo1.yaml` is as follows: + + ```yaml + --- + apiVersion: pingcap.com/v1alpha1 + kind: CompactBackup + metadata: + name: demo1-compact-backup + namespace: backup-test + spec: + startTs: "***" + endTs: "***" + concurrency: 8 + maxRetryTimes: 2 + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + gcs: + projectId: ${project_id} + secretName: gcs-secret + bucket: my-bucket + prefix: my-log-backup-folder + ``` + + The `startTs` and `endTs` fields specify the time range for the logs to be compacted by `demo1-compact-backup`. Any log that contains at least one write within this time range will be included in the compaction process. As a result, the final compacted data might include data written outside this range. + + The `gcs` settings should be the same as the storage settings of the log backup to be compacted. `CompactBackup` reads log files from the corresponding location and compact them. + +#### View the status of log backup compaction + +After creating the `CompactBackup` CR, TiDB Operator automatically starts compacting the log backup. You can check the backup status using the following command: + +```shell +kubectl get cpbk -n backup-test +``` + +From the output, you can find the status of the `CompactBackup` CR named `demo1-compact-backup`. An example output is as follows: + +``` +NAME STATUS PROGRESS MESSAGE +demo1-compact-backup Complete [READ_META(17/17),COMPACT_WORK(1291/1291)] +``` + +If the `STATUS` field displays `Complete`, the compact log backup process has finished successfully. + ### Backup CR examples
@@ -730,6 +788,89 @@ The steps to prepare for a scheduled snapshot backup are the same as those of [P log-integrated-backup-schedule-gcs log Running .... ``` +## Integrated management of scheduled snapshot backup, log backup, and compact log backup + +To accelerate downstream recovery, you can enable `CompactBackup` CR in the `BackupSchedule` CR. This feature periodically compacts log backup files in remote storage. You must enable log backup before using log backup compaction. This section extends the configuration from the previous section. + +### Prerequisites: Prepare for a scheduled snapshot backup + +The steps to prepare for a scheduled snapshot backup are the same as that of [Prepare for an ad-hoc backup](#prerequisites-prepare-for-an-ad-hoc-backup). + +### Create `BackupSchedule` + +1. Create a `BackupSchedule` CR named `integrated-backup-schedule-gcs` in the `backup-test` namespace. + + ```shell + kubectl apply -f integrated-backup-schedule-s3.yaml + ``` + + The content of `integrated-backup-schedule-gcs.yaml` is as follows: + + ```yaml + --- + apiVersion: pingcap.com/v1alpha1 + kind: BackupSchedule + metadata: + name: integrated-backup-schedule-gcs + namespace: backup-test + spec: + maxReservedTime: "3h" + schedule: "* */2 * * *" + backupTemplate: + backupType: full + cleanPolicy: Delete + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + gcs: + projectId: ${project_id} + secretName: gcs-secret + bucket: my-bucket + prefix: schedule-backup-folder-snapshot + logBackupTemplate: + backupMode: log + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + gcs: + projectId: ${project_id} + secretName: gcs-secret + bucket: my-bucket + prefix: schedule-backup-folder-log + compactBackupTemplate: + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + gcs: + projectId: ${project_id} + secretName: gcs-secret + bucket: my-bucket + prefix: schedule-backup-folder-log + ``` + + In the preceding example of `integrated-backup-schedule-gcs.yaml`, the `backupSchedule` configuration is based on the previous section, with the following additions for `compactBackup`: + + * Added the `BackupSchedule.spec.compactInterval` field to specify the time interval for log backup compaction. It is recommended not to exceed the interval of scheduled snapshot backups and to keep it between one-half to one-third of the scheduled snapshot backup interval. + + * Added the `BackupSchedule.spec.compactBackupTemplate` field. Ensure that the `BackupSchedule.spec.compactBackupTemplate.gcs` configuration matches the `BackupSchedule.spec.logBackupTemplate.gcs` configuration. + + For the field description of `backupSchedule`, refer to [BackupSchedule CR fields](backup-restore-cr.md#backupschedule-cr-fields). + +2. After creating `backupSchedule`, use the following command to check the backup status: + + ```shell + kubectl get bks -n backup-test -o wide + ``` + + A compact log backup task is created together with `backupSchedule`. You can check the `CompactBackup` CR using the following command: + + ```shell + kubectl get cpbk -n backup-test + ``` + ## Delete the backup CR If you no longer need the backup CR, refer to [Delete the Backup CR](backup-restore-overview.md#delete-the-backup-cr). diff --git a/zh/backup-restore-cr.md b/zh/backup-restore-cr.md index f8ef6b0888..c18e92b70b 100644 --- a/zh/backup-restore-cr.md +++ b/zh/backup-restore-cr.md @@ -247,6 +247,33 @@ summary: 介绍用于备份与恢复的 Custom Resource (CR) 资源的各字段 * `.spec.local.volume`:持久卷配置。 * `.spec.local.volumeMount`:持久卷挂载配置。 +## CompactBackup CR 字段介绍 + +对于 TiDB v9.0.0 及以上版本的集群,你可以使用 `CompactBackup` 加速日志恢复。要将日志备份数据压缩为结构化 SST 文件,你可以通过创建一个自定义的 `CompactBackup` CR 对象来描述一次备份任务。以下是 `CompactBackup` CR 各个字段的具体含义: + +* `.spec.startTs`:指定日志压缩备份的起始时间戳。 +* `.spec.endTs`:指定日志压缩备份的结束时间戳。 +* `.spec.concurrency`:指定同时进行的压缩日志任务的最大数量,默认值为 `4`。 +* `.spec.maxRetryTimes`:指定压缩任务失败的最大重试次数,默认值为 `6`。 +* `.spec.toolImage`:指定 `CompactBackup` 使用的工具镜像。在 `CompactBackup` 中,唯一使用的工具镜像为 BR。使用 BR 备份时,你可以使用该字段指定 BR 的版本: + + - 如果未指定或者为空,默认使用镜像 `pingcap/br:${tikv_version}` 进行备份。 + - 如果指定了 BR 的版本,例如 `.spec.toolImage: pingcap/br:v9.0.0`,那么使用指定的版本镜像进行备份。 + - 如果指定了镜像但未指定版本,例如 `.spec.toolImage: private/registry/br`,那么使用镜像 `private/registry/br:${tikv_version}` 进行备份。 + +* `.spec.env`:指定运行压缩备份任务的 Pod 的环境变量信息。 +* `.spec.affinity`:指定运行备份任务的 Pod 亲和性 (affinity) 配置。关于亲和性的详细说明,请参阅[亲和性与反亲和性](https://kubernetes.io/zh-cn/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)。 +* `.spec.tolerations`:指定运行压缩备份任务的 Pod 能够被调度到带有与之匹配的[污点 (Taint)](https://kubernetes.io/docs/reference/glossary/?all=true#term-taint) 的节点上。关于污点与容忍度的更多说明,请参阅[污点和容忍度](https://kubernetes.io/zh-cn/docs/concepts/scheduling-eviction/taint-and-toleration/)。 +* `.spec.podSecurityContext`:指定运行压缩备份任务的 Pod 的安全上下文配置,以支持非 root 用户运行 Pod。关于 `podSecurityContext` 的更多说明,请参阅[以非 root 用户运行容器](containers-run-as-non-root-user.md)。 +* `.spec.priorityClassName`:指定运行压缩备份任务的 Pod 的 `priorityClass` 的名称,用于设置运行优先级。关于 `priorityClass` 的更多说明,请参阅 [Pod 优先级和抢占](https://kubernetes.io/zh-cn/docs/concepts/scheduling-eviction/pod-priority-preemption/)。 +* `.spec.imagePullSecrets`:指定运行压缩备份任务的 Pod 使用的 [`imagePullSecrets`](https://kubernetes.io/zh-cn/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod)。 +* `.spec.serviceAccount`:指定恢复时使用的 ServiceAccount 名称。 +* `.spec.useKMS`:指定恢复时是否使用 AWS-KMS 解密备份使用的 S3 存储密钥。 +* `.spec.br`:BR 相关配置,详情请参阅 [BR 字段介绍](#br-字段介绍)。 +* `.spec.s3`:S3 兼容存储相关配置,详情请参阅 [S3 字段介绍](#s3-存储字段介绍)。 +* `.spec.gcs`:GCS 存储相关配置,详情请参阅 [GCS 字段介绍](#gcs-存储字段介绍)。 +* `.spec.azblob`:Azure Blob Storage 存储相关配置,详情请参阅 [Azure Blob Storage 字段介绍](#azure-blob-storage-存储字段介绍)。 + ## Restore CR 字段介绍 为了对 Kubernetes 上的 TiDB 集群进行数据恢复,用户可以通过创建一个自定义的 `Restore` CR 对象来描述一次恢复,具体恢复过程可参考[备份与恢复简介](backup-restore-overview.md#数据恢复)中列出的文档。以下介绍 Restore CR 各个字段的具体含义。 @@ -334,6 +361,7 @@ summary: 介绍用于备份与恢复的 Custom Resource (CR) 资源的各字段 + `backupTemplate`:快照备份相关配置。指定快照备份集群及远程存储相关的配置,字段和 Backup CR 中的 `spec` 一样,详细介绍可参考 [Backup CR 字段介绍](#backup-cr-字段介绍)。 + `logBackupTemplate`:日志备份相关配置。指定日志备份集群及远程存储相关的配置,字段和 Backup CR 中的 `spec` 一样,详细介绍可参考 [Backup CR 字段介绍](#backup-cr-字段介绍),日志备份随 `backupSchedule` 创建、删除,且根据 `.spec.maxReservedTime` 进行回收。日志备份名称在 `status.logBackup` 中保存。 ++ `compactBackupTemplate`:压缩日志备份的配置模板,字段和 CompactBackup CR 中的 `spec` 一样,详细介绍可参考 [CompactBackup CR 字段介绍](#compactbackup-cr-字段介绍)。压缩日志备份会随 `backupSchedule` 创建和删除,日志备份名称存储在 `status.logBackup` 中。压缩日志备份的存储设置应与同一 `backupSchedule` 中的 `logBackupTemplate` 保持一致。 > **注意:** > @@ -343,4 +371,5 @@ summary: 介绍用于备份与恢复的 Custom Resource (CR) 资源的各字段 + `.spec.maxBackups`:一种备份保留策略,决定定时备份最多可保留的备份个数。超过该数目,就会将过时的备份删除。如果将该项设置为 `0`,则表示保留所有备份。 + `.spec.maxReservedTime`:一种备份保留策略,按时间保留备份。例如将该参数设置为 `24h`,表示只保留最近 24 小时内的备份条目。超过这个时间的备份都会被清除。时间设置格式参考 [`func ParseDuration`](https://golang.org/pkg/time/#ParseDuration)。如果同时设置 `.spec.maxBackups` 和 `.spec.maxReservedTime`,则以 `.spec.maxReservedTime` 为准。 + `.spec.schedule`:Cron 的时间调度格式。具体格式可参考 [Cron](https://en.wikipedia.org/wiki/Cron)。 - + `.spec.pause`:是否暂停定时备份,默认为 `false`。如果将该值设置为 `true`,表示暂停定时备份,此时即使到了指定时间点,也不会进行备份。在定时备份暂停期间,备份 Garbage Collection (GC) 仍然正常进行。如需重新开启定时快照备份,将 `true` 改为 `false`。由于目前日志备份暂不支持暂停,因此该配置对日志备份无效。 + + `.spec.compactInterval`:用于触发新压缩任务的时间间隔。 + + `.spec.pause`:是否暂停定时备份,默认为 `false`。如果将该值设置为 `true`,表示暂停定时备份,此时即使到了指定时间点,也不会进行备份。在定时备份暂停期间,备份 Garbage Collection (GC) 仍然正常进行。如需重新开启定时快照备份,将 `true` 改为`false`。由于目前日志备份暂不支持暂停,因此该配置对日志备份无效。 diff --git a/zh/backup-to-aws-s3-using-br.md b/zh/backup-to-aws-s3-using-br.md index d18a715e88..9480bea22b 100644 --- a/zh/backup-to-aws-s3-using-br.md +++ b/zh/backup-to-aws-s3-using-br.md @@ -487,8 +487,7 @@ demo1-log-backup-s3 log Stopped .... backupMode: log logSubcommand: log-start/log-pause/log-stop br: - cluster: demo1 - clusterNamespace: test1 + mespace: test1 sendCredToTikv: true s3: provider: aws @@ -534,6 +533,65 @@ demo1-log-backup-s3 log Stopped .... demo1-log-backup-s3 log Stopped ... 2022-10-10T15:21:00+08:00 ``` +### 压缩日志备份 + +对于 TiDB v9.0.0 及以上版本的集群,你可以使用 `CompactBackup` CR 将日志备份数据压缩为 SST 格式,以加速下游的日志恢复 (Point-in-time recovery, PITR)。 + +本节基于前文的日志备份示例,介绍如何使用压缩日志备份。 + +1. 在 `backup-test` namespace 中创建一个名为 `demo1-compact-backup` 的 CompactBackup CR。 + + ```shell + kubectl apply -f compact-backup-demo1.yaml + ``` + + `compact-backup-demo1.yaml` 的内容如下: + + ```yaml + --- + apiVersion: pingcap.com/v1alpha1 + kind: CompactBackup + metadata: + name: demo1-compact-backup + namespace: backup-test + spec: + startTs: "***" + endTs: "***" + concurrency: 8 + maxRetryTimes: 2 + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + s3: + provider: aws + secretName: s3-secret + region: us-west-1 + bucket: my-bucket + prefix: my-log-backup-folder + ``` + + 其中,`startTs` 和 `endTs` 指定 `demo1-compact-backup` 需要压缩的日志备份时间范围。任何包含至少一个该时间区间内写入的日志都会被送去压缩。因此,最终的压缩结果可能包含该时间范围之外的写入数据。 + + `s3` 设置应与需要压缩的日志备份的存储设置相同,`CompactBackup` 会读取相应地址的日志文件并进行压缩。 + +#### 查看压缩日志备份状态 + +创建 `CompactBackup` CR 后,TiDB Operator 会自动开始压缩日志备份。你可以运行以下命令查看备份状态: + +```shell +kubectl get cpbk -n backup-test +``` + +从上述命令的输出中,你可以找到描述名为 `demo1-compact-backup` 的 `CompactBackup` CR 的信息,输出示例如下: + +``` +NAME STATUS PROGRESS MESSAGE +demo1-compact-backup Complete [READ_META(17/17),COMPACT_WORK(1291/1291)] +``` + +如果 `STATUS` 字段显示为 `Complete` 则代表压缩日志备份已经完成。 + ### 备份示例
@@ -913,6 +971,93 @@ kubectl get bk -l tidb.pingcap.com/backup-schedule=demo1-backup-schedule-s3 -n t log-integrated-backup-schedule-s3 log Running .... ``` +## 集成定时快照备份、日志备份和压缩日志备份 + +为了加快下游恢复速度,可以在 `BackupSchedule` CR 中添加压缩日志备份。压缩日志备份可以定期压缩远程存储中的日志备份文件。你必须先启用日志备份,才能使用压缩日志备份。本节基于上一节内容进行扩展。 + +### 前置条件:准备定时快照备份环境 + +同[准备 Ad-hoc 备份环境](#前置条件准备-ad-hoc-备份环境)。 + +### 创建 `BackupSchedule` + +1. 在 `backup-test` 这个 namespace 中创建一个名为 `integrated-backup-schedule-s3` 的 `BackupSchedule` CR。 + + ```shell + kubectl apply -f integrated-backup-schedule-s3.yaml + ``` + + `integrated-backup-schedule-s3` 文件内容如下: + + ```yaml + --- + apiVersion: pingcap.com/v1alpha1 + kind: BackupSchedule + metadata: + name: integrated-backup-schedule-s3 + namespace: backup-test + spec: + maxReservedTime: "3h" + schedule: "* */2 * * *" + compactInterval: "1h" + backupTemplate: + backupType: full + cleanPolicy: Delete + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + s3: + provider: aws + secretName: s3-secret + region: us-west-1 + bucket: my-bucket + prefix: my-folder-snapshot + logBackupTemplate: + backupMode: log + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + s3: + provider: aws + secretName: s3-secret + region: us-west-1 + bucket: my-bucket + prefix: my-folder-log + compactBackupTemplate: + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + s3: + provider: aws + secretName: s3-secret + region: us-west-1 + bucket: my-bucket + prefix: my-folder-log + ``` + + 以上 `integrated-backup-schedule-s3.yaml` 文件配置示例中,`backupSchedule` 配置基于上一节内容,新增了 `compactBackup` 相关设置,主要改动如下: + + - 新增 `BackupSchedule.spec.compactInterval` 字段,用于指定日志压缩备份的时间间隔。建议不要超过定时快照备份的间隔,并控制在定时快照备份间隔的二分之一至三分之一之间。 + + - 新增 `BackupSchedule.spec.compactBackupTemplate` 字段。请确保 `BackupSchedule.spec.compactBackupTemplate.s3` 配置与 `BackupSchedule.spec.logBackupTemplate.s3` 保持一致。 + + 关于 `backupSchedule` 配置项具体介绍,请参考 [BackupSchedule CR 字段介绍](backup-restore-cr.md#backupschedule-cr-字段介绍)。 + +2. `backupSchedule` 创建完成后,可以通过以下命令查看定时快照备份的状态: + + ```shell + kubectl get bks -n backup-test -o wide + ``` + + 压缩日志备份会随着 `backupSchedule` 创建,可以通过如下命令查看 `CompactBackup` CR 的信息。 + + ```shell + kubectl get cpbk -n backup-test + ``` + ## 删除备份的 Backup CR 如果你不再需要已备份的 Backup CR,请参考[删除备份的 Backup CR](backup-restore-overview.md#删除备份的-backup-cr)。 diff --git a/zh/backup-to-azblob-using-br.md b/zh/backup-to-azblob-using-br.md index 327ef84ed2..e863762163 100644 --- a/zh/backup-to-azblob-using-br.md +++ b/zh/backup-to-azblob-using-br.md @@ -449,6 +449,63 @@ demo1-log-backup-azblob log Stopped .... demo1-log-backup log Complete ... 2022-10-10T15:21:00+08:00 ``` +### 压缩日志备份 + +对于 TiDB v9.0.0 及以上版本的集群,你可以使用 `CompactBackup` CR 将日志备份数据压缩为 SST 格式,以加速下游的日志恢复 (Point-in-time recovery, PITR)。 + +本节基于前文的日志备份示例,介绍如何使用压缩日志备份。 + +1. 在 `backup-test` namespace 中创建一个名为 `demo1-compact-backup` 的 CompactBackup CR。 + + ```shell + kubectl apply -f compact-backup-demo1.yaml + ``` + + `compact-backup-demo1.yaml` 的内容如下: + + ```yaml + --- + apiVersion: pingcap.com/v1alpha1 + kind: CompactBackup + metadata: + name: demo1-compact-backup + namespace: backup-test + spec: + startTs: "***" + endTs: "***" + concurrency: 8 + maxRetryTimes: 2 + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + azblob: + secretName: azblob-secret + container: my-container + prefix: my-log-backup-folder + ``` + + 其中,`startTs` 和 `endTs` 指定 `demo1-compact-backup` 需要压缩的日志备份时间范围。任何包含至少一个该时间区间内写入的日志都会被送去压缩。因此,最终的压缩结果可能包含该时间范围之外的写入数据。 + + `azblob` 设置应与需要压缩的日志备份的存储设置相同,`CompactBackup` 会读取相应地址的日志文件并进行压缩。 + +#### 查看压缩日志备份状态 + +创建 `CompactBackup` CR 后,TiDB Operator 会自动开始压缩日志备份。你可以运行以下命令查看备份状态: + +```shell +kubectl get cpbk -n backup-test +``` + +从上述命令的输出中,你可以找到描述名为 `demo1-compact-backup` 的 `CompactBackup` CR 的信息,输出示例如下: + +``` +NAME STATUS PROGRESS MESSAGE +demo1-compact-backup Complete [READ_META(17/17),COMPACT_WORK(1291/1291)] +``` + +如果 `STATUS` 字段显示为 `Complete` 则代表压缩日志备份已经完成。 + ### 备份示例
@@ -767,6 +824,80 @@ kubectl get backup -l tidb.pingcap.com/backup-schedule=demo1-backup-schedule-azb log-integrated-backup-schedule-azblob log Running .... ``` +## 集成定时快照备份、日志备份和压缩日志备份 + +为了加快下游恢复速度,可以在 `BackupSchedule` CR 中添加压缩日志备份。压缩日志备份可以定期压缩远程存储中的日志备份文件。你必须先启用日志备份,才能使用压缩日志备份。本节基于上一节内容进行扩展。 + +### 前置条件:准备定时快照备份环境 + +同[准备 Ad-hoc 备份环境](#前置条件准备-ad-hoc-备份环境)。 + +### 创建 `BackupSchedule` + +1. 在 `backup-test` 这个 namespace 中创建一个名为 `integrated-backup-schedule-azblob` 的 `BackupSchedule` CR。 + + ```shell + kubectl apply -f integrated-backup-scheduler-azblob.yaml + ``` + + `integrated-backup-scheduler-azblob` 文件内容如下: + + ```yaml + --- + apiVersion: pingcap.com/v1alpha1 + kind: BackupSchedule + metadata: + name: integrated-backup-schedule-azblob + namespace: backup-test + spec: + maxReservedTime: "3h" + schedule: "* */2 * * *" + compactInterval: "1h" + backupTemplate: + backupType: full + cleanPolicy: Delete + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + azblob: + secretName: azblob-secret + container: my-container + prefix: schedule-backup-folder-snapshot + #accessTier: Hot + logBackupTemplate: + backupMode: log + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + azblob: + secretName: azblob-secret + container: my-container + prefix: schedule-backup-folder-log + #accessTier: Hot + ``` + + 以上 `integrated-backup-scheduler-azblob.yaml` 文件配置示例中,`backupSchedule` 配置基于上一节内容,新增了 `compactBackup` 相关设置,主要改动如下: + + - 新增 `BackupSchedule.spec.compactInterval` 字段,用于指定日志压缩备份的时间间隔。建议不要超过定时快照备份的间隔,并控制在定时快照备份间隔的二分之一至三分之一之间。 + + - 新增 `BackupSchedule.spec.compactBackupTemplate` 字段。请确保 `BackupSchedule.spec.compactBackupTemplate.azblob` 配置与 `BackupSchedule.spec.logBackupTemplate.azblob` 保持一致。 + + 关于 `backupSchedule` 配置项具体介绍,请参考 [BackupSchedule CR 字段介绍](backup-restore-cr.md#backupschedule-cr-字段介绍)。 + +2. `backupSchedule` 创建完成后,可以通过以下命令查看定时快照备份的状态: + + ```shell + kubectl get bks -n backup-test -o wide + ``` + + 压缩日志备份会随着 `backupSchedule` 创建,可以通过如下命令查看 `CompactBackup` CR 的信息。 + + ```shell + kubectl get cpbk -n backup-test + ``` + ## 删除备份的 Backup CR 如果你不再需要已备份的 Backup CR,请参考[删除备份的 Backup CR](backup-restore-overview.md#删除备份的-backup-cr)。 diff --git a/zh/backup-to-gcs-using-br.md b/zh/backup-to-gcs-using-br.md index dfa8764992..5b94a81c31 100644 --- a/zh/backup-to-gcs-using-br.md +++ b/zh/backup-to-gcs-using-br.md @@ -449,6 +449,64 @@ demo1-log-backup-gcs log Stopped .... demo1-log-backup-gcs log Stopped ... 2022-10-10T15:21:00+08:00 ``` +### 压缩日志备份 + +对于 TiDB v9.0.0 及以上版本的集群,你可以使用 `CompactBackup` CR 将日志备份数据压缩为 SST 格式,以加速下游的日志恢复 (Point-in-time recovery, PITR)。 + +本节基于前文的日志备份示例,介绍如何使用压缩日志备份。 + +1. 在 `backup-test` namespace 中创建一个名为 `demo1-compact-backup` 的 CompactBackup CR。 + + ```shell + kubectl apply -f compact-backup-demo1.yaml + ``` + + `compact-backup-demo1.yaml` 的内容如下: + + ```yaml + --- + apiVersion: pingcap.com/v1alpha1 + kind: CompactBackup + metadata: + name: demo1-compact-backup + namespace: backup-test + spec: + startTs: "***" + endTs: "***" + concurrency: 8 + maxRetryTimes: 2 + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + gcs: + projectId: ${project_id} + secretName: gcs-secret + bucket: my-bucket + prefix: my-log-backup-folder + ``` + + 其中,`startTs` 和 `endTs` 指定 `demo1-compact-backup` 需要压缩的日志备份时间范围。任何包含至少一个该时间区间内写入的日志都会被送去压缩。因此,最终的压缩结果可能包含该时间范围之外的写入数据。 + + `gcs` 设置应与需要压缩的日志备份的存储设置相同,`CompactBackup` 会读取相应地址的日志文件并进行压缩。 + +#### 查看压缩日志备份状态 + +创建 `CompactBackup` CR 后,TiDB Operator 会自动开始压缩日志备份。你可以运行以下命令查看备份状态: + +```shell +kubectl get cpbk -n backup-test +``` + +从上述命令的输出中,你可以找到描述名为 `demo1-compact-backup` 的 `CompactBackup` CR 的信息,输出示例如下: + +``` +NAME STATUS PROGRESS MESSAGE +demo1-compact-backup Complete [READ_META(17/17),COMPACT_WORK(1291/1291)] +``` + +如果 `STATUS` 字段显示为 `Complete` 则代表压缩日志备份已经完成。 + ### 备份示例
@@ -736,6 +794,90 @@ spec: log-integrated-backup-schedule-gcs log Running .... ``` +## 集成定时快照备份、日志备份和压缩日志备份 + +为了加快下游恢复速度,可以在 `BackupSchedule` CR 中添加压缩日志备份。压缩日志备份可以定期压缩远程存储中的日志备份文件。你必须先启用日志备份,才能使用压缩日志备份。本节基于上一节内容进行扩展。 + +### 前置条件:准备定时快照备份环境 + +同[准备 Ad-hoc 备份环境](#前置条件准备-ad-hoc-备份环境)。 + +### 创建 `BackupSchedule` + +1. 在 `backup-test` 这个 namespace 中创建一个名为 `integrated-backup-schedule-gcs` 的 `BackupSchedule` CR。 + + ```shell + kubectl apply -f integrated-backup-scheduler-gcs.yaml + ``` + + `integrated-backup-scheduler-gcs` 文件内容如下: + + ```yaml + --- + apiVersion: pingcap.com/v1alpha1 + kind: BackupSchedule + metadata: + name: integrated-backup-schedule-gcs + namespace: backup-test + spec: + maxReservedTime: "3h" + schedule: "* */2 * * *" + compactInterval: "1h" + backupTemplate: + backupType: full + cleanPolicy: Delete + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + gcs: + projectId: ${project_id} + secretName: gcs-secret + bucket: my-bucket + prefix: schedule-backup-folder-snapshot + logBackupTemplate: + backupMode: log + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + gcs: + projectId: ${project_id} + secretName: gcs-secret + bucket: my-bucket + prefix: schedule-backup-folder-log + compactBackupTemplate: + br: + cluster: demo1 + clusterNamespace: test1 + sendCredToTikv: true + gcs: + projectId: ${project_id} + secretName: gcs-secret + bucket: my-bucket + prefix: schedule-backup-folder-log + ``` + + 以上 `integrated-backup-schedule-gcs.yaml` 文件配置示例中,`backupSchedule` 配置基于上一节内容,新增了 `compactBackup` 相关设置,主要改动如下: + + - 新增 `BackupSchedule.spec.compactInterval` 字段,用于指定日志压缩备份的时间间隔。建议不要超过定时快照备份的间隔,并控制在定时快照备份间隔的二分之一至三分之一之间。 + + - 新增 `BackupSchedule.spec.compactBackupTemplate` 字段。请确保 `BackupSchedule.spec.compactBackupTemplate.gcs` 配置与 `BackupSchedule.spec.logBackupTemplate.gcs` 保持一致。 + + 关于 `backupSchedule` 配置项具体介绍,请参考 [BackupSchedule CR 字段介绍](backup-restore-cr.md#backupschedule-cr-字段介绍)。 + +2. `backupSchedule` 创建完成后,可以通过以下命令查看定时快照备份的状态: + + ```shell + kubectl get bks -n backup-test -o wide + ``` + + 压缩日志备份会随着 `backupSchedule` 创建,可以通过如下命令查看 `CompactBackup` CR 的信息。 + + ```shell + kubectl get cpbk -n backup-test + ``` + ## 删除备份的 Backup CR 如果你不再需要已备份的 Backup CR,请参考[删除备份的 Backup CR](backup-restore-overview.md#删除备份的-backup-cr)。