br compact backup user document

en/backup-restore-cr.md

@@ -258,6 +258,33 @@ This section introduces the fields in the `Backup` CR.
 * `.spec.local.volume`: the persistent volume configuration.
 * `.spec.local.volumeMount`: the persistent volume mount configuration.
 
+## CompactBackup CR fields
+
+For TiDB v9.0.0 and later versions, you can use `CompactBackup` to accelerate PITR (Point-in-time recovery). To compact log backup data into structured SST files, you can create a custom `CompactBackup` CR object to define a backup task. The following introduces the fields in the `CompactBackup` CR:
+
+* `.spec.startTs`: the start timestamp for log compaction backup.
+* `.spec.endTs`: the end timestamp for log compaction backup.
+* `.spec.concurrency`: the maximum number of concurrent log compaction tasks. The default value is `4`.
+* `.spec.maxRetryTimes`: the maximum number of retries for failed compaction tasks. The default value is `6`.
+* `.spec.toolImage`：the tool image used by `CompactBackup`. BR is the only tool image used in `CompactBackup`. When using BR for backup, you can specify the BR version with this field:
+
+    - If not specified or left empty, the `pingcap/br:${tikv_version}` image is used for backup by default.
+    - If a BR version is specified, such as `.spec.toolImage: pingcap/br:v9.0.0`, the image of the specified version is used for backup.
+    - If an image is specified without a version, such as `.spec.toolImage: private/registry/br`, the `private/registry/br:${tikv_version}` image is used for backup.
+
+* `.spec.env`: the environment variables for the Pod that runs the compaction task.
+* `.spec.affinity`: the affinity configuration for the Pod that runs the compaction task. For details on affinity, refer to [Affinity and anti-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity).
+* `.spec.tolerations`: specifies that the Pod that runs the compaction task can schedule onto nodes with matching [taints](https://kubernetes.io/docs/reference/glossary/?all=true#term-taint). For details on taints and tolerations, refer to [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/).
+* `.spec.podSecurityContext`: the security context configuration for the Pod that runs the compaction task, which allows the Pod to run as a non-root user. For details on `podSecurityContext`, refer to [Run Containers as a Non-root User](containers-run-as-non-root-user.md).
+* `.spec.priorityClassName`: the name of the priority class for the Pod that runs the compaction task, which sets priority for the Pod. For details on priority classes, refer to [Pod Priority and Preemption](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/).
+* `.spec.imagePullSecrets`: the [imagePullSecrets](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod) for the Pod that runs the compaction task.
+* `.spec.serviceAccount`: the name of the ServiceAccount used for compact.
+* `.spec.useKMS`: whether to use AWS-KMS to decrypt the S3 storage key used for the backup.
+* `.spec.br`: BR-related configuration. For more information, refer to [BR fields](#br-fields).
+* `.spec.s3`: S3-related configuration. For more information, refer to [S3 storage fields](#s3-storage-fields).
+* `.spec.gcs`: GCS-related configuration. For more information, refer to [GCS fields](#gcs-fields).
+* `.spec.azblob`：Azure Blob Storage-related configuration. For more information, refer to [Azure Blob Storage fields](#azure-blob-storage-fields).
+
 ## Restore CR fields
 
 To restore data to a TiDB cluster on Kubernetes, you can create a `Restore` CR object. For detailed restore process, refer to documents listed in [Restore data](backup-restore-overview.md#restore-data).
@@ -352,6 +379,7 @@ The `backupSchedule` configuration consists of three parts: the configuration of
 
 * `backupTemplate`: the configuration of the snapshot backup. Specifies the configuration related to the cluster and remote storage of the snapshot backup, which is the same as the `spec` configuration of [the `Backup` CR](#backup-cr-fields).
 * `logBackupTemplate`: the configuration of the log backup. Specifies the configuration related to the cluster and remote storage of the log backup, which is the same as the `spec` configuration of [the `Backup` CR](#backup-cr-fields). The log backup is created and deleted along with `backupSchedule` and recycled according to `.spec.maxReservedTime`. The log backup name is saved in `status.logBackup`.
+* `compactBackupTemplate`: the configuration template of the log compaction backup. The fields are the same as those in the `spec` configuration of [the `CompactBackup` CR](#compactbackup-cr-fields). The compaction backup is created and deleted along with `backupSchedule`. The log backup names are stored in `status.logBackup`. The storage settings of the compaction backup should be the same as that of `logBackupTemplate` in the same `backupSchedule`.
 
     > **Note:**
     >
@@ -362,4 +390,5 @@ The `backupSchedule` configuration consists of three parts: the configuration of
     * `.spec.maxBackups`: a backup retention policy, which determines the maximum number of backup files to be retained. When the number of backup files exceeds this value, the outdated backup file will be deleted. If you set this field to `0`, all backup items are retained.
     * `.spec.maxReservedTime`: a backup retention policy based on time. For example, if you set the value of this field to `24h`, only backup files within the recent 24 hours are retained. All backup files older than this value are deleted. For the time format, refer to [`func ParseDuration`](https://golang.org/pkg/time/#ParseDuration). If you have set `.spec.maxBackups` and `.spec.maxReservedTime` at the same time, the latter takes effect.
     * `.spec.schedule`: the time scheduling format of Cron. Refer to [Cron](https://en.wikipedia.org/wiki/Cron) for details.
+    * `.spec.compactInterval`: the time interval used to trigger a new compaction task.
     * `.spec.pause`: `false` by default. If this field is set to `true`, the scheduled scheduling is paused. In this situation, the backup operation will not be performed even if the scheduling time point is reached. During this pause, the backup garbage collection runs normally. If you change `true` to `false`, the scheduled snapshot backup process is restarted. Because currently, log backup does not support pause, this configuration does not take effect for log backup.

en/backup-to-aws-s3-using-br.md

@@ -491,8 +491,7 @@ In TiDB Operator v1.5.4, v1.6.0, and earlier versions, you can use the `logStop:
       backupMode: log
       logSubcommand: log-start/log-pause/log-stop
       br:
-        cluster: demo1
-        clusterNamespace: test1
+        mespace: test1
         sendCredToTikv: true
       s3:
         provider: aws
@@ -538,6 +537,65 @@ In TiDB Operator v1.5.4, v1.6.0, and earlier versions, you can use the `logStop:
     demo1-log-backup-s3    log        Stopped    ...   2022-10-10T15:21:00+08:00
     ```
 
+### Compact log backup
+
+For TiDB v9.0.0 and later versions, you can use a `CompactBackup` CR to compact log backup data into SST format, accelerating downstream PITR (Point-in-time recovery).
+
+This section explains how to compact log backup based on the log backup example from previous sections.
+
+1. In the `backup-test` namespace, create a `CompactBackup` CR named `demo1-compact-backup`.
+
+    ```shell
+    kubectl apply -f compact-backup-demo1.yaml
+    ```
+
+    The content of `compact-backup-demo1.yaml` is as follows:
+
+    ```yaml
+    ---
+    apiVersion: pingcap.com/v1alpha1
+    kind: CompactBackup
+    metadata:
+      name: demo1-compact-backup
+      namespace: backup-test
+    spec:
+      startTs: "***"
+      endTs: "***"
+      concurrency: 8
+      maxRetryTimes: 2
+      br:
+        cluster: demo1
+        clusterNamespace: test1
+        sendCredToTikv: true
+      s3:
+        provider: aws
+        secretName: s3-secret
+        region: us-west-1
+        bucket: my-bucket
+        prefix: my-log-backup-folder
+    ```
+
+    The `startTs` and `endTs` fields specify the time range for the logs to be compacted by `demo1-compact-backup`. Any log that contains at least one write within this time range will be included in the compaction process. As a result, the final compacted data might include data written outside this range.
+
+    The `s3` settings should be the same as the storage settings of the log backup to be compacted. `CompactBackup` reads log files from the corresponding location and compact them.
+
+#### View the status of log backup compaction
+
+After creating the `CompactBackup` CR, TiDB Operator automatically starts compacting the log backup. You can check the backup status using the following command:
+
+```shell
+kubectl get cpbk -n backup-test
+```
+
+From the output, you can find the status of the `CompactBackup` CR named `demo1-compact-backup`. An example output is as follows:
+
+```
+NAME                   STATUS                   PROGRESS                                     MESSAGE
+demo1-compact-backup   Complete   [READ_META(17/17),COMPACT_WORK(1291/1291)]   
+```
+
+If the `STATUS` field displays `Complete`, the compact log backup process has finished successfully.
+
 ### Backup CR examples
 
 <details>
@@ -917,6 +975,93 @@ The steps to prepare for a scheduled snapshot backup are the same as those of [P
     log-integrated-backup-schedule-s3                      log        Running   ....
     ```
 
+## Integrated management of scheduled snapshot backup, log backup, and compact log backup
+
+To accelerate downstream recovery, you can enable `CompactBackup` CR in the `BackupSchedule` CR. This feature periodically compacts log backup files in remote storage. You must enable log backup before using log backup compaction. This section extends the configuration from the previous section.
+
+### Prerequisites: Prepare for a scheduled snapshot backup
+
+The steps to prepare for a scheduled snapshot backup are the same as that of [Prepare for an ad-hoc backup](#prerequisites-prepare-for-an-ad-hoc-backup).
+
+### Create `BackupSchedule`
+
+1. Create a `BackupSchedule` CR named `integrated-backup-schedule-s3` in the `backup-test` namespace.
+
+    ```shell
+    kubectl apply -f integrated-backup-schedule-s3.yaml
+    ```
+
+    The content of `integrated-backup-schedule-s3.yaml` is as follows:
+
+    ```yaml
+    ---
+    apiVersion: pingcap.com/v1alpha1
+    kind: BackupSchedule
+    metadata:
+      name: integrated-backup-schedule-s3
+      namespace: backup-test
+    spec:
+      maxReservedTime: "3h"
+      schedule: "* */2 * * *"
+      compactInterval: "1h"
+      backupTemplate:
+        backupType: full
+        cleanPolicy: Delete
+        br:
+          cluster: demo1
+          clusterNamespace: test1
+          sendCredToTikv: true
+        s3:
+          provider: aws
+          secretName: s3-secret
+          region: us-west-1
+          bucket: my-bucket
+          prefix: my-folder-snapshot
+      logBackupTemplate:
+        backupMode: log
+        br:
+          cluster: demo1
+          clusterNamespace: test1
+          sendCredToTikv: true
+        s3:
+          provider: aws
+          secretName: s3-secret
+          region: us-west-1
+          bucket: my-bucket
+          prefix: my-folder-log
+      compactBackupTemplate:
+        br:
+          cluster: demo1
+          clusterNamespace: test1
+          sendCredToTikv: true
+        s3:
+          provider: aws
+          secretName: s3-secret
+          region: us-west-1
+          bucket: my-bucket
+          prefix: my-folder-log
+    ```
+
+    In the preceding example of `integrated-backup-schedule-s3.yaml`, the `backupSchedule` configuration is based on the previous section, with the following additions for `compactBackup`:
+
+    * Added the `BackupSchedule.spec.compactInterval` field to specify the interval for log backup compaction. It is recommended not to exceed the interval of scheduled snapshot backups and to keep it between one-half to one-third of the scheduled snapshot backup interval.
+
+    * Added the `BackupSchedule.spec.compactBackupTemplate` field. Ensure that the `BackupSchedule.spec.compactBackupTemplate.s3` configuration matches the `BackupSchedule.spec.logBackupTemplate.s3` configuration.
+
+    For the field description of `backupSchedule`, refer to [BackupSchedule CR fields](backup-restore-cr.md#backupschedule-cr-fields).
+
+2. After creating `backupSchedule`, use the following command to check the backup status:
+
+    ```shell
+    kubectl get bks -n backup-test -o wide
+    ```
+
+    A compact log backup task is created together with `backupSchedule`. You can check the `CompactBackup` CR using the following command:
+
+    ```shell
+    kubectl get cpbk -n backup-test
+    ```
+
 ## Delete the backup CR
 
 If you no longer need the backup CR, refer to [Delete the Backup CR](backup-restore-overview.md#delete-the-backup-cr).

en/backup-to-azblob-using-br.md

@@ -449,6 +449,63 @@ In TiDB Operator v1.5.4, v1.6.0, and earlier versions, you can use the `logStop:
     demo1-log-backup    log        Complete   ...   2022-10-10T15:21:00+08:00
     ```
 
+### Compact log backup
+
+For TiDB v9.0.0 and later versions, you can use a `CompactBackup` CR to compact log backup data into SST format, accelerating downstream PITR (Point-in-time recovery).
+
+This section explains how to compact log backup based on the log backup example from previous sections.
+
+1. In the `backup-test` namespace, create a `CompactBackup` CR named `demo1-compact-backup`.
+
+    ```shell
+    kubectl apply -f compact-backup-demo1.yaml
+    ```
+
+    The content of `compact-backup-demo1.yaml` is as follows:
+
+    ```yaml
+    ---
+    apiVersion: pingcap.com/v1alpha1
+    kind: CompactBackup
+    metadata:
+      name: demo1-compact-backup
+      namespace: backup-test
+    spec:
+      startTs: "***"
+      endTs: "***"
+      concurrency: 8
+      maxRetryTimes: 2
+      br:
+        cluster: demo1
+        clusterNamespace: test1
+        sendCredToTikv: true
+      azblob:
+        secretName: azblob-secret
+        container: my-container
+        prefix: my-log-backup-folder
+    ```
+
+    The `startTs` and `endTs` fields specify the time range for the logs to be compacted by `demo1-compact-backup`. Any log that contains at least one write within this time range will be included in the compaction process. As a result, the final compacted data might include data written outside this range.
+
+    The `azblob` settings should be the same as the storage settings of the log backup to be compacted. `CompactBackup` reads log files from the corresponding location and compact them.
+
+#### View the status of log backup compaction
+
+After creating the `CompactBackup` CR, TiDB Operator automatically starts compacting the log backup. You can check the backup status using the following command:
+
+```shell
+kubectl get cpbk -n backup-test
+```
+
+From the output, you can find the status of the `CompactBackup` CR named `demo1-compact-backup`. An example output is as follows:
+
+```
+NAME                   STATUS                   PROGRESS                                     MESSAGE
+demo1-compact-backup   Complete   [READ_META(17/17),COMPACT_WORK(1291/1291)]   
+```
+
+If the `STATUS` field displays `Complete`, the compact log backup process has finished successfully.
+
 ### Backup CR examples
 
 <details>
@@ -768,6 +825,89 @@ The steps to prepare for a scheduled snapshot backup are the same as those of [P
     log-integrated-backup-schedule-azblob                      log        Running   ....
     ```
 
+## Integrated management of scheduled snapshot backup, log backup, and compact log backup
+
+To accelerate downstream recovery, you can enable `CompactBackup` CR in the `BackupSchedule` CR. This feature periodically compacts log backup files in remote storage. You must enable log backup before using log backup compaction. This section extends the configuration from the previous section.
+
+### Prerequisites: Prepare for a scheduled snapshot backup
+
+The steps to prepare for a scheduled snapshot backup are the same as that of [Prepare for an ad-hoc backup](#prerequisites-prepare-an-ad-hoc-backup-environment).
+
+### Create `BackupSchedule`
+
+1. Create a `BackupSchedule` CR named `integrated-backup-schedule-azblob` in the `backup-test` namespace.
+
+    ```shell
+    kubectl apply -f integrated-backup-schedule-azblob.yaml
+    ```
+
+    The content of `integrated-backup-schedule-azblob.yaml` is as follows:
+
+    ```yaml
+    ---
+    apiVersion: pingcap.com/v1alpha1
+    kind: BackupSchedule
+    metadata:
+      name: integrated-backup-schedule-azblob
+      namespace: backup-test
+    spec:
+      maxReservedTime: "3h"
+      schedule: "* */2 * * *"
+      backupTemplate:
+        backupType: full
+        cleanPolicy: Delete
+        br:
+          cluster: demo1
+          clusterNamespace: test1
+          sendCredToTikv: true
+        azblob:
+          secretName: azblob-secret
+          container: my-container
+          prefix: schedule-backup-folder-snapshot
+          #accessTier: Hot
+      logBackupTemplate:
+        backupMode: log
+        br:
+          cluster: demo1
+          clusterNamespace: test1
+          sendCredToTikv: true
+        azblob:
+          secretName: azblob-secret
+          container: my-container
+          prefix: schedule-backup-folder-log
+          #accessTier: Hot
+      compactBackupTemplate:
+        br:
+          cluster: demo1
+          clusterNamespace: test1
+          sendCredToTikv: true
+        azblob:
+          secretName: azblob-secret
+          container: my-container
+          prefix: schedule-backup-folder-log
+          #accessTier: Hot
+    ```
+
+    In the preceding example of `integrated-backup-schedule-azblob.yaml`, the `backupSchedule` configuration is based on the previous section, with the following additions for `compactBackup`:
+
+    * Added the `BackupSchedule.spec.compactInterval` field to specify the time interval for log backup compaction. It is recommended not to exceed the interval of scheduled snapshot backups and to keep it between one-half to one-third of the scheduled snapshot backup interval.
+
+    * Added the `BackupSchedule.spec.compactBackupTemplate` field. Ensure that the `BackupSchedule.spec.compactBackupTemplate.azblob` configuration matches the `BackupSchedule.spec.logBackupTemplate.azblob` configuration.
+
+    For the field description of `backupSchedule`, refer to [BackupSchedule CR fields](backup-restore-cr.md#backupschedule-cr-fields).
+
+2. After creating `backupSchedule`, use the following command to check the backup status:
+
+    ```shell
+    kubectl get bks -n backup-test -o wide
+    ```
+
+    A compact log backup task is created together with `backupSchedule`. You can check the `CompactBackup` CR using the following command:
+
+    ```shell
+    kubectl get cpbk -n backup-test
+    ```
+
 ## Delete the backup CR
 
 If you no longer need the backup CR, you can delete it by referring to [Delete the Backup CR](backup-restore-overview.md#delete-the-backup-cr).