Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 26 additions & 4 deletions br/backup-and-restore-storages.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,11 +139,33 @@ It is recommended that you configure access to S3 using either of the following
</div>
<div label="GCS" value="gcs">

You can configure the account used to access GCS by specifying the access key. If you specify the `credentials-file` parameter, the authentication is performed using the specified `credentials-file`. Besides specifying the key in the URI, the following methods are also supported:
You can configure the credentials used to access GCS in the following ways:

- BR reads the file in the path specified by the environment variable `$GOOGLE_APPLICATION_CREDENTIALS`
- BR reads the file `~/.config/gcloud/application_default_credentials.json`.
- BR obtains the credentials from the metadata server when the cluster is running in GCE or GAE.
- Method 1: Explicitly specify `credentials-file`

- If `credentials-file` points to a Service Account JSON file, BR and TiKV access GCS using this credential.
- If TiKV uses the `gcp_v2` external storage backend, `credentials-file` can also point to the `external_account` JSON used by Google Cloud WIF.

- Method 2: Use Application Default Credentials (ADC)

- BR reads the file in the path specified by the environment variable `$GOOGLE_APPLICATION_CREDENTIALS`.
- BR reads the file `~/.config/gcloud/application_default_credentials.json`.
- When BR runs in GCE or GAE, it uses the credentials obtained from the metadata server.

If you want TiKV to use GCS WIF or ADC, you need to enable the `gcp_v2` external storage backend. Starting from v8.5.7, TiKV enables the `gcp_v2` external storage backend by default. You can configure `gcp_v2` in the following ways:

- full backup and restore: set `[backup].gcp-v2-enable` to `true` in [TiKV Configuration File Descriptions](/tikv-configuration-file.md)
- log backup: set `[log-backup].gcp-v2-enable` to `true` in [TiKV Configuration File Descriptions](/tikv-configuration-file.md)

The default values of the preceding two configuration items are both `true`. If you disable `gcp_v2`, TiKV continues to use the legacy GCS implementation. This implementation supports only Service Account JSON and does not support using WIF directly.

> **Note:**
>
> The GCS JSON credentials explicitly passed to `gcp_v2` support only the `service_account` and `external_account` types. If you are using the `authorized_user` JSON generated by ADC and need TiKV to access GCS directly, it is recommended to set `--send-credentials-to-tikv=false` and configure ADC on each TiKV node. Otherwise, BR might send the `authorized_user` JSON to TiKV as an explicit credential, but `gcp_v2` does not accept this type of explicit JSON.

> **Tip:**
>
> After `gcp_v2` is enabled, if GCS JSON credentials are not explicitly provided on the TiKV side, TiKV uses the Google Default Credentials flow. Therefore, when using ADC and setting `--send-credentials-to-tikv=false`, make sure that each TiKV node itself has a usable Google credentials environment.

</div>
<div label="Azure Blob Storage" value="azure">
Expand Down
18 changes: 17 additions & 1 deletion encryption-at-rest.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,23 @@ credential-file-path = "/path/to/credential.json"
```

- `key-id` specifies the key ID of the KMS CMK.
- `credential-file-path` specifies the path of the authentication credentials file, which currently supports two types of credentials: Service Account and Authentication User. If the TiKV environment is already configured with [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials), there is no need to configure `credential-file-path`.
- When `vendor = "gcp"`, `credential-file-path` specifies the path of the verification credentials file, which currently supports two types of credentials: Service Account and Authentication User. If the TiKV operating environment is already configured with [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials), there is no need to configure `credential-file-path`.

If you need to use Workload Identity Federation (WIF) in the Google Cloud KMS scenario, use `gcp_v2` instead:

```toml
[security.encryption.master-key]
type = "kms"
key-id = "projects/project-name/locations/global/keyRings/key-ring-name/cryptoKeys/key-name"
vendor = "gcp_v2"

[security.encryption.master-key.gcp]
credential-file-path = "/path/to/external-account.json"
```

- When `vendor = "gcp_v2"`, explicit credentials support only Service Account and `external_account`.
- If you are using an `authorized_user` JSON generated by ADC, you cannot directly configure that JSON as `credential-file-path`. In this case, omit `credential-file-path` and let TiKV obtain authentication information through [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials) in the operating environment.
- The old `vendor = "gcp"` does not support using `external_account` as explicit credentials, so WIF cannot be used in this way.

</div>
<div label="Azure KMS">
Expand Down
16 changes: 16 additions & 0 deletions tikv-configuration-file.md
Original file line number Diff line number Diff line change
Expand Up @@ -2427,6 +2427,14 @@ Configuration items related to BR backup.
+ If data is backed up to S3 and the backup file is larger than the value of this configuration item, [multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) is automatically enabled. Based on the compression ratio, the backup file generated by a 96-MiB Region is approximately 10 MiB to 30 MiB.
+ Default value: 5MiB

### `gcp-v2-enable` <span class="version-mark">New in v8.5.7</span>

+ Whether to enable the `gcp_v2` external storage backend when using Google Cloud Storage (GCS) to execute full backup or restore.
+ Default value: `true`
+ When this configuration item is `true`, TiKV uses the `gcp_v2` implementation to access GCS; when this configuration item is `false`, TiKV continues to use the old GCS implementation.
+ If you need to use Google Cloud Workload Identity Federation (WIF) in full backup or restore scenarios, keep this configuration item set to `true`.
+ For information about GCS authentication methods and how to use WIF/ADC, see [Backup storage](/br/backup-and-restore-storages.md).

## backup.hadoop

### `home`
Expand Down Expand Up @@ -2482,6 +2490,14 @@ Configuration items related to log backup.
+ The temporary path to which log files are written before being flushed to external storage.
+ Default value: `${deploy-dir}/data/log-backup-temp`

### `gcp-v2-enable` <span class="version-mark">New in v8.5.7</span>

+ Whether to enable the `gcp_v2` external storage backend when using Google Cloud Storage (GCS) for log backup.
+ Default value: `true`
+ When this configuration item is `true`, TiKV uses the `gcp_v2` implementation to access GCS; when this configuration item is `false`, TiKV continues to use the old GCS implementation.
+ If you need to use Google Cloud Workload Identity Federation (WIF) in log backup scenarios, keep this configuration item set to `true`.
+ For information about GCS authentication methods and how to use WIF/ADC, see [Backup storage](/br/backup-and-restore-storages.md).

## cdc

Configuration items related to TiCDC.
Expand Down
Loading