Overcloud database backup S3 upload playbook

Added playbook uploading overcloud MariaDB backups to S3 compatible
storage. Can be linked as overcloud database backup post hook.

Co-authored-by: Matt Crees <mattc@stackhpc.com>

Author: darmach

doc/source/operations/database-backups.rst

@@ -0,0 +1,40 @@
+================
+Database Backups
+================
+
+An OpenStack deployment includes MariaDB to be used as a database by the
+OpenStack services. Kayobe has `built-in support
+<https://docs.openstack.org/kayobe/latest/administration/overcloud.html#performing-database-backups>`__
+for backing up this database, but these backups are just stored on one of the
+OpenStack controller hosts.
+
+We have a playbook ``tools/upload-database-backup-s3.yml`` which can be used to
+upload these backups to an S3 object store. To use this, you will need:
+
+* The endpoint of the S3 object store.
+
+* EC2 access and secret keys to authenticate to the S3 object store.
+
+* The name of a pre-existing bucket in the S3 object store.
+
+These should be set as follows:
+
+.. code-block:: yaml
+   :caption: ``$KAYOBE_CONFIG_PATH/inventory/group_vars/all/mariadb-backup``
+
+   s3_mariadb_backup_url: "<s3-endpoint>"
+   s3_mariadb_backup_bucket: "<s3-bucket-name>"
+
+.. code-block:: yaml
+   :caption: ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/secrets.yml``
+
+   secrets_s3_mariadb_backup_access_key: "<s3-access-key>"
+   secrets_s3_mariadb_backup_secret_key: "<s3-secret-access-key"
+
+You may also want to hook this to run after ``kayobe overcloud database
+backup``:
+
+.. code-block:: bash
+
+   mkdir -p $KAYOBE_CONFIG_PATH/hooks/overcloud-database-backup/post.d/
+   ln -s ../../../ansible/tools/upload-database-backup-s3.yml $KAYOBE_CONFIG_PATH/hooks/overcloud-database-backup/post.d/10-upload-database-backup-s3.yml

doc/source/operations/index.rst

@@ -10,6 +10,7 @@ This guide is for operators of the StackHPC Kayobe configuration project.
    ceph-management
    control-plane-operation
    customising-horizon
+   database-backups
    gpu-in-openstack
    bifrost-hardware-inventory-management
    hotfix-playbook

etc/kayobe/ansible/tools/upload-database-backup-s3.yml

@@ -0,0 +1,36 @@
+---
+# This playbook uploads MariaDB backups to an AWS S3 object store.
+# Can be linked as a post hook for overcloud-database-backup.
+
+- name: Upload MariaDB backups to S3
+  hosts: controllers[0]
+  vars:
+    backup_directory: "/var/lib/docker/volumes/mariadb_backup/_data"
+    kayobe_venv: "{{ virtualenv_path }}/kayobe"
+  tasks:
+    - name: Ensure AWS S3 module prerequisites are available
+      ansible.builtin.pip:
+        name:
+          - boto3
+          - botocore
+        virtualenv: "{{ kayobe_venv }}"
+
+    - name: Build backup file list
+      ansible.builtin.find:
+        paths: "{{ backup_directory }}"
+      become: True
+      register: backups
+
+    - name: Upload backup files to S3
+      amazon.aws.s3_object:
+        endpoint_url: "{{ s3_mariadb_backup_url }}"
+        access_key: "{{ secrets_s3_mariadb_backup_access_key }}"
+        secret_key: "{{ secrets_s3_mariadb_backup_secret_key }}"
+        bucket: "{{ s3_mariadb_backup_bucket }}"
+        object: "{{ item.path | basename }}"
+        src: "{{ item.path }}"
+        mode: put
+        overwrite: different
+        validate_certs: False
+      become: True
+      with_items: "{{ backups.files }}"