From 712ac8aff48044e795d50a5dbbcfc808a880e06a Mon Sep 17 00:00:00 2001 From: Andrea Hoffer Date: Tue, 10 Dec 2019 15:56:00 -0500 Subject: [PATCH] OSDOCS-471: Docs for backing up and restoring encryption keys --- .../scenario-1-infra-recovery.adoc | 2 +- modules/backup-etcd.adoc | 12 ++++++----- modules/disabling-etcd-encryption.adoc | 3 ++- .../dr-recover-lost-control-plane-hosts.adoc | 14 +++++++------ modules/dr-restoring-cluster-state.adoc | 20 ++++++++++--------- modules/enabling-etcd-encryption.adoc | 3 ++- 6 files changed, 31 insertions(+), 23 deletions(-) diff --git a/backup_and_restore/disaster_recovery/scenario-1-infra-recovery.adoc b/backup_and_restore/disaster_recovery/scenario-1-infra-recovery.adoc index bd2c71e58224..d2dc4e0f8dac 100644 --- a/backup_and_restore/disaster_recovery/scenario-1-infra-recovery.adoc +++ b/backup_and_restore/disaster_recovery/scenario-1-infra-recovery.adoc @@ -15,7 +15,7 @@ At a high level, the procedure is to: . Correct DNS and load balancer entries. . Grow etcd to full membership. -If the majority of master hosts have been lost, you will need a xref:../../backup_and_restore/backing-up-etcd.adoc#backing-up-etcd-data_backup-etcd[backed up etcd snapshot] to restore etcd quorum on the remaining master host. +If the majority of master hosts have been lost, you will need an xref:../../backup_and_restore/backing-up-etcd.adoc#backing-up-etcd-data_backup-etcd[etcd backup] to restore etcd quorum on the remaining master host. [NOTE] ==== diff --git a/modules/backup-etcd.adoc b/modules/backup-etcd.adoc index 09c3bc021ca2..e93f28e26bd5 100644 --- a/modules/backup-etcd.adoc +++ b/modules/backup-etcd.adoc @@ -5,9 +5,11 @@ [id="backing-up-etcd-data_{context}"] = Backing up etcd data -Follow these steps to back up etcd data by creating a snapshot. This snapshot can be saved and used at a later time if you need to restore etcd. +Follow these steps to back up etcd data by creating an etcd snapshot and backing up static Kubernetes API server resources. This backup can be saved and used at a later time if you need to restore etcd. -You should only save a snapshot from a single master host. You do not need a snapshot from each master host in the cluster. +// TODO: Need to mention that encryption keys are backed up if etcd encryption is enabled? + +You should only save a backup from a single master host. You do not need a backup from each master host in the cluster. .Prerequisites @@ -17,10 +19,10 @@ You should only save a snapshot from a single master host. You do not need a sna . Access a master host as the root user. -. Run the `etcd-snapshot-backup.sh` script and pass in the location to save the etcd snapshot to. +. Run the `etcd-snapshot-backup.sh` script and pass in the location to save the backup to. + ---- -$ sudo /usr/local/bin/etcd-snapshot-backup.sh ./assets/backup/snapshot.db +$ sudo /usr/local/bin/etcd-snapshot-backup.sh ./assets/backup ---- + -In this example, the snapshot is saved to `./assets/backup/snapshot.db` on the master host. +In this example, the etcd snapshot and static Kubernetes API server resources are saved to `./assets/backup/snapshot_db_kuberesources_.tar.gz` on the master host. diff --git a/modules/disabling-etcd-encryption.adoc b/modules/disabling-etcd-encryption.adoc index 60bd9bbc406d..1b7d8b41ffc5 100644 --- a/modules/disabling-etcd-encryption.adoc +++ b/modules/disabling-etcd-encryption.adoc @@ -25,8 +25,9 @@ $ oc edit apiserver ---- spec: encryption: - type: identity + type: identity <1> ---- +<1> The `identity` type is the default value and means that no encryption is performed. . Save the file to apply the changes. + diff --git a/modules/dr-recover-lost-control-plane-hosts.adoc b/modules/dr-recover-lost-control-plane-hosts.adoc index 2e73b805ccd2..c165ee85b254 100644 --- a/modules/dr-recover-lost-control-plane-hosts.adoc +++ b/modules/dr-recover-lost-control-plane-hosts.adoc @@ -11,15 +11,17 @@ Follow these steps to recover from the loss of the majority of master hosts, lea * Access to the cluster as a user with the `cluster-admin` role. * SSH access to a remaining master host. -* A backed-up etcd snapshot. +* An etcd backup file that includes an etcd snapshot and static Kubernetes API server resources. ++ +This backup file must be in the format of `snapshot_db_kuberesources_.tar.gz`. .Procedure . Restore etcd quorum on the remaining master. -.. Copy the etcd snapshot file to the remaining master host. +.. Copy the etcd backup file to the remaining master host. + -This procedure assumes that you have copied a snapshot file called `snapshot.db` to the `/home/core/` directory of your master host. +This procedure assumes that you copied the `snapshot_db_kuberesources_.tar.gz` file containing the etcd snapshot and static Kubernetes API server resources to the `/home/core/` directory of your master host. .. Access the remaining master host. @@ -31,10 +33,10 @@ This procedure assumes that you have copied a snapshot file called `snapshot.db` .. Run the `etcd-snapshot-restore.sh` script. + -Pass in two parameters to the `etcd-snapshot-restore.sh` script: the path to the backed up etcd snapshot file and list of members, which is defined by the `INITIAL_CLUSTER` variable. +Pass in two parameters to the `etcd-snapshot-restore.sh` script: the path to the etcd backup file and list of members, which is defined by the `INITIAL_CLUSTER` variable. + ---- -[core@ip-10-0-143-125 ~]$ sudo /usr/local/bin/etcd-snapshot-restore.sh /home/core/snapshot.db $INITIAL_CLUSTER +[core@ip-10-0-143-125 ~]$ sudo /usr/local/bin/etcd-snapshot-restore.sh /home/core/snapshot_db_kuberesources_.tar.gz $INITIAL_CLUSTER Creating asset directory ./assets Downloading etcdctl binary.. etcdctl version: 3.3.10 @@ -280,7 +282,7 @@ sh-4.2# etcdctl member list -w table +------------------+---------+------------------------------------------+----------------------------------------------------------------+---------------------------+ ---- + -It may take up to 10 minutes for the new member to start. +It may take up to 20 minutes for the new member to start. ... Repeat these steps to add your other new master host until you have achieved full etcd membership. diff --git a/modules/dr-restoring-cluster-state.adoc b/modules/dr-restoring-cluster-state.adoc index cbcca755c4a5..b24313f4ce7d 100644 --- a/modules/dr-restoring-cluster-state.adoc +++ b/modules/dr-restoring-cluster-state.adoc @@ -5,17 +5,19 @@ [id="dr-scenario-2-restoring-cluster-state_{context}"] = Restoring to a previous cluster state -You can use a saved etcd snapshot to restore back to a previous cluster state. +You can use a saved etcd backup to restore back to a previous cluster state. .Prerequisites * Access to the cluster as a user with the `cluster-admin` role. * SSH access to master hosts. -* A backed-up etcd snapshot. +* An etcd backup file that includes an etcd snapshot and static Kubernetes API server resources. ++ +This backup file must be in the format of `snapshot_db_kuberesources_.tar.gz`. + [NOTE] ==== -You must use the same etcd snapshot file on all master hosts in the cluster. +You must use the same etcd backup file on all master hosts in the cluster. ==== .Procedure @@ -24,9 +26,9 @@ You must use the same etcd snapshot file on all master hosts in the cluster. + You should run the restore script on all of your master hosts within a short period of time so that the cluster members come up at about the same time and form a quorum. For this reason, it is recommended to stage each master host in a separate terminal, so that the restore script can then be started quickly on each. -.. Copy the etcd snapshot file to a master host. +.. Copy the etcd backup file to a master host. + -This procedure assumes that you have copied a snapshot file called `snapshot.db` to the `/home/core/` directory of your master host. +This procedure assumes that you copied the `snapshot_db_kuberesources_.tar.gz` file containing the etcd snapshot and static Kubernetes API server resources to the `/home/core/` directory of your master host. .. Access the master host. @@ -36,14 +38,14 @@ This procedure assumes that you have copied a snapshot file called `snapshot.db` [core@ip-10-0-143-125 ~]$ export INITIAL_CLUSTER="etcd-member-ip-10-0-143-125.ec2.internal=https://etcd-0.clustername.devcluster.openshift.com:2380,etcd-member-ip-10-0-35-108.ec2.internal=https://etcd-1.clustername.devcluster.openshift.com:2380,etcd-member-ip-10-0-10-16.ec2.internal=https://etcd-2.clustername.devcluster.openshift.com:2380" ---- -.. Repeat these steps on your other master hosts, each in a separate terminal. Be sure to use the same etcd snapshot file on each master host. +.. Repeat these steps on your other master hosts, each in a separate terminal. Be sure to use the same etcd backup file on each master host. . Run the restore script on all of your master hosts. -.. Start the `etcd-snapshot-restore.sh` script on your first master host. Pass in two parameters: the path to the snapshot file and list of members, which is defined by the `INITIAL_CLUSTER` variable. +.. Start the `etcd-snapshot-restore.sh` script on your first master host. Pass in two parameters: the path to the etcd backup file and list of members, which is defined by the `INITIAL_CLUSTER` variable. + ---- -[core@ip-10-0-143-125 ~]$ sudo /usr/local/bin/etcd-snapshot-restore.sh /home/core/snapshot.db $INITIAL_CLUSTER +[core@ip-10-0-143-125 ~]$ sudo /usr/local/bin/etcd-snapshot-restore.sh /home/core/snapshot_db_kuberesources_.tar.gz $INITIAL_CLUSTER Creating asset directory ./assets Downloading etcdctl binary.. etcdctl version: 3.3.10 @@ -118,4 +120,4 @@ sh-4.2# etcdctl member list -w table +------------------+---------+------------------------------------------+------------------------------------------------------------------+---------------------------+ ---- + -It may take up to 10 minutes for each new member to start. +It may take up to 20 minutes for each new member to start. diff --git a/modules/enabling-etcd-encryption.adoc b/modules/enabling-etcd-encryption.adoc index 294d0a2d973e..90b32538b937 100644 --- a/modules/enabling-etcd-encryption.adoc +++ b/modules/enabling-etcd-encryption.adoc @@ -30,8 +30,9 @@ $ oc edit apiserver ---- spec: encryption: - type: aescbc + type: aescbc <1> ---- +<1> The `aescbc` type means that AES-CBC with PKCS#7 padding and a 32 byte key is used to perform the encryption. . Save the file to apply the changes. +