Skip to content

Commit

Permalink
OSDOCS-471: Docs for backing up and restoring encryption keys
Browse files Browse the repository at this point in the history
  • Loading branch information
bergerhoffer committed Dec 17, 2019
1 parent 6feb3bf commit 712ac8a
Show file tree
Hide file tree
Showing 6 changed files with 31 additions and 23 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ At a high level, the procedure is to:
. Correct DNS and load balancer entries.
. Grow etcd to full membership.

If the majority of master hosts have been lost, you will need a xref:../../backup_and_restore/backing-up-etcd.adoc#backing-up-etcd-data_backup-etcd[backed up etcd snapshot] to restore etcd quorum on the remaining master host.
If the majority of master hosts have been lost, you will need an xref:../../backup_and_restore/backing-up-etcd.adoc#backing-up-etcd-data_backup-etcd[etcd backup] to restore etcd quorum on the remaining master host.

[NOTE]
====
Expand Down
12 changes: 7 additions & 5 deletions modules/backup-etcd.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@
[id="backing-up-etcd-data_{context}"]
= Backing up etcd data

Follow these steps to back up etcd data by creating a snapshot. This snapshot can be saved and used at a later time if you need to restore etcd.
Follow these steps to back up etcd data by creating an etcd snapshot and backing up static Kubernetes API server resources. This backup can be saved and used at a later time if you need to restore etcd.

You should only save a snapshot from a single master host. You do not need a snapshot from each master host in the cluster.
// TODO: Need to mention that encryption keys are backed up if etcd encryption is enabled?

You should only save a backup from a single master host. You do not need a backup from each master host in the cluster.

.Prerequisites

Expand All @@ -17,10 +19,10 @@ You should only save a snapshot from a single master host. You do not need a sna

. Access a master host as the root user.

. Run the `etcd-snapshot-backup.sh` script and pass in the location to save the etcd snapshot to.
. Run the `etcd-snapshot-backup.sh` script and pass in the location to save the backup to.
+
----
$ sudo /usr/local/bin/etcd-snapshot-backup.sh ./assets/backup/snapshot.db
$ sudo /usr/local/bin/etcd-snapshot-backup.sh ./assets/backup
----
+
In this example, the snapshot is saved to `./assets/backup/snapshot.db` on the master host.
In this example, the etcd snapshot and static Kubernetes API server resources are saved to `./assets/backup/snapshot_db_kuberesources_<datetimestamp>.tar.gz` on the master host.
3 changes: 2 additions & 1 deletion modules/disabling-etcd-encryption.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,9 @@ $ oc edit apiserver
----
spec:
encryption:
type: identity
type: identity <1>
----
<1> The `identity` type is the default value and means that no encryption is performed.

. Save the file to apply the changes.
+
Expand Down
14 changes: 8 additions & 6 deletions modules/dr-recover-lost-control-plane-hosts.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,17 @@ Follow these steps to recover from the loss of the majority of master hosts, lea

* Access to the cluster as a user with the `cluster-admin` role.
* SSH access to a remaining master host.
* A backed-up etcd snapshot.
* An etcd backup file that includes an etcd snapshot and static Kubernetes API server resources.
+
This backup file must be in the format of `snapshot_db_kuberesources_<datetimestamp>.tar.gz`.

.Procedure

. Restore etcd quorum on the remaining master.

.. Copy the etcd snapshot file to the remaining master host.
.. Copy the etcd backup file to the remaining master host.
+
This procedure assumes that you have copied a snapshot file called `snapshot.db` to the `/home/core/` directory of your master host.
This procedure assumes that you copied the `snapshot_db_kuberesources_<datetimestamp>.tar.gz` file containing the etcd snapshot and static Kubernetes API server resources to the `/home/core/` directory of your master host.

.. Access the remaining master host.

Expand All @@ -31,10 +33,10 @@ This procedure assumes that you have copied a snapshot file called `snapshot.db`

.. Run the `etcd-snapshot-restore.sh` script.
+
Pass in two parameters to the `etcd-snapshot-restore.sh` script: the path to the backed up etcd snapshot file and list of members, which is defined by the `INITIAL_CLUSTER` variable.
Pass in two parameters to the `etcd-snapshot-restore.sh` script: the path to the etcd backup file and list of members, which is defined by the `INITIAL_CLUSTER` variable.
+
----
[core@ip-10-0-143-125 ~]$ sudo /usr/local/bin/etcd-snapshot-restore.sh /home/core/snapshot.db $INITIAL_CLUSTER
[core@ip-10-0-143-125 ~]$ sudo /usr/local/bin/etcd-snapshot-restore.sh /home/core/snapshot_db_kuberesources_<datetimestamp>.tar.gz $INITIAL_CLUSTER
Creating asset directory ./assets
Downloading etcdctl binary..
etcdctl version: 3.3.10
Expand Down Expand Up @@ -280,7 +282,7 @@ sh-4.2# etcdctl member list -w table
+------------------+---------+------------------------------------------+----------------------------------------------------------------+---------------------------+
----
+
It may take up to 10 minutes for the new member to start.
It may take up to 20 minutes for the new member to start.

... Repeat these steps to add your other new master host until you have achieved full etcd membership.

Expand Down
20 changes: 11 additions & 9 deletions modules/dr-restoring-cluster-state.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,19 @@
[id="dr-scenario-2-restoring-cluster-state_{context}"]
= Restoring to a previous cluster state

You can use a saved etcd snapshot to restore back to a previous cluster state.
You can use a saved etcd backup to restore back to a previous cluster state.

.Prerequisites

* Access to the cluster as a user with the `cluster-admin` role.
* SSH access to master hosts.
* A backed-up etcd snapshot.
* An etcd backup file that includes an etcd snapshot and static Kubernetes API server resources.
+
This backup file must be in the format of `snapshot_db_kuberesources_<datetimestamp>.tar.gz`.
+
[NOTE]
====
You must use the same etcd snapshot file on all master hosts in the cluster.
You must use the same etcd backup file on all master hosts in the cluster.
====

.Procedure
Expand All @@ -24,9 +26,9 @@ You must use the same etcd snapshot file on all master hosts in the cluster.
+
You should run the restore script on all of your master hosts within a short period of time so that the cluster members come up at about the same time and form a quorum. For this reason, it is recommended to stage each master host in a separate terminal, so that the restore script can then be started quickly on each.

.. Copy the etcd snapshot file to a master host.
.. Copy the etcd backup file to a master host.
+
This procedure assumes that you have copied a snapshot file called `snapshot.db` to the `/home/core/` directory of your master host.
This procedure assumes that you copied the `snapshot_db_kuberesources_<datetimestamp>.tar.gz` file containing the etcd snapshot and static Kubernetes API server resources to the `/home/core/` directory of your master host.

.. Access the master host.

Expand All @@ -36,14 +38,14 @@ This procedure assumes that you have copied a snapshot file called `snapshot.db`
[core@ip-10-0-143-125 ~]$ export INITIAL_CLUSTER="etcd-member-ip-10-0-143-125.ec2.internal=https://etcd-0.clustername.devcluster.openshift.com:2380,etcd-member-ip-10-0-35-108.ec2.internal=https://etcd-1.clustername.devcluster.openshift.com:2380,etcd-member-ip-10-0-10-16.ec2.internal=https://etcd-2.clustername.devcluster.openshift.com:2380"
----

.. Repeat these steps on your other master hosts, each in a separate terminal. Be sure to use the same etcd snapshot file on each master host.
.. Repeat these steps on your other master hosts, each in a separate terminal. Be sure to use the same etcd backup file on each master host.

. Run the restore script on all of your master hosts.

.. Start the `etcd-snapshot-restore.sh` script on your first master host. Pass in two parameters: the path to the snapshot file and list of members, which is defined by the `INITIAL_CLUSTER` variable.
.. Start the `etcd-snapshot-restore.sh` script on your first master host. Pass in two parameters: the path to the etcd backup file and list of members, which is defined by the `INITIAL_CLUSTER` variable.
+
----
[core@ip-10-0-143-125 ~]$ sudo /usr/local/bin/etcd-snapshot-restore.sh /home/core/snapshot.db $INITIAL_CLUSTER
[core@ip-10-0-143-125 ~]$ sudo /usr/local/bin/etcd-snapshot-restore.sh /home/core/snapshot_db_kuberesources_<datetimestamp>.tar.gz $INITIAL_CLUSTER
Creating asset directory ./assets
Downloading etcdctl binary..
etcdctl version: 3.3.10
Expand Down Expand Up @@ -118,4 +120,4 @@ sh-4.2# etcdctl member list -w table
+------------------+---------+------------------------------------------+------------------------------------------------------------------+---------------------------+
----
+
It may take up to 10 minutes for each new member to start.
It may take up to 20 minutes for each new member to start.
3 changes: 2 additions & 1 deletion modules/enabling-etcd-encryption.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,9 @@ $ oc edit apiserver
----
spec:
encryption:
type: aescbc
type: aescbc <1>
----
<1> The `aescbc` type means that AES-CBC with PKCS#7 padding and a 32 byte key is used to perform the encryption.

. Save the file to apply the changes.
+
Expand Down

0 comments on commit 712ac8a

Please sign in to comment.