Skip to content

Commit

Permalink
Major Changes (#66)
Browse files Browse the repository at this point in the history 
Rewritten script to automatically get inputs for remote instances; removed and renamed inputs; updated the README significantly
  • Loading branch information
@jacksegal
jacksegal authored Nov 13, 2018
1 parent 2a48481 commit 2c30488
Show file tree
Hide file tree
Showing 2 changed files with 272 additions and 244 deletions.
207 changes: 132 additions & 75 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,23 +1,28 @@
# Automatic Snapshots for Google (gcloud) Compute Engine

Bash script for Automatic Snapshots and Cleanup on Google Compute Engine. **Requires no user input!**
Bash script for Automatic Snapshots and Cleanup on Google Compute Engine.

Inspiration (and the installation instructions) taken from AWS script [aws-ec2-ebs-automatic-snapshot-bash](https://github.com/CaseyLabs/aws-ec2-ebs-automatic-snapshot-bash)
**Requires no user input!**

_Inspiration (and the installation instructions) taken from AWS script [aws-ec2-ebs-automatic-snapshot-bash](https://github.com/CaseyLabs/aws-ec2-ebs-automatic-snapshot-bash)_

## How it works
gcloud-snapshot.sh will:

- Determine the Instance ID of the Google Compute Engine server on which the script runs
- Get all the Disk IDs attached to that instance
- Take a snapshot of each Disk
- The script will then delete all associated snapshots taken by the script for the Instance that are older than 7 days (optional: [default snapshot retention can be changed by using -d flag](#snapshot-retention))
- The script will then delete all associated snapshots taken by the script for the Instance that are older than 7 days

The script has a number of **optional** usage options - for example you can:

- Retain snapshots for as long as you'd like [(-d)](#snapshot-retention)
- Backup remote instances too [(-r)](#backing-up-remote-instances)
- Only backup certain disks [(-f)](#matching-on-specific-disks)

## Prerequisites
* `cURL` must be installed
* The VM must have the sufficient gcloud permissions, including "compute" set to "enabled":

[ http://stackoverflow.com/questions/31905966/gcloud-compute-list-networks-error-some-requests-did-not-succeed-insufficie#31928399](http://stackoverflow.com/questions/31905966/gcloud-compute-list-networks-error-some-requests-did-not-succeed-insufficie#31928399)
* The VM must have the sufficient gcloud permissions, including "compute" set to "enabled": [ http://stackoverflow.com/questions/31905966/gcloud-compute-list-networks-error-some-requests-did-not-succeed-insufficie#31928399](http://stackoverflow.com/questions/31905966/gcloud-compute-list-networks-error-some-requests-did-not-succeed-insufficie#31928399)
* The version of gcloud is up to date: `gcloud components update`
* You are authenticated with gcloud - normally this happens automatically, but some users get the error "Insufficient Permission" and need to authenticate: `sudo gcloud auth login`

Expand All @@ -41,66 +46,37 @@ sudo /opt/google-compute-snapshot/gcloud-snapshot.sh

## Automation

**Setup CRON**: You should then setup a cron job in order to schedule a daily backup. Example cron for Debian based Linux:
**Setup CRON**: You should setup a cron job in order to schedule a daily backup. Example cron for Debian based Linux:
```
0 5 * * * root /opt/google-compute-snapshot/gcloud-snapshot.sh >> /var/log/cron/snapshot.log 2>&1
0 5 * * * root /opt/google-compute-snapshot/gcloud-snapshot.sh > /dev/null 2>&1
```

Please note: the above command sends the output to a log file: `/var/log/cron/snapshot.log` - instructions for creating & managing the log file are below.

**Manage CRON Output**: You should then create a directory for all cron outputs and add it to logrotate:

- Create new directory:
```
sudo mkdir /var/log/cron
```
- Create empty file for snapshot log:
```
sudo touch /var/log/cron/snapshot.log
```
- Change permissions on file:
```
sudo chgrp adm /var/log/cron/snapshot.log
sudo chmod 664 /var/log/cron/snapshot.log
```
- Create new entry in logrotate so cron files don't get too big :
```
sudo nano /etc/logrotate.d/cron
```
- Add the following text to the above file:
```
/var/log/cron/*.log {
daily
missingok
rotate 14
compress
notifempty
create 664 root adm
sharedscripts
}
```
If you'd like to save the output of the script to a log, see [Saving output to Log]((#saving-output-to-log))

## Usage Options

```
Usage:
./gcloud-snapshot.sh [-d <days>] [-t <label_name>] [-T <gcloud_filter_expression>] [-i <instance_name>] [-z <instance_zone>] [-p <prefix>] [-a <service_account>]
./gcloud-snapshot.sh [-d <days>] [-r <remote_instances>] [-f <gcloud_filter_expression>] [-p <prefix>] [-a <service_account>] [-n <dry_run>]
Options:
-d Number of days to keep snapshots. Snapshots older than this number deleted.
Default if not set: 7 [OPTIONAL]
-t Only back up disks that have this specified label with value set to 'true'.
-T Only back up disks returned from querying with this filter. Uses gcloud filter expressions
If both -t and -T are used, both terms are joined by the operator AND
-i Instance name to create backups for. If empty, makes backup for the calling
host.
-z Instance zone. If empty, uses the zone of the calling host.
-p Prefix to be used for naming snapshots, default to 'gcs'
-a Service Account to use. If empty, it uses the gcloud default.
-r Backup remote instances - takes snapshots of all disks calling instance has
access to [OPTIONAL].
-f gcloud filter expression to query disk selection [OPTIONAL]
-p Prefix to be used for naming snapshots.
Default if not set: 'gcs' [OPTIONAL]
-a Service Account to use.
Blank if not set [OPTIONAL]
-n Dry run: causes script to print debug variables and doesn't execute any
create / delete commands [OPTIONAL]
```

## Usage Examples

### Snapshot Retention
By default snapshots will be kept for 7 days, however they can be kept for longer / shorter, by using the the -d flag:

Expand All @@ -111,54 +87,135 @@ By default snapshots will be kept for 7 days, however they can be kept for longe
-d Number of days to keep snapshots. Snapshots older than this number deleted.
Default if not set: 7 [OPTIONAL]

For example if you wanted to keep your snapshots for a year, you could run:

./gcloud-snapshot.sh -d 365

### Backing up Remote Instances
By default the script will only backup disks attached to the calling Instance, however you can backup all remote disks that the instance has access to, by using the -r flag:

Usage: ./snapshot.sh [-r <remote_instances>]

Options:

-r Backup remote instances - takes snapshots of all disks calling instance has
access to [OPTIONAL].

For example:

./gcloud-snapshot.sh -r

### Matching on specific disks
By default, snapshots will be created for all attached disks. To only snapshot specific disks (ie. data volumes while skipping boot volumes), use the -t flag:
By default snapshots will be created for all attached disks. To only snapshot specific disks, pass in [gcloud filter expressions](https://cloud.google.com/sdk/gcloud/reference/topic/filters) using the -f flag:

Usage: ./gcloud-snapshot.sh [-t <label>]
Usage: ./gcloud-snapshot.sh [-f <gcloud_filter_expression>]

Options:

-t Only back up disks that have this specified label with value set to 'true'
-f gcloud filter expression to query disk selection

Using Labels: You could add a label of `auto_snapshot=true` to all disks that you wanted backed up and then run:

./gcloud-compute-snapshot.sh -f "labels.auto_snapshot=true"

Example: If you set the label to "auto_snapshot", only disks matching this key/value pair will be snapshotted:
Backup specific zone: If you wanted to only backup disks in a specific zone you could run:

auto_snapshot=true
./gcloud-compute-snapshot.sh -f "zone: us-central1-c"

Use -T for a more flexible way to specify the disks to snapshot.
Usage: ./gcloud-compute-snapshot.sh [-T <gcloud_filter_expression>]
### Snapshot prefix
By default snapshots are created with a prefix of `gcs`. To give a custom prefix use the -p flag:

Usage: ./gcloud-snapshot.sh [-p <prefix>]

Options:

-p Prefix to be used for naming snapshots.
Default if not set: 'gcs' [OPTIONAL]

-T Only back up disks returned from querying with this filter. Uses gcloud filter expressions"
If both -t and -T are used, both terms are joined by the operator AND"
For example:

Example: `./gcloud-compute-snapshot.sh -t auto_snapshot -T "sizeGb = 10 AND name: ubuntu"`. Attached disks matching this expression will be snapshotted
./gcloud-snapshot.sh -p "my-snap"

--format="labels.auto_snapshot=true AND sizeGb = 10 AND name: ubuntu"
(Note: Snapshot names are limited to 62 characters)

This also allows you to bake snapshotting into your Google images by setting a cron job with a label on every image you create, and then you can set a label on the volumes you want to snapshot in your infrastructure management tool (Terraform) to selectively snapshot them.
### Service Account
By default snapshots are created with the default gcloud service account. To use a custom service account use the -a flag:

### Choosing instance for the attached disks
By default, all the disks attached to the calling instance will be snapshotted. To change this behaviour:
Usage: ./gcloud-snapshot.sh [-a <service_account>]

Options:

-a Service Account to use.
Blank if not set [OPTIONAL]

./gcloud-compute-snapshot.sh [-i <instance_name]
For example:

Note: The calling instance (rather than the named instance) needs to have the correct gcloud permissions
./gcloud-snapshot.sh -a "[email protected]"

### Choosing zone
By default, the zone of the calling instance is used. To change this behaviour:
### Dry Run
The script can be run in dry run mode, which doesn't execute any create / delete commands, and prints out debug information.

./gcloud-compute-snapshot.sh [-z <zone>]
Usage: ./gcloud-snapshot.sh [-n <dry_run>]

Options:

-n Dry run: causes script to print debug variables and doesn't execute any
create / delete commands [OPTIONAL]

Note: Even if an instance is named with `-i`, the zone of the calling instance is used
For example if you wanted to a test a gcloud filter expression against remote instances:

### Snapshot names
By default, snapshots are created with names in the format of `prefix-diskName-instanceId-unixTimestamp`. To give a custom prefix:
./gcloud-snapshot.sh -n -r -f "labels.auto_snapshot=true"

./gcloud-compute-snapshot.sh [-p <prefix>]
It would output something like:

Note: Snapshot names are limited to 62 characters.
[2018-11-13 21:04:27]: Start of google-compute-snapshot
[DEBUG]: OLDER_THAN=7
[DEBUG]: REMOTE_CLAUSE=true
[DEBUG]: PREFIX=gcs
[DEBUG]: OPT_ACCOUNT=
[DEBUG]: DRY_RUN=true
[DEBUG]: DELETION_DATE=20181111
[DEBUG]: INSTANCE_NAME=
[2018-11-13 21:04:28]: Handling Snapshots for disk-1
[DEBUG]: gcloud compute disks snapshot disk-1 --snapshot-names gcs-disk-1-1542143067 --zone us-central1-c
[2018-11-13 21:04:29]: Handling Snapshots for instance-1
[DEBUG]: gcloud compute disks snapshot instance-1 --snapshot-names gcs-instance-1-1542143067 --zone us-central1-c
[2018-11-13 21:04:29]: End of google-compute-snapshot

## Saving output to Log

You can easily store the output from this command in a separate CRON log. Example cron for Debian based Linux:
```
0 5 * * * root /opt/google-compute-snapshot/gcloud-snapshot.sh >> /var/log/cron/snapshot.log 2>&1
```

The above command sends the output to a log file: `/var/log/cron/snapshot.log` - instructions for creating & managing the log file are below.

**Manage CRON Output**: You should then create a directory for all cron outputs and add it to logrotate:

- Create the folder, log file, and update the permissions:
```
sudo mkdir /var/log/cron
sudo touch /var/log/cron/snapshot.log
sudo chgrp adm /var/log/cron/snapshot.log
sudo chmod 664 /var/log/cron/snapshot.log
```
- Create new entry in logrotate so cron files don't get too big :
```
sudo nano /etc/logrotate.d/cron
```
- Add the following text to the above file:
```
/var/log/cron/*.log {
daily
missingok
rotate 14
compress
notifempty
create 664 root adm
sharedscripts
}
```

## License

Expand All @@ -170,4 +227,4 @@ Permission is hereby granted, free of charge, to any person obtaining a copy of

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Loading

0 comments on commit 2c30488

Please sign in to comment.