diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 000000000..4a3c6fe92 --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,11 @@ +# Required +version: 2 + +mkdocs: + configuration: mkdocs.yml + +# Optionally set the version of Python and requirements required to build your docs +python: + version: 3.7 + install: + - requirements: docs/requirements.txt diff --git a/FoundationDB.md b/docs/FoundationDB.md similarity index 93% rename from FoundationDB.md rename to docs/FoundationDB.md index 12232bfc3..8a05b8180 100644 --- a/FoundationDB.md +++ b/docs/FoundationDB.md @@ -1,4 +1,4 @@ -## WAL-G for FoundationDB +# WAL-G for FoundationDB **Work in progress** @@ -13,7 +13,7 @@ Optional: - To build with libsodium, just set `USE_LIBSODIUM` environment variable. - To build with lzo decompressor, just set `USE_LZO` environment variable. -``` +```plaintext go get github.com/wal-g/wal-g cd $GOPATH/src/github.com/wal-g/wal-g make install @@ -21,7 +21,7 @@ make deps make fdb_build ``` Users can also install WAL-G by using `make install`. Specifying the GOBIN environment variable before installing allows the user to specify the installation location. On default, `make install` puts the compiled binary in `go/bin`. -``` +```plaintext export GOBIN=/usr/local/bin cd $GOPATH/src/github.com/wal-g/wal-g make install @@ -32,11 +32,11 @@ make fdb_install Usage ----- -* ``backup-fetch`` +### ``backup-fetch`` Command for sending backup from storage to stream in order to restore it in the database. -``` +```bash wal-g backup-fetch example_backup ``` @@ -45,15 +45,15 @@ Variable _WALG_STREAM_RESTORE_COMMAND_ is required for use backup-fetch WAL-G can also fetch the latest backup using: -``` +```bash wal-g backup-fetch LATEST ``` -* ``backup-push`` +### ``backup-push`` Command for compressing, encrypting and sending backup from stream to storage. -``` +```bash wal-g backup-push ``` diff --git a/MongoDB.md b/docs/MongoDB.md similarity index 92% rename from MongoDB.md rename to docs/MongoDB.md index b07ea2066..bfdde9f63 100644 --- a/MongoDB.md +++ b/docs/MongoDB.md @@ -1,4 +1,4 @@ -## WAL-G for MongoDB +# WAL-G for MongoDB **Interface of MongoDB features is currently unstable** @@ -13,7 +13,7 @@ Optional: - To build with libsodium, just set `USE_LIBSODIUM` environment variable. - To build with lzo decompressor, just set `USE_LZO` environment variable. -``` +```plaintext go get github.com/wal-g/wal-g cd $GOPATH/src/github.com/wal-g/wal-g make install @@ -21,7 +21,7 @@ make deps make mongo_build ``` Users can also install WAL-G by using `make install`. Specifying the GOBIN environment variable before installing allows the user to specify the installation location. On default, `make install` puts the compiled binary in `go/bin`. -``` +```plaintext export GOBIN=/usr/local/bin cd $GOPATH/src/github.com/wal-g/wal-g make install @@ -32,41 +32,41 @@ make mongo_install Configuration ------------- -#### `WALG_STREAM_CREATE_COMMAND` +* `WALG_STREAM_CREATE_COMMAND` Command to create MongoDB backup, should return backup as single stream to STDOUT. Required for backup procedure. -#### `WALG_STREAM_RESTORE_COMMAND` +* `WALG_STREAM_RESTORE_COMMAND` Command to unpack MongoDB backup, should take backup (created by `WALG_STREAM_CREATE_COMMAND`) to STDIN and push it to MongoDB instance. Required for restore procedure. -#### `MONGODB_URI` +* `MONGODB_URI` URI used to connect to a MongoDB instance. Required for backup and oplog archiving procedure. -#### `OPLOG_ARCHIVE_AFTER_SIZE` +* `OPLOG_ARCHIVE_AFTER_SIZE` Oplog archive batch in bytes which triggers upload to storage. -#### `OPLOG_ARCHIVE_TIMEOUT_INTERVAL` +* `OPLOG_ARCHIVE_TIMEOUT_INTERVAL` Time interval (passed since previous upload) to trigger upload to storage. Format: [golang duration string](https://golang.org/pkg/time/#ParseDuration). -#### `MONGODB_LAST_WRITE_UPDATE_INTERVAL` +* `MONGODB_LAST_WRITE_UPDATE_INTERVAL` Interval to update the latest majority optime. wal-g archives only majority committed operations. Format: [golang duration string](https://golang.org/pkg/time/#ParseDuration). -#### `OPLOG_PUSH_WAIT_FOR_BECOME_PRIMARY` +* `OPLOG_PUSH_WAIT_FOR_BECOME_PRIMARY` Wait for primary and start archiving or exit immediately. Archiving works only on primary, but it's useful to run wal-g on all replicaset nodes with `OPLOG_PUSH_WAIT_FOR_BECOME_PRIMARY: true` to handle replica set elections. Then new primary will catch up archiving after elections. -#### `OPLOG_PITR_DISCOVERY_INTERVAL` +* `OPLOG_PITR_DISCOVERY_INTERVAL` Defines the longest possible point-in-time recovery period. It's lasts from starting timestamp of the oldest backup (within `OPLOG_PITR_DISCOVERY_INTERVAL`) until now. @@ -75,20 +75,20 @@ Setting is used by oplog archives [purging](#oplog-purge). Format: [golang duration string](https://golang.org/pkg/time/#ParseDuration). -#### `OPLOG_PUSH_STATS_ENABLED` +* `OPLOG_PUSH_STATS_ENABLED` Enables statistics collecting of oplog archiving procedure. -#### `OPLOG_PUSH_STATS_UPDATE_INTERVAL` +* `OPLOG_PUSH_STATS_UPDATE_INTERVAL` Interval to update oplog archiving statistics. Disabled if reset to 0. -#### `OPLOG_PUSH_STATS_LOGGING_INTERVAL` +* `OPLOG_PUSH_STATS_LOGGING_INTERVAL` Interval to log oplog archiving statistics. Disabled if reset to 0. Format: [golang duration string](https://golang.org/pkg/time/#ParseDuration). -#### `OPLOG_PUSH_STATS_EXPOSE_HTTP` +* `OPLOG_PUSH_STATS_EXPOSE_HTTP` Exposes http-handler with oplog archiving statistics: `stats/oplog_push`. HTTP-server listens `HTTP_LISTEN` port (default: 8090). @@ -99,41 +99,41 @@ Usage WAL-G mongodb extension currently supports these commands: -#### ``backup-push`` +### ``backup-push`` Creates new backup and send it to storage. Runs `WALG_STREAM_CREATE_COMMAND` to create backup. -``` +```bash wal-g backup-push ``` -#### `backup-list` +### `backup-list` Lists currently available backups in storage. -``` +```bash wal-g backup-list ``` -#### `backup-fetch` +### `backup-fetch` Fetches backup from storage and restores passes data to `WALG_STREAM_RESTORE_COMMAND` to restore backup. User should specify the name of the backup to fetch. -``` +```bash wal-g backup-fetch example_backup ``` -#### `backup-show` +### `backup-show` Fetches backup metadata from storage to STDOUT. User should specify the name of the backup to show. -``` +```bash # wal-g backup-show stream_20201027T224823Z { "BackupName": "stream_20201027T224823Z", @@ -165,23 +165,23 @@ User should specify the name of the backup to show. "StartLocalTime": "2020-10-28T01:48:23.121314+03:00" } ``` -#### `backup-delete` +### `backup-delete` Deletes backup from storage. User should specify the name of the backup to delete. Dry-run -``` +```bash wal-g backup-delete example_backup ``` Perform delete -``` +```bash wal-g backup-delete example_backup --confirm ``` -#### `oplog-push` +### `oplog-push` Fetches oplog from mongodb instance (`MONGODB_URI`) and uploads to storage. @@ -191,31 +191,31 @@ wal-g forces upload when, Archiving collects writes if optime is readable by majority reads. Optime is updated every `MONGODB_LAST_WRITE_UPDATE_INTERVAL`. -``` +```bash wal-g oplog-push ``` Note: archiving works only on primary, but you can run it on any replicaset node using config option `OPLOG_PUSH_WAIT_FOR_BECOME_PRIMARY: true`. -#### `oplog-replay` +### `oplog-replay` Fetches oplog archives from storage and applies to mongodb instance (`MONGODB_URI`) User should specify SINCE and UNTIL boundaries (format: `timestamp.inc`, eg `1593554809.32`). Both of them should exist in storage. SINCE is included and UNTIL is NOT. -``` +```bash wal-g oplog-replay 1593554109.1 1593559109.1 ``` -##### Common constraints: +### Common constraints: - SINCE: operation timestamp before full backup started. - UNTIL: operation timestamp after backup finished. Use `MongoMeta.Before.LastMajTS` and `MongoMeta.After.LastMajTS` fields from backup [metadata](#backup-show). -#### `oplog-fetch` +### `oplog-fetch` Fetches oplog archives from storage and passes to STDOUT. @@ -224,23 +224,23 @@ SINCE is included and UNTIL is NOT. Supported formats to output: `json`, `bson`, `bson-raw` -``` +```bash wal-g oplog-fetch 1593554109.1 1593559109.1 --format json ``` -#### `oplog-purge` +### `oplog-purge` Purges outdated oplog archives from storage. Clean-up will retain: - oplog archives in [PITR interval](#oplog_pitr_discovery_interval) - oplog archives within backup creation period Dry-run -``` +```bash wal-g oplog-purge ``` Perform clean-up -``` +```bash wal-g oplog-purge --confirm ``` @@ -250,7 +250,7 @@ Typical configurations ### Full backup/restore only Here's typical wal-g configuration for that case: -``` +```bash MONGODB_URI: 'mongodb://user:password@localhost:27018/?authSource=admin&connect=direct&socketTimeoutMS=60000&connectTimeoutMS=10000&tls=true' WALG_STREAM_CREATE_COMMAND: 'mongodump --archive --oplog --uri="mongodb://user:password@localhost:27018/?authSource=admin&connectTimeoutMS=10000&tls=true"' WALG_STREAM_RESTORE_COMMAND: 'mongorestore --archive --oplogReplay --uri="mongodb://user:password@localhost:27018/?authSource=admin&connectTimeoutMS=10000&tls=true"' @@ -261,7 +261,7 @@ You may also add `--drop` option to restore command. This option drops the colle ### Continuous archiving and point-in-time recovery Here's typical wal-g configuration for that case: -``` +```bash # Used to fetch oplog documents MONGODB_URI: 'mongodb://user:password@localhost:27018/?authSource=admin&connect=direct&socketTimeoutMS=60000&connectTimeoutMS=10000&tls=true' @@ -291,7 +291,7 @@ WALG_STREAM_RESTORE_COMMAND: 'mongorestore --archive --uri="mongodb://user:passw ### How to restore backup to point in time Suppose you want to restore your cluster to `2020-10-28T12:11:10+03:00`. -``` +```bash # wal-g backup-list -v name finish_local_time ts_before ts_after data_size permanent stream_20201025T222118Z 2020-10-26T01:50:17+03:00 1603664478.21 1603666217.6 18521875261 false @@ -299,10 +299,10 @@ stream_20201026T220833Z 2020-10-27T01:37:42+03:00 1603750113.39 1603751861.6 18 stream_20201027T224823Z 2020-10-28T02:08:55+03:00 1603838903.4 1603840135.34 18952713762 false ``` Pick the closest backup and restore it (don't forget about [constraints](#common-constraints)): -``` +```bash # wal-g backup-fetch stream_20201027T224823Z ``` Replay oplog from backup `ts_before` to `2020-10-28T12:11:10+03:00`: -``` +```bash # wal-g oplog-replay 1603838903.4 1603876270.1 -``` \ No newline at end of file +``` diff --git a/MySQL.md b/docs/MySQL.md similarity index 96% rename from MySQL.md rename to docs/MySQL.md index 73eaf1ee4..9e51693ee 100644 --- a/MySQL.md +++ b/docs/MySQL.md @@ -1,4 +1,4 @@ -## WAL-G for MySQL +# WAL-G for MySQL **Interface of MySQL and MariaDB now is unstable** @@ -13,7 +13,7 @@ Optional: - To build with libsodium, just set `USE_LIBSODIUM` environment variable. - To build with lzo decompressor, just set `USE_LZO` environment variable. -``` +```plaintext go get github.com/wal-g/wal-g cd $GOPATH/src/github.com/wal-g/wal-g make install @@ -21,7 +21,7 @@ make deps make mysql_build ``` Users can also install WAL-G by using `make install`. Specifying the GOBIN environment variable before installing allows the user to specify the installation location. On default, `make install` puts the compiled binary in `go/bin`. -``` +```plaintext export GOBIN=/usr/local/bin cd $GOPATH/src/github.com/wal-g/wal-g make install @@ -40,11 +40,11 @@ To configure the connection string for MySQL. Required. Format ```user:password@ To use SSL, a path to file with certificates should be set to this variable. -* `WALG_STREAM_CREATE_COMMAND` +* `WALG_STREAM_CREATE_COMMAND` Command to create MySQL backup, should return backup as single stream to STDOUT. Requried. -* `WALG_STREAM_RESTORE_COMMAND` +* `WALG_STREAM_RESTORE_COMMAND` Command to unpack MySQL backup, should take backup (created by `WALG_STREAM_CREATE_COMMAND`) to STDIN and unpack it to MySQL datadir. Required. @@ -74,66 +74,66 @@ Usage WAL-G mysql extension currently supports these commands: -* ``backup-push`` +### ``backup-push`` Creates new backup and send it to storage. Runs `WALG_STREAM_CREATE_COMMAND` to create backup. -``` +```bash wal-g backup-push ``` -* ``backup-list`` +### ``backup-list`` Lists currently available backups in storage -``` +```bash wal-g backup-list ``` -* ``backup-fetch`` +### ``backup-fetch`` Fetches backup from storage and restores it to datadir. Runs `WALG_STREAM_RESTORE_COMMAND` to restore backup. User should specify the name of the backup to fetch. -``` +```bash wal-g backup-fetch example_backup ``` WAL-G can also fetch the latest backup using: -``` +```bash wal-g backup-fetch LATEST ``` -* ``binlog-push`` +### ``binlog-push`` Sends (not yet archived) binlogs to storage. Typically run in CRON. -``` +```bash wal-g binlog-push ``` -* ``binlog-fetch`` +### ``binlog-fetch`` Fetches binlogs from storage and saves them to `WALG_MYSQL_BINLOG_DST` folder. User should specify the name of the backup starting with which to fetch an binlog. User may also specify time in RFC3339 format until which should be fetched (used for PITR). User have to replay binlogs manually in that case. -``` +```bash wal-g binlog-fetch --since "backupname" ``` or -``` +```bash wal-g binlog-fetch --since "backupname" --until "2006-01-02T15:04:05Z07:00" ``` or -``` +```bash wal-g binlog-fetch --since LATEST --until "2006-01-02T15:04:05Z07:00" ``` -* ``binlog-replay`` +### ``binlog-replay`` Fetches binlogs from storage and passes them to `WALG_MYSQL_BINLOG_REPLAY_COMMAND` to replay on running MySQL server. User should specify the name of the backup starting with which to fetch an binlog. @@ -142,15 +142,15 @@ If `until` timestamp is in the future, wal-g will search for newly uploaded binl Binlogs are temporarily save in `WALG_MYSQL_BINLOG_DST` folder. Replay command gets name of binlog to replay via environment variable `WALG_MYSQL_CURRENT_BINLOG` and stop-date via `WALG_MYSQL_BINLOG_END_TS`, which are set for each invocation. -``` +```bash wal-g binlog-replay --since "backupname" ``` or -``` +```bash wal-g binlog-replay --since "backupname" --until "2006-01-02T15:04:05Z07:00" ``` or -``` +```bash wal-g binlog-replay --since LATEST --until "2006-01-02T15:04:05Z07:00" ``` @@ -163,7 +163,7 @@ Typical configurations It's recommended to use wal-g with xtrabackup tool in case of MySQL for creating lock-less backups. Here's typical wal-g configuration for that case: -``` +```bash WALG_MYSQL_DATASOURCE_NAME=user:pass@tcp(localhost:3306)/mysql WALG_STREAM_CREATE_COMMAND="xtrabackup --backup --stream=xbstream --datadir=/var/lib/mysql" WALG_STREAM_RESTORE_COMMAND="xbstream -x -C /var/lib/mysql" @@ -182,7 +182,7 @@ gtids=$(tr -d '\n' < /var/lib/mysql/xtrabackup_binlog_info | awk '{print $3}') mysql -e "RESET MASTER; SET @@GLOBAL.GTID_PURGED='$gtids';" ``` * for PITR, replay binlogs with -``` +```bash wal-g binlog-replay --since "backup_name" --until "2006-01-02T15:04:05Z07:00" ``` @@ -193,7 +193,7 @@ It's possible to use wal-g with standard mysqldump/mysql tools. In that case MySQL mysql backup is a plain SQL script. Here's typical wal-g configuration for that case: -``` +```bash WALG_MYSQL_DATASOURCE_NAME=user:pass@localhost/mysql WALG_STREAM_CREATE_COMMAND="mysqldump --all-databases --single-transaction --set-gtid-purged=ON" WALG_STREAM_RESTORE_COMMAND="mysql" @@ -209,7 +209,7 @@ Restore procedure is straightforward: It's recommended to use wal-g with `mariabackup` tool in case of MariaDB for creating lock-less backups. Here's typical wal-g configuration for that case: -``` +```bash WALG_MYSQL_DATASOURCE_NAME=user:pass@tcp(localhost:3305)/mysql WALG_STREAM_CREATE_COMMAND="mariabackup --backup --stream=xbstream --datadir=/var/lib/mysql" WALG_STREAM_RESTORE_COMMAND="mbstream -x -C /var/lib/mysql" @@ -224,7 +224,7 @@ For the restore procedure you have to do similar things to [what the offical doc * after the previous step you might have to fix file permissions: `chown -R mysql:mysql /var/lib/mysql` * in case of restoring for a replication slave you can follow the [official docs](https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/#gtids) * for PITR, replay binlogs with -``` +```bash wal-g binlog-replay --since "backup_name" --until "2006-01-02T15:04:05Z07:00" ``` * start mariadb diff --git a/PostgreSQL.md b/docs/PostgreSQL.md similarity index 97% rename from PostgreSQL.md rename to docs/PostgreSQL.md index 913baf9c5..f799d7fbc 100644 --- a/PostgreSQL.md +++ b/docs/PostgreSQL.md @@ -1,4 +1,4 @@ -## WAL-G for PostgreSQL +# WAL-G for PostgreSQL You can use wal-g as a tool for making encrypted, compressed PostgreSQL backups(full and incremental) and push/fetch them to/from storage without saving it on your filesystem. @@ -7,12 +7,12 @@ Development ### Installing Prepare on Ubuntu: -``` +```plaintext sudo apt-get install liblzo2-dev ``` Prepare on Mac OS: -``` +``` plaintext # brew command is Homebrew for Mac OS brew install cmake export USE_LIBSODIUM="true" # since we're linking libsodium later @@ -28,7 +28,7 @@ Optional: - To build with libsodium, just set `USE_LIBSODIUM` environment variable. - To build with lzo decompressor, just set `USE_LZO` environment variable. -``` +```plaintext go get github.com/wal-g/wal-g cd $GOPATH/src/github.com/wal-g/wal-g make install @@ -38,7 +38,7 @@ make pg_build Users can also install WAL-G by using `make install`. Specifying the GOBIN environment variable before installing allows the user to specify the installation location. On default, `make install` puts the compiled binary in `go/bin`. -``` +```plaintext export GOBIN=/usr/local/bin cd $GOPATH/src/github.com/wal-g/wal-g make install @@ -139,7 +139,7 @@ To configure the wal segment size if different from the postgres default of 16 M To upload metadata related to wal files. `WALG_UPLOAD_WAL_METADATA` can be INDIVIDUAL (generates metadata for all the wal logs) or BULK( generates metadata for set of wal files) Sample metadata file (000000020000000300000071.json) -``` +```bash { "000000020000000300000071": { "created_time": "2021-02-23T00:51:14.195209969Z", @@ -152,22 +152,22 @@ If the parameter value is NOMETADATA or not specified, it will fallback to defau Usage ----- -* ``backup-fetch`` +### ``backup-fetch`` When fetching base backups, the user should pass in the name of the backup and a path to a directory to extract to. If this directory does not exist, WAL-G will create it and any dependent subdirectories. -``` +```bash wal-g backup-fetch ~/extract/to/here example-backup ``` WAL-G can also fetch the latest backup using: -``` +```bash wal-g backup-fetch ~/extract/to/here LATEST ``` WAL-G can fetch the backup with specific UserData (stored in backup metadata) using the `--target-user-data` flag or `WALG_FETCH_TARGET_USER_DATA` variable: -``` +```bash wal-g backup-fetch /path --target-user-data "{ \"x\": [3], \"y\": 4 }" ``` @@ -181,7 +181,7 @@ To activate this feature, do one of the following: * set the `WALG_USE_REVERSE_UNPACK`environment variable * add the --reverse-unpack flag -``` +```bash wal-g backup-fetch /path LATEST --reverse-unpack ``` @@ -197,15 +197,15 @@ Since this feature involves both backup creation and restore process, in order t * set the `WALG_USE_REVERSE_UNPACK` and `WALG_SKIP_REDUNDANT_TARS` environment variables * add the `--reverse-unpack` and `--skip-redundant-tars` flags -``` +```bash wal-g backup-fetch /path LATEST --reverse-unpack --skip-redundant-tars ``` -* ``backup-push`` +### ``backup-push`` When uploading backups to S3, the user should pass in the path containing the backup started by Postgres as in: -``` +```bash wal-g backup-push /backup/directory/path ``` @@ -221,7 +221,7 @@ WAL-G backup-push allows for two data streaming options: For uploading backups to S3 in streaming option 1, the user should pass in the path containing the backup started by Postgres as in: -``` +```bash wal-g backup-push /backup/directory/path ``` @@ -229,7 +229,7 @@ wal-g backup-push /backup/directory/path To stream the backup data, leave out the data directory. And to set the hostname for the postgres server, you can use the environment variable PGHOST, or the WAL-G argument --pghost. -``` +```bash # Inline PGHOST=srv1 wal-g backup-push @@ -255,7 +255,7 @@ To activate this feature, do one of the following: * set the `WALG_USE_RATING_COMPOSER`environment variable * add the --rating-composer flag -``` +```bash wal-g backup-push /path --rating-composer ``` @@ -267,13 +267,13 @@ When creating delta backup (`WALG_DELTA_MAX_STEPS` > 0), WAL-G uses the latest b * `--delta-from-user-data` flag or `WALG_DELTA_FROM_USER_DATA` environment variable to choose the backup with specified user data as the base for the delta backup Examples: -``` +```bash wal-g backup-push /path --delta-from-name base_000000010000000100000072_D_000000010000000100000063 wal-g backup-push /path --delta-from-user-data "{ \"x\": [3], \"y\": 4 }" ``` When using the above flags in combination with `WALG_DELTA_ORIGIN` setting, `WALG_DELTA_ORIGIN` logic applies to the specified backup. For example: -``` +```bash list of backups in storage: base_000000010000000100000040 # full backup base_000000010000000100000046_D_000000010000000100000040 # 1st delta @@ -289,32 +289,32 @@ INFO: Delta will be made from full backup. INFO: Delta backup from base_000000010000000100000040 with LSN 140000060. ``` -* ``wal-fetch`` +### ``wal-fetch`` When fetching WAL archives from S3, the user should pass in the archive name and the name of the file to download to. This file should not exist as WAL-G will create it for you. WAL-G will also prefetch WAL files ahead of asked WAL file. These files will be cached in `./.wal-g/prefetch` directory. Cache files older than recently asked WAL file will be deleted from the cache, to prevent cache bloat. If the file is requested with `wal-fetch` this will also remove it from cache, but trigger fulfilment of cache with new file. -``` +```bash wal-g wal-fetch example-archive new-file-name ``` -* ``wal-push`` +### ``wal-push`` When uploading WAL archives to S3, the user should pass in the absolute path to where the archive is located. -``` +```bash wal-g wal-push /path/to/archive ``` -* ``wal-show`` +### ``wal-show`` Show information about the WAL storage folder. `wal-show` shows all WAL segment timelines available in storage, displays the available backups for them, and checks them for missing segments. * if there are no gaps (missing segments) in the range, final status is `OK` * if there are some missing segments found, final status is `LOST_SEGMENTS` -``` +```bash wal-g wal-show ``` @@ -322,7 +322,7 @@ By default, `wal-show` shows available backups for each timeline. To turn it off By default, `wal-show` output is plaintext table. For detailed JSON output, add the `--detailed-json` flag. -* ``wal-verify`` +### ``wal-verify`` Run series of checks to ensure that WAL segment storage is healthy. Available checks: @@ -359,7 +359,7 @@ Output consists of: 3. The highest timeline id found in WAL storage folder. Usage: -``` bash +```bash wal-g wal-verify [space separated list of checks] # For example: wal-g wal-verify integrity timeline # perform integrity and timeline checks @@ -369,7 +369,7 @@ wal-g wal-verify integrity # perform only integrity check By default, `wal-verify` output is plaintext. To enable JSON output, add the `--json` flag. Example of the plaintext output: -``` +```bash [wal-verify] integrity check status: OK [wal-verify] integrity check details: +-----+--------------------------+--------------------------+----------------+--------+ @@ -385,7 +385,7 @@ Current cluster timeline: 4 ``` Example of the JSON output: -``` +```bash { "integrity":{ "status":"OK", @@ -416,26 +416,26 @@ Example of the JSON output: } ``` -* ``wal-receive`` +### ``wal-receive`` Set environment variabe WALG_SLOTNAME to define the slot to be used (defaults to walg). The slot name can only consist of the following characters: [0-9A-Za-z_]. When uploading WAL archives to S3, the user should pass in the absolute path to where the archive is located. -``` +```bash wal-g wal-receive ``` -* ``backup-mark`` +### ``backup-mark`` Backups can be marked as permanent to prevent them from being removed when running ``delete``. Backup permanence can be altered via this command by passing in the name of the backup (retrievable via `wal-g backup-list --pretty --detail --json`), which will mark the named backup and all previous related backups as permanent. The reverse is also possible by providing the `-i` flag. -``` +```bash wal-g backup-mark example-backup -i ``` -* ``catchup-push`` +### ``catchup-push`` To create an catchup incremental backup, the user should pass the path to the master Postgres directory and the LSN of the replica for which the backup is created. @@ -450,7 +450,7 @@ wal-g catchup-push /path/to/master/postgres --from-lsn replica_lsn ``` -* ``catchup-fetch`` +### ``catchup-fetch`` To accept catchup incremental backup created by `catchup-push`, the user should pass the path to the replica Postgres directory and name of the backup. @@ -460,7 +460,7 @@ wal-g catchup-fetch /path/to/replica/postgres backup_name ``` -* ``copy`` +### ``copy`` This command will help to change the storage and move the set of backups there or write the backups on magnetic tape. For example, `wal-g copy --from=config_from.json --to=config_to.json` will copy all backups. diff --git a/README.md b/docs/README.md similarity index 86% rename from README.md rename to docs/README.md index 06106fb32..891934f59 100644 --- a/README.md +++ b/docs/README.md @@ -3,7 +3,9 @@ ![Unit-tests-status](https://github.com/wal-g/wal-g/workflows/Unit%20tests/badge.svg) [![Go Report Card](https://goreportcard.com/badge/github.com/wal-g/wal-g)](https://goreportcard.com/report/github.com/wal-g/wal-g) -WAL-G is an archival restoration tool for Postgres(beta for MySQL, MariaDB, MongoDB and SQLServer) +[Full documentation can be found here](https://wal-g.readthedocs.io) + +WAL-G is an archival restoration tool for Postgres(beta for MySQL, MariaDB, and MongoDB) WAL-G is the successor of WAL-E with a number of key differences. WAL-G uses LZ4, LZMA, or Brotli compression, multiple processors, and non-exclusive base backups for Postgres. More information on the design and implementation of WAL-G can be found on the Citus Data blog post ["Introducing WAL-G by Citus: Faster Disaster Recovery for Postgres"](https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-faster-restores-for-postgres/). @@ -13,9 +15,9 @@ WAL-G is the successor of WAL-E with a number of key differences. WAL-G uses LZ4 - [Usage](#usage) - [Databases](#databases) - [Development](#development) - - [Installing](#installing) - - [Testing](#testing) - - [Development on windows](#development-on-windows) + - [Installing](#installing) + - [Testing](#testing) + - [Development on windows](#development-on-windows) - [Authors](#authors) - [License](#license) - [Acknowledgements](#acknowledgements) @@ -27,7 +29,7 @@ A precompiled binary for Linux AMD 64 of the latest version of WAL-G can be obta To decompress the binary, use: -``` +```plaintext tar -zxvf wal-g.linux-amd64.tar.gz mv wal-g /usr/local/bin/ ``` @@ -40,21 +42,24 @@ Configuration To connect to Amazon S3, WAL-G requires that this variable be set: -* `WALG_S3_PREFIX` (e.g. `s3://bucket/path/to/folder`) (alternative form `WALE_S3_PREFIX`) +### `WALG_S3_PREFIX` +(e.g. `s3://bucket/path/to/folder`) (alternative form `WALE_S3_PREFIX`) WAL-G determines AWS credentials [like other AWS tools](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#config-settings-and-precedence). You can set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (optionally with `AWS_SESSION_TOKEN`), or `~/.aws/credentials` (optionally with `AWS_PROFILE`), or you can set nothing to fetch credentials from the EC2 metadata service automatically. To store backups in Google Cloud Storage, WAL-G requires that this variable be set: -* `WALG_GS_PREFIX` to specify where to store backups (e.g. `gs://x4m-test-bucket/walg-folder`) +### `WALG_GS_PREFIX` +to specify where to store backups (e.g. `gs://x4m-test-bucket/walg-folder`) WAL-G determines Google Cloud credentials using [application-default credentials](https://cloud.google.com/docs/authentication/production) like other GCP tools. You can set `GOOGLE_APPLICATION_CREDENTIALS` to point to a service account json key from GCP. If you set nothing, WAL-G will attempt to fetch credentials from the GCE/GKE metadata service. To store backups in Azure Storage, WAL-G requires that this variable be set: -* `WALG_AZ_PREFIX` to specify where to store backups in Azure storage (e.g. `azure://test-container/walg-folder`) +### `WALG_AZ_PREFIX` +to specify where to store backups in Azure storage (e.g. `azure://test-container/walg-folder`) WAL-G determines Azure Storage credentials using [azure default credentials](https://docs.microsoft.com/en-us/azure/storage/common/storage-azure-cli#azure-cli-sample-script). You can set `AZURE_STORAGE_ACCOUNT`, `AZURE_STORAGE_ACCESS_KEY` to provide azure storage credentials. @@ -67,14 +72,16 @@ WAL-G sets default upload buffer size to 64 Megabytes and uses 3 buffers by defa To store backups in Swift object storage, WAL-G requires that this variable be set: -* `WALG_SWIFT_PREFIX` to specify where to store backups in Swift object storage (e.g. `swift://test-container/walg-folder`) +### `WALG_SWIFT_PREFIX` +to specify where to store backups in Swift object storage (e.g. `swift://test-container/walg-folder`) WAL-G determines Swift object storage credentials using [openStack default credentials](https://www.swiftstack.com/docs/cookbooks/swift_usage/auth.html). You can use any of V1, V2, V3 of the SwiftStack Auth middleware to provide Swift object storage credentials. To store backups on files system, WAL-G requires that these variables be set: -* `WALG_FILE_PREFIX` (e.g. `/tmp/wal-g-test-data`) +### `WALG_FILE_PREFIX` +(e.g. `/tmp/wal-g-test-data`) Please, keep in mind that by default storing backups on disk along with database is not safe. Do not use it as a disaster recovery plan. @@ -86,49 +93,54 @@ To store backups via ssh, WAL-G requires that these variables be set: **Optional variables** -* `AWS_REGION`(e.g. `us-west-2`) +### `AWS_REGION` +(e.g. `us-west-2`) WAL-G can automatically determine the S3 bucket's region using `s3:GetBucketLocation`, but if you wish to avoid this API call or forbid it from the applicable IAM policy, specify this variable. -* `AWS_ENDPOINT` +### `AWS_ENDPOINT` Overrides the default hostname to connect to an S3-compatible service. i.e, `http://s3-like-service:9000` -* `AWS_S3_FORCE_PATH_STYLE` +### `AWS_S3_FORCE_PATH_STYLE` To enable path-style addressing (i.e., `http://s3.amazonaws.com/BUCKET/KEY`) when connecting to an S3-compatible service that lack of support for sub-domain style bucket URLs (i.e., `http://BUCKET.s3.amazonaws.com/KEY`). Defaults to `false`. -* `GCS_CONTEXT_TIMEOUT` +### `GCS_CONTEXT_TIMEOUT` Default: 1 hour. -* `GCS_NORMALIZE_PREFIX` +### `GCS_NORMALIZE_PREFIX` Controls the trimming of extra slashes in paths. The default is `true`. To allow restoring from WAL-E archives on GCS, set it to `false` and keep double slashes in `WALG_GS_PREFIX` values. -* `GCS_ENCRYPTION_KEY` +### `GCS_ENCRYPTION_KEY` To configure GCS Customer Supplied Encryption Key (CSEK) for client-side encryption and decryption. By default, Google-managed keys are used. CSEK must be a 32-byte AES-256 key, encoded in standard Base64. -* `GCS_MAX_CHUNK_SIZE` (e.g. `16777216`) +### `GCS_MAX_CHUNK_SIZE` +(e.g. `16777216`) Overrides the default `maximum chunk size` of 52428800 bytes (50 MiB). The size of the chunk must be specified in bytes. This parameter could be useful for different types of uploading (e.g. `16777216` (16MiB) would be perfect for `wal-push`, `52428800` (50MiB) is suitable for `backup-push`). -* `GCS_MAX_RETRIES` (e.g. `1`) +### `GCS_MAX_RETRIES` +(e.g. `1`) Overrides the default upload and download retry limit while interacting with GCS. Default: 16. -* `WALG_AZURE_BUFFER_SIZE` (e.g. `33554432`) +### `WALG_AZURE_BUFFER_SIZE` +(e.g. `33554432`) Overrides the default `upload buffer size` of 67108864 bytes (64 MB). Note that the size of the buffer must be specified in bytes. Therefore, to use 32 MB sized buffers, this variable should be set to 33554432 bytes. -* `WALG_AZURE_MAX_BUFFERS` (e.g. `5`) +### `WALG_AZURE_MAX_BUFFERS` +(e.g. `5`) Overrides the default `maximum number of upload buffers`. By default, at most 3 buffers are used concurrently. ***Example: Using Minio.io S3-compatible storage*** -``` +```bash AWS_ACCESS_KEY_ID: "" AWS_SECRET_ACCESS_KEY: "" WALG_S3_PREFIX: "s3://my-minio-bucket/sub-dir" @@ -138,42 +150,42 @@ AWS_REGION: us-east-1 WALG_S3_CA_CERT_FILE: "/path/to/custom/ca/file" ``` -* `WALG_S3_STORAGE_CLASS` +### `WALG_S3_STORAGE_CLASS` To configure the S3 storage class used for backup files, use `WALG_S3_STORAGE_CLASS`. By default, WAL-G uses the "STANDARD" storage class. Other supported values include "STANDARD_IA" for Infrequent Access and "REDUCED_REDUNDANCY" for Reduced Redundancy. -* `WALG_S3_SSE` +### `WALG_S3_SSE` To enable S3 server-side encryption, set to the algorithm to use when storing the objects in S3 (i.e., `AES256`, `aws:kms`). -* `WALG_S3_SSE_KMS_ID` +### `WALG_S3_SSE_KMS_ID` If using S3 server-side encryption with `aws:kms`, the KMS Key ID to use for object encryption. -* `WALG_CSE_KMS_ID` +### `WALG_CSE_KMS_ID` To configure AWS KMS key for client-side encryption and decryption. By default, no encryption is used. (AWS_REGION or WALG_CSE_KMS_REGION required to be set when using AWS KMS key client-side encryption) -* `WALG_CSE_KMS_REGION` +### `WALG_CSE_KMS_REGION` To configure AWS KMS key region for client-side encryption and decryption (i.e., `eu-west-1`). -* `WALG_COMPRESSION_METHOD` +### `WALG_COMPRESSION_METHOD` To configure the compression method used for backups. Possible options are: `lz4`, 'lzma', 'brotli'. The default method is `lz4`. LZ4 is the fastest method, but the compression ratio is bad. LZMA is way much slower. However, it compresses backups about 6 times better than LZ4. Brotli is a good trade-off between speed and compression ratio, which is about 3 times better than LZ4. -* `S3_USE_LIST_OBJECTS_V1` +### `S3_USE_LIST_OBJECTS_V1` By default, WAL-G uses [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) to fetch S3 storage folder listings. However, some S3-compatible storages may not support it. Set this setting to `true` to use [ListObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html) instead. -* `YC_CSE_KMS_KEY_ID` +### `YC_CSE_KMS_KEY_ID` To configure Yandex Cloud KMS key for client-side encryption and decryption. By default, no encryption is used. -* `YC_SERVICE_ACCOUNT_KEY_FILE` +### `YC_SERVICE_ACCOUNT_KEY_FILE` To configure the name of a file containing private key of Yandex Cloud Service Account. If not set a token from the metadata service (http://169.254.169.254) will be used to make API calls to Yandex Cloud KMS. @@ -184,7 +196,7 @@ Usage WAL-G currently supports these commands for all type of databases: -* ``backup-list`` +### ``backup-list`` Lists names and creation time of available backups. @@ -194,7 +206,7 @@ Lists names and creation time of available backups. ``--detail`` flag prints extra backup details, pretty-printed if combined with ``--pretty`` , json-encoded if combined with ``--json`` -* ``delete`` +### ``delete`` Is used to delete backups and WALs before them. By default ``delete`` will perform a dry run. If you want to execute deletion, you have to add ``--confirm`` flag at the end of the command. Backups marked as permanent will not be deleted. @@ -215,7 +227,7 @@ If `FIND_FULL` is specified, WAL-G will calculate minimum backup needed to keep (Only in Postgres) By default, if delta backup is provided as the target, WAL-G will also delete all the dependant delta backups. If `FIND_FULL` is specified, WAL-G will delete all backups with the same base backup as the target. -Examples: +### Examples ``everything`` all backups will be deleted (if there are no permanent backups) @@ -235,9 +247,9 @@ Examples: ``target base_0000000100000000000000C9`` delete the base backup and all dependant delta backups -`` target --target-user-data "{ \"x\": [3], \"y\": 4 }"`` delete backup specified by user data +`` target --target-user-data "{ \"x\": [3], \"y\": 4 }"`` delete backup specified by user data -``target base_0000000100000000000000C9_D_0000000100000000000000C4`` delete delta backup and all dependant delta backups +``target base_0000000100000000000000C9_D_0000000100000000000000C4`` delete delta backup and all dependant delta backups ``target FIND_FULL base_0000000100000000000000C9_D_0000000100000000000000C4`` delete delta backup and all delta backups with the same base backup @@ -246,19 +258,21 @@ Examples: Databases ----------- ### PostgreSQL -[Information about installing, configuration and usage](https://github.com/wal-g/wal-g/blob/master/PostgreSQL.md) +[Information about installing, configuration and usage](PostgreSQL.md) ### MySQL/MariaDB -[Information about installing, configuration and usage](https://github.com/wal-g/wal-g/blob/master/MySQL.md) +[Information about installing, configuration and usage](MySQL.md) ### SQLServer -[Information about installing, configuration and usage](https://github.com/wal-g/wal-g/blob/master/SQLServer.md) +[Information about installing, configuration and usage](SQLServer.md) ### Mongo -[Information about installing, configuration and usage](https://github.com/wal-g/wal-g/blob/master/MongoDB.md) +[Information about installing, configuration and usage](MongoDB.md) ### FoundationDB [Work in progress] -[Information about installing, configuration and usage](https://github.com/wal-g/wal-g/blob/master/FoundationDB.md) +[Information about installing, configuration and usage](FoundationDB.md) + + Development ----------- @@ -268,7 +282,7 @@ It is specified for your type of [database](#databases). ### Testing WAL-G relies heavily on unit tests. These tests do not require S3 configuration as the upload/download parts are tested using mocked objects. Unit tests can be run using -``` +```bash make unittest ``` For more information on testing, please consult [test](test), [testtools](testtools) and `unittest` section in [Makefile](Makefile). @@ -276,13 +290,13 @@ For more information on testing, please consult [test](test), [testtools](testto WAL-G will perform a round-trip compression/decompression test that generates a directory for data (e.g. data...), compressed files (e.g. compressed), and extracted files (e.g. extracted). These directories will only get cleaned up if the files in the original data directory match the files in the extracted one. Test coverage can be obtained using: -``` +```bash make coverage ``` This command generates `coverage.out` file and opens HTML representation of the coverage. ### Development on Windows -[Information about installing and usage](https://github.com/wal-g/wal-g/blob/master/Windows.md) +[Information about installing and usage](Windows.md) Authors diff --git a/SQLServer.md b/docs/SQLServer.md similarity index 70% rename from SQLServer.md rename to docs/SQLServer.md index 344d5d0b2..695b62171 100644 --- a/SQLServer.md +++ b/docs/SQLServer.md @@ -1,3 +1,5 @@ +# SQL Server + WAL-G facilitates SQLServer backups by emulating Azure Blob storage, allowing You to backup database to URL with standard BACKUP/RESTORE commands. @@ -17,50 +19,46 @@ So you need to: * create and import self-signed certificate for `backup.local` domain * create SQLServer login, able to backup/restore any desired database * create SQLServer credential for URL `https://backup.local/folder/` - ``` - CREATE CREDENTIAL [https://backup.local/folder] - WITH IDENTITY='SHARED ACCESS SIGNATURE', SECRET = 'does_not_matter' - ``` - +```bash +CREATE CREDENTIAL [https://backup.local/folder] +WITH IDENTITY='SHARED ACCESS SIGNATURE', SECRET = 'does_not_matter' +``` Configuration ------------- Your wal-g.yaml config for SQLServer than may look like - - ``` - WALG_FILE_PREFIX: "C:/backup" - SQLSERVER_BLOB_CERT_FILE: "C:/Path/To/cert.pem" - SQLSERVER_BLOB_KEY_FILE: "C:/Path/To/key.pem" - SQLSERVER_BLOB_LOCK_FILE: "C:/ProgramData/wal-g.lock" - SQLSERVER_BLOB_HOSTNAME: "backup.local" - SQLSERVER_CONNECTION_STRING: "sqlserver://backupuser:backuppass1!@localhost:1433/instance" - - WALG_UPLOAD_CONCURRENCY: 8 # how many block upload requests handle concurrently - WALG_DOWNLOAD_CONCURRENCY: 8 # how many block read requests handle concurrently - ``` +```bash +WALG_FILE_PREFIX: "C:/backup" +SQLSERVER_BLOB_CERT_FILE: "C:/Path/To/cert.pem" +SQLSERVER_BLOB_KEY_FILE: "C:/Path/To/key.pem" +SQLSERVER_BLOB_LOCK_FILE: "C:/ProgramData/wal-g.lock" +SQLSERVER_BLOB_HOSTNAME: "backup.local" +SQLSERVER_CONNECTION_STRING: "sqlserver://backupuser:backuppass1!@localhost:1433/instance" + +WALG_UPLOAD_CONCURRENCY: 8 # how many block upload requests handle concurrently +WALG_DOWNLOAD_CONCURRENCY: 8 # how many block read requests handle concurrently +``` Of course, you may use any wal-g storage instead of FILE You also need some configuration in SQLServer for wal-g to connect it. - - ``` - CREATE LOGIN [backupuser] WITH PASSWORD = 'backuppass1!'; - ALTER SERVER ROLE [sysadmin] ADD MEMBER [backupuser]; - CREATE CREDENTIAL [https://backup.local/basebackups_005] - WITH IDENTITY='SHARED ACCESS SIGNATURE', SECRET = 'does_not_matter'; - CREATE CREDENTIAL [https://backup.local/wal_005] - WITH IDENTITY='SHARED ACCESS SIGNATURE', SECRET = 'does_not_matter'; - ``` +```bash +CREATE LOGIN [backupuser] WITH PASSWORD = 'backuppass1!'; +ALTER SERVER ROLE [sysadmin] ADD MEMBER [backupuser]; +CREATE CREDENTIAL [https://backup.local/basebackups_005] +WITH IDENTITY='SHARED ACCESS SIGNATURE', SECRET = 'does_not_matter'; +CREATE CREDENTIAL [https://backup.local/wal_005] +WITH IDENTITY='SHARED ACCESS SIGNATURE', SECRET = 'does_not_matter'; +``` S3 Configuration ------------- SQLServer backups/restores database by 4MB blocks (MAXTRANSFERSIZE). As we upload every block as a separate file to S3, it makes sense to set S3 part size to the smalles possible value (5MB) to prevent overuse of memory: - - ``` - WALG_S3_MAX_PART_SIZE: 5242880 # 5MB - ``` +```bash +WALG_S3_MAX_PART_SIZE: 5242880 # 5MB +``` Usage @@ -68,9 +66,9 @@ Usage WAL-G SQLServer extension currently supports these commands: -* ``proxy`` +### ``proxy`` -``` +```bash wal-g proxy ``` @@ -80,9 +78,9 @@ While proxy is running, you are able to any BACKUP/RESTORE TO/FROM URL commands Proxy intended for debug/manual backups only, it rely on user to maintain proper backups folder structure. For simple backup/restore process please consider using `backup-push` and `backup-restore` commands. -* ``backup-push`` +### ``backup-push`` -``` +```bash wal-g backup-push wal-g backup-push -d db1 -d db2 wal-g backup-push -d ALL @@ -93,9 +91,9 @@ You can specify which databases to backup via `-d` flag. You can backup all (including system) databases using `-d ALL` flag. By default it will backup all non-system databases. -* ``backup-restore`` +### ``backup-restore`` -``` +```bash wal-g backup-restore backup_name wal-g backup-restore LATEST wal-g backup-restore backup_name -d db1 @@ -111,15 +109,15 @@ You can restore database with new name (create copy of database) using flag `-f` By default it will restore all non-system databases found in backup. -* ``backup-list`` +### ``backup-list`` -``` +```bash wal-g backup-list ``` -* ``delete`` +### ``delete`` -``` +```bash wal-g delete retain 3 wal-g delete before backup_name wal-g delete everything diff --git a/Windows.md b/docs/Windows.md similarity index 87% rename from Windows.md rename to docs/Windows.md index 0a4012422..50fc04d92 100644 --- a/Windows.md +++ b/docs/Windows.md @@ -1,6 +1,8 @@ -## Building Windows executable on Linux +# WAL-G for Windows -See here for available GOOS and GOARCH values. +Building Windows executable on Linux + +## See here for available GOOS and GOARCH values. ----------- Go version >= 1.5 @@ -15,8 +17,6 @@ Since Go version 1.5 cross-compiling of pure Go executables has become very easy * mv main/sqlserver/wal-g{,.exe} -## WAL-G for Windows - Development on Windows ----------- ### Installing diff --git a/docs/requirements.txt b/docs/requirements.txt new file mode 100644 index 000000000..da389ea0f --- /dev/null +++ b/docs/requirements.txt @@ -0,0 +1,2 @@ +# requirements.txt +mkdocs==1.1.2 \ No newline at end of file diff --git a/resources/wal_verify_segment_statuses.png b/docs/resources/wal_verify_segment_statuses.png similarity index 100% rename from resources/wal_verify_segment_statuses.png rename to docs/resources/wal_verify_segment_statuses.png diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 000000000..4f94116a2 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,10 @@ +site_name: WAL-G +nav: + - Overview: README.md + - Foundation DB: FoundationDB.md + - Mongo DB: MongoDB.md + - MySQL: MySQL.md + - PostgreSQL: PostgreSQL.md + - SQLServer: SQLServer.md + - Windows: Windows.md +theme: readthedocs