First Backup Sync Implementation #115

fboucquez · 2021-01-14T03:33:07Z

Added --backupSync to config command. This command downloads a zip file of the mongo and rocksdb databases from a known trusted HTTP location (in this case S3 but it could be somewhere else).

The zip file is locally cached. --backupSync only redownloads if there is a new remove backup (by checking local and remote file sizes).

The first example of the backup is:

https://symbol-bootstrap.s3-eu-west-1.amazonaws.com/testnet/testnet-partial-backup.zip

@Jaguar0625 @gimer could you validate the zip file checking if there is any node related information that should not be there?

Note that I need both mongo and rockdb backups as mongo cannot be recreated from the rockdb database.
https://github.com/nemtech/catapult-server/issues/123

This backup only has 25k height of testnet. Once bootstrap start, it starts synchronizing from that height.
Eventually, we will have a full-size backup of the current network. The current testnet db size is 35GBs, too large for this initial testing.

To create a backup (to be automatized by NGL):

Have a testnet fully sync running
Stop the server/bootstrap
Run a script/command to create the zip file
Push the ZIP into S3 keeping the same s3 file path
Start bootstrap

Then, when a new bootstrap node starts with --backupSync, it will download this backup starting the node from the backup height and not from 0.

A very basic script to create a backup can be found in the backup-sync-testnet-backup.sh

I'll convert this script to a bootstrap command for easier use.

Fixes #70

segfaultxavi

Minor grammar fix

src/commands/config.ts

docs/config.md

docs/start.md

rg911

Does the state of the backup needs validated?

presets/testnet/network.yml

fboucquez · 2021-01-17T11:53:01Z

There are 2 possibilities of the backup-sync feature:

Full Sync Backup

RocksDB and Mongo databases are backed up and restored.

Pros:

It works!
There is no need to call the server's recovery
Once downloaded faster to restore, less CPU intensive.

Const:

Backup zip file is around 50% bigger.
Slower to upload and download
It requires more disk space for backup management.

I've added the symbol-bootstrap backup that generates a backup zip from node after stoping it. Then the zip can be uploaded to the public s3 location. Backup, at least for testnet/public net should only be called by NGL.

Numbers:

35 GB the size of testenet target folder
21 GB the size of the Full Sync Backup is https://symbol-bootstrap.s3-eu-west-1.amazonaws.com/testnet/backup.zip
10 minutes to download the current full backup (s3 to EC2 box in Ireland, hours if I want to download down here)
4 minutes to unzip the RockDB backup
3 minutes to unzip the Mongo database.
20 minutes for a full resync!!!

My restored node http://54.155.40.57:3000/chain/info is full synced, up and running

If you want to use the --backupSync feature. Your node would need at least 80 GB for the current testnet size. 21GB to download the zip file, 35 GB for the generated target folder, and some space just in case. Note that once the node's restored, the backup zip file could be removed to reclaim the 21GB space for the future node use.

Partial Sync Backup

Only RocksDB database is backed up and restored. Mongo is regenerated locally from the restored RocksDB

Pros:

Smaller zip file
Faster to upload and download.
Less space required for backup management

Const:

It doesn't work yet!!!
It requires to call the server recovery command, I'm guessing it can take time and CPU for full restore/recovery of the mongo DB

The question would be if we should keep trying to make Partial Sync Backup to work or if the Full Sync Backup is good enough / the best, at least for a first version.

CHANGELOG.md

segfaultxavi · 2021-01-19T08:15:30Z

README.md

@@ -264,6 +264,7 @@ General users should install this tool like any other node module.
 <!-- commands -->
 # Command Topics

+* [`symbol-bootstrap backup`](docs/backup.md) - The command backups the Mongo and RocksDb data folder into a Zip file that can be used for --backupSync feature. Bootstrap compose services must be stopped before calling this command.


Suggested change

* [`symbol-bootstrap backup`](docs/backup.md) - The command backups the Mongo and RocksDb data folder into a Zip file that can be used for --backupSync feature. Bootstrap compose services must be stopped before calling this command.

* [`symbol-bootstrap backup`](docs/backup.md) - The command backs up the Mongo and RocksDb data folder into a Zip file that can then be used by the `--backupSync` feature. Bootstrap compose services must be stopped before calling this command.

segfaultxavi · 2021-01-19T08:16:36Z

docs/backup.md

+`symbol-bootstrap backup`
+=========================
+
+The command backups the Mongo and RocksDb data folder into a Zip file that can be used for --backupSync feature. Bootstrap compose services must be stopped before calling this command.


Suggested change

The command backups the Mongo and RocksDb data folder into a Zip file that can be used for --backupSync feature. Bootstrap compose services must be stopped before calling this command.

The command backs up the Mongo and RocksDb data folder into a Zip file that can then be used by the `--backupSync` feature. Bootstrap compose services must be stopped before calling this command.

segfaultxavi · 2021-01-19T08:18:17Z

docs/backup.md

+
+The command backups the Mongo and RocksDb data folder into a Zip file that can be used for --backupSync feature. Bootstrap compose services must be stopped before calling this command.
+
+Note: this command is designed for NGL to be used when running public main or public test networks. It's not backing up any node specific information.


Is this true? Couldn't it be used by a private network to speed up sync?

Yes, it can be used. Atm we don't have a well-documented process of a user joining a private network with bootstrap. Pieces are there, I just need to put them together. Main issue is how to easily share a private network preset and seed in new private nodes

It's just a warning for the current users that may want to use it on testnet backups. This backup is not backing a node (like's node configuration or keys), it's just the data that can be shared between multiples nodes

docs/backup.md

docs/config.md

src/commands/backup.ts

src/service/BackupSyncService.ts

Added --backup-sync to config command. This command downloads a known location via s3. First example back is https://symbol-bootstrap.s3-eu-west-1.amazonaws.com/testnet/testnet-partial-backup.zip

fboucquez · 2021-03-01T19:31:46Z

Superseded by #151

fboucquez requested review from Wayonb and rg911 January 14, 2021 03:33

segfaultxavi suggested changes Jan 14, 2021

View reviewed changes

src/commands/config.ts Outdated Show resolved Hide resolved

docs/config.md Outdated Show resolved Hide resolved

docs/start.md Outdated Show resolved Hide resolved

rg911 reviewed Jan 14, 2021

View reviewed changes

presets/testnet/network.yml Outdated Show resolved Hide resolved

fboucquez changed the title ~~First Fast Sync Implementation~~ First Backup Sync Implementation Jan 14, 2021

fboucquez force-pushed the fast-sync branch from c4f2845 to 78dda66 Compare January 18, 2021 18:25

fboucquez requested review from rg911 and segfaultxavi January 18, 2021 18:43

fboucquez force-pushed the fast-sync branch from 78dda66 to 350a62d Compare January 18, 2021 18:44

segfaultxavi suggested changes Jan 19, 2021

View reviewed changes

fboucquez force-pushed the fast-sync branch from b25cb1b to ec64d1b Compare January 19, 2021 12:57

fboucquez force-pushed the fast-sync branch from d376965 to 91ffeeb Compare February 2, 2021 13:24

Backup Sync

4e1b584

Added --backup-sync to config command. This command downloads a known location via s3. First example back is https://symbol-bootstrap.s3-eu-west-1.amazonaws.com/testnet/testnet-partial-backup.zip

fboucquez force-pushed the fast-sync branch from 91ffeeb to 4e1b584 Compare February 3, 2021 11:57

fboucquez closed this Mar 1, 2021

fboucquez mentioned this pull request Mar 1, 2021

Backup sync using importer (Post-Launch) #151

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First Backup Sync Implementation #115

First Backup Sync Implementation #115

fboucquez commented Jan 14, 2021 •

edited

Loading

segfaultxavi left a comment

rg911 left a comment

fboucquez commented Jan 17, 2021 •

edited

Loading

segfaultxavi Jan 19, 2021

segfaultxavi Jan 19, 2021

segfaultxavi Jan 19, 2021

fboucquez Jan 19, 2021

fboucquez Jan 19, 2021

fboucquez commented Mar 1, 2021

	* [`symbol-bootstrap backup`](docs/backup.md) - The command backups the Mongo and RocksDb data folder into a Zip file that can be used for --backupSync feature. Bootstrap compose services must be stopped before calling this command.
	* [`symbol-bootstrap backup`](docs/backup.md) - The command backs up the Mongo and RocksDb data folder into a Zip file that can then be used by the `--backupSync` feature. Bootstrap compose services must be stopped before calling this command.

	The command backups the Mongo and RocksDb data folder into a Zip file that can be used for --backupSync feature. Bootstrap compose services must be stopped before calling this command.
	The command backs up the Mongo and RocksDb data folder into a Zip file that can then be used by the `--backupSync` feature. Bootstrap compose services must be stopped before calling this command.


		The command backups the Mongo and RocksDb data folder into a Zip file that can be used for --backupSync feature. Bootstrap compose services must be stopped before calling this command.

		Note: this command is designed for NGL to be used when running public main or public test networks. It's not backing up any node specific information.

First Backup Sync Implementation #115

First Backup Sync Implementation #115

Conversation

fboucquez commented Jan 14, 2021 • edited Loading

segfaultxavi left a comment

Choose a reason for hiding this comment

rg911 left a comment

Choose a reason for hiding this comment

fboucquez commented Jan 17, 2021 • edited Loading

Full Sync Backup

Numbers:

Partial Sync Backup

segfaultxavi Jan 19, 2021

Choose a reason for hiding this comment

segfaultxavi Jan 19, 2021

Choose a reason for hiding this comment

segfaultxavi Jan 19, 2021

Choose a reason for hiding this comment

fboucquez Jan 19, 2021

Choose a reason for hiding this comment

fboucquez Jan 19, 2021

Choose a reason for hiding this comment

fboucquez commented Mar 1, 2021

fboucquez commented Jan 14, 2021 •

edited

Loading

fboucquez commented Jan 17, 2021 •

edited

Loading