tuftool: Add `clone` subcommand #404

zmrow · 2021-08-13T22:27:27Z

Issue #, if available:
Fixes #401

Description of changes:
This PR does a small amount of refactoring before the main event, a new clone subcommand! This subcommand gives the user the opportunity to clone a repository and it's metadata, including all, none or some of the targets. It can be considered a bit more "low-level" than tuftool download as the files end up on disk identically named as they are in the TUF repository, including the SHA256 prefix. tuftool download also does not give the user the opportunity to download metadata.

Reading through by commit is probably easiest as they're fairly small.

    tuftool: Add `download_root` module
    
    This change adds a `download_root` module, which abstracts the logic of
    downloading `root.json` out of `download` (the subcommand).  It also
    makes the HTTP request a bit more robust by handling the initial request
    error as well as a bad response.  Additionally, it avoids creating the
    file until after the request has succeeded to avoid creating cruft on
    the user's system.

    tuftool: use `download_root` module in `download`
    
    This change updates the `download` module/subcommand to make use of the
    previously added `download_root` function.
    
    It also defines a default of "1" for the `root_version` argument.
    Previously, we effectively had this default in code by using
    `1.root.json` in the event the argument wasn't passed.  It also has the
    nice side effect of not needing to deal with an `Option` for this
    argument

    tuftool: Add `clone` subcommand
    
    This adds a `clone` subcommand to `tuftool`, allowing a user to download
    a fully functioning TUF repository.  A user has the option to download a
    full repository, a subset of the targets, or just metadata

Testing:

All new/old unit tests pass
Manual testing:
- Download just a repository's metadata (with root.json and with the flag --allow-root-download)

$ cargo run --bin tuftool clone --metadata-only --metadata-url "https://updates.bottlerocket.aws/2020-07-07/aws-k8s-1.20/x86_64/" --allow-root-download --metadata-dir ./foo/metadata

# also ran with a downloaded root.json
$ curl -O "https://cache.bottlerocket.aws/root.json
$ cargo run --bin tuftool clone --metadata-only --metadata-url "https://updates.bottlerocket.aws/2020-07-07/aws-k8s-1.20/x86_64/" --root ./root.json --metadata-dir ./foo/metadata

 $ tree ./foo/
./foo/
└── metadata
    ├── 1628878016.snapshot.json
    ├── 1628878016.targets.json
    ├── 1.root.json
    ├── 2.root.json
    └── timestamp.json

Download an entire repository to ./foo (with root.json and with the flag --allow-root-download)

$ cargo run --bin tuftool clone --metadata-url "https://updates.bottlerocket.aws/2020-07-07/aws-k8s-1.20/x86_64/" --targets-url "https://updates.bottlerocket.aws/targets/" --allow-root-download --metadata-dir ./foo/metadata --targets-dir ./foo/targets

# also ran with a downloaded root.json
$ curl -O "https://cache.bottlerocket.aws/root.json
$ cargo run --bin tuftool clone --metadata-url "https://updates.bottlerocket.aws/2020-07-07/aws-k8s-1.20/x86_64/" --targets-url "https://updates.bottlerocket.aws/targets/" --root ./root.json --metadata-dir ./foo/metadata --targets-dir ./foo/targets

$ tree ./foo
 ./foo/                                                                                                                                            
├── metadata                                                                                                                                      
│   ├── 1628878016.snapshot.json                                                                                                                  
│   ├── 1628878016.targets.json                                                                                                                   
│   ├── 1.root.json                                                                                                                               
│   ├── 2.root.json                                                                                                                               
│   └── timestamp.json                                                                                                                            
└── targets                                                                                                                                       
    ├── 0c8485912fd928a185f32a74705f8e5efa692177375c27eac7ee5c11219d259e.migrate_v1.1.0_kubelet-registry-qps-registry-burst.lz4
    ├── 148f1c10a790313c19e4967c0136c58cf433c20a41676bffb5f3c976a1170d4a.bottlerocket-aws-k8s-1.20-x86_64-1.1.1-28c4e013-root.verity.lz4
...

Download a subset of targets (just manifest.json) (with root.json and with the flag --allow-root-download)

$ cargo run --bin tuftool clone --metadata-url "https://updates.bottlerocket.aws/2020-07-07/aws-k8s-1.20/x86_64/" --targets-url "https://updates.bottlerocket.aws/targets/" --allow-root-download --metadata-dir ./foo/metadata --targets-dir ./foo/targets --target-names manifest.json

# also ran with a downloaded root.json
$ curl -O "https://cache.bottlerocket.aws/root.json
$ cargo run --bin tuftool clone --metadata-url "https://updates.bottlerocket.aws/2020-07-07/aws-k8s-1.20/x86_64/" --targets-url "https://updates.bottlerocket.aws/targets/" --root ./root.json --metadata-dir ./foo/metadata --targets-dir ./foo/targets --target-names manifest.json

$ tree ./foo
./foo/
├── metadata
│   ├── 1628878016.snapshot.json
│   ├── 1628878016.targets.json
│   ├── 1.root.json
│   ├── 2.root.json
│   └── timestamp.json
└── targets
    └── aabc9db0c44ff6811aaf0f847c3bba6830e16bb0ecb6d2ec5eef837f68440cd2.manifest.json

Ensure that targets arguments collide with the --metadata-only flag

$ cargo run --bin tuftool clone --metadata-only --metadata-url "https://updates.bottlerocket.aws/2020-07-07/aws-k8s-1.20/x86_64/" --allow-root-download --metadata-dir ./foo/metadata --target-names foo
...
error: The argument '--metadata-only' cannot be used with '--target-names <target-names>...'

USAGE:
    tuftool clone --allow-root-download --metadata-url <metadata-base-url> --metadata-dir <metadata-dir> --metadata-only --root-version <root-version> --target-names <target-names>... --targets-url <targets-base-url> --targets-dir <targets-dir>
----------------------------------
$ cargo run --bin tuftool clone --metadata-only --metadata-url "https://updates.bottlerocket.aws/2020-07-07/aws-k8s-1.20/x86_64/" --allow-root-download --metadata-dir ./foo/metadata --targets-dir foo
...
error: The argument '--targets-dir <targets-dir>' cannot be used with '--metadata-only'

USAGE:
    tuftool clone --allow-root-download --metadata-url <metadata-base-url> --metadata-dir <metadata-dir> --metadata-only --root-version <root-version> --targets-url <targets-base-url> --targets-dir <targets-dir>

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

zmrow · 2021-08-16T15:06:40Z

^ Fix the clippy complaint - I didn't see it when I ran the make build command locally...

webern

Nice testing.

tuftool/src/clone.rs

tuftool/src/error.rs

tuftool/src/download_root.rs

tuftool/src/main.rs

tuftool/tests/clone_command.rs

tjkirch · 2021-08-16T17:37:17Z

tuftool/tests/clone_command.rs

+fn clone_base_command<'a>(cmd: &'a mut Command, repo_paths: &RepoPaths) -> &'a mut Command {
+    cmd.args(&[
+        "clone",
+        "--root",


Nice tests! Should we have one for --allow-root-download as well?

Good point - I'll add one.

:( Bad news everyone! We use reqwest to get the root.json, and reqwest doesn't support file:/// URLs.

I think what we normally do is add a branch, where if url.scheme() is file, we just std::fs::copy it instead.

Opened a new issue to track this: #406

tuftool/src/clone.rs

This change adds a `download_root` module, which abstracts the logic of downloading `root.json` out of `download` (the subcommand). It also makes the HTTP request a bit more robust by handling the initial request error as well as a bad response. Additionally, it avoids creating the file until after the request has succeeded to avoid creating cruft on the user's system.

This change updates the `download` module/subcommand to make use of the previously added `download_root` function. It also defines a default of "1" for the `root_version` argument. Previously, we effectively had this default in code by using `1.root.json` in the event the argument wasn't passed. It also has the nice side effect of not needing to deal with an `Option` for this argument.

zmrow · 2021-08-17T16:07:15Z

^ addressed all the comments

This adds a `clone` subcommand to `tuftool`, allowing a user to download a fully functioning TUF repository. A user has the option to download a full repository, a subset of the targets, or just metadata.

zmrow · 2021-08-17T16:08:08Z

^ removed a dangling println!()

zmrow requested review from bcressey, webern and tjkirch August 13, 2021 22:27

zmrow force-pushed the clone-subcommand branch from c277a55 to 4257c49 Compare August 16, 2021 15:06

webern approved these changes Aug 16, 2021

View reviewed changes

tuftool/src/clone.rs Outdated Show resolved Hide resolved

tuftool/src/error.rs Outdated Show resolved Hide resolved

tjkirch reviewed Aug 16, 2021

View reviewed changes

zmrow added 2 commits August 17, 2021 15:53

zmrow force-pushed the clone-subcommand branch from 4257c49 to 37b3596 Compare August 17, 2021 16:05

tuftool: Add clone subcommand

e62cfee

This adds a `clone` subcommand to `tuftool`, allowing a user to download a fully functioning TUF repository. A user has the option to download a full repository, a subset of the targets, or just metadata.

zmrow force-pushed the clone-subcommand branch from 37b3596 to e62cfee Compare August 17, 2021 16:07

tjkirch approved these changes Aug 17, 2021

View reviewed changes

webern approved these changes Aug 17, 2021

View reviewed changes

zmrow mentioned this pull request Aug 18, 2021

tuftool: download_root() function should handle file:/// URLs #406

Open

zmrow merged commit 1d223a1 into awslabs:develop Aug 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tuftool: Add `clone` subcommand #404

tuftool: Add `clone` subcommand #404

zmrow commented Aug 13, 2021 •

edited

Loading

zmrow commented Aug 16, 2021

webern left a comment

tjkirch Aug 16, 2021

zmrow Aug 16, 2021

zmrow Aug 17, 2021

tjkirch Aug 17, 2021

zmrow Aug 18, 2021

zmrow commented Aug 17, 2021

zmrow commented Aug 17, 2021

tuftool: Add clone subcommand #404

tuftool: Add clone subcommand #404

Conversation

zmrow commented Aug 13, 2021 • edited Loading

zmrow commented Aug 16, 2021

webern left a comment

Choose a reason for hiding this comment

tjkirch Aug 16, 2021

Choose a reason for hiding this comment

zmrow Aug 16, 2021

Choose a reason for hiding this comment

zmrow Aug 17, 2021

Choose a reason for hiding this comment

tjkirch Aug 17, 2021

Choose a reason for hiding this comment

zmrow Aug 18, 2021

Choose a reason for hiding this comment

zmrow commented Aug 17, 2021

zmrow commented Aug 17, 2021

tuftool: Add `clone` subcommand #404

tuftool: Add `clone` subcommand #404

zmrow commented Aug 13, 2021 •

edited

Loading