Add storage import and export features to command line and web API (E…

…pistimio#1082)

* Add files for scripts `orion db dump` and `orion db load`

* Write orion db dump

* Write orion db load for parameter --exp

* Simplify code to get an experiment from its name

* Wrap destination database into a storage.
Move dump logic into a function.

* dump:
- add and use CLI basic arguments
- use both version and name to find experiment to dump

* Rewrite load

* Hardcode collection list in dump

* Raise runtime error if resolve is 'bump' for benchmarks.

* - Add tests for orion db dump
- Raise DatabaseError when error occurs in the script
- If found many experiments for a given name, pick experiment with lower version

* - Add tests for orion db load
- raise DatabaseError when errors occur in th script
- If found many experiments for a given name, pick experiment with lower version

* Reformat code

* Move function dump_database() into new module orion.core.worker.storage_backup

* Add entry /dump to Web API.

* [web api] Add download suffix to dumped file

* Use one module for both import/export web API endpoints.

* [web api] Receive a POST request to import data

* Add function load_database into module storage_backup and move import logic into this function.

* Rename param `experiment` to `name` for function dump_database() to have same param names as load_database()

* Check conflicts before making import.

* [Web API]

Make load entry launch a process to manage import

Capture import progress messages printed by log

Add entry import-status to get import progress messages and status

* [web api] Allow to follow import progress using a callback in backend

* Add documentation for web API.

* Add documentation for command line.

* Add tests for web API /dump, /load and /import-status

* Fix tests.

* Move main functions to top of module storage_backup

* For dump from pickledb, use db object directly instead of locking database before (database is always locked before anything read/write/remove operation)

* Use storage instead of database for export.
- NB: With this new interface, dumping whole database is slower, because we must dump experiments one by one, making many more database calls.

Add heartbeat field to LockedAlgorithmState

Update BaseStorageProtocol interface:
- Allow to set initial algo state in create_experiment()
- Allow to set algo state in initialize_algorithm_lock(), and rename it to write_algorithm_lock()

Update test_db_dump
- WIth new dumping interface, only algos related to available experiments are dumped, dumped data does not contain algo related to unknown experiment anymore
- Check dumped algo state for an experiment

* Update doc in storage_resource
Update test for storage_resource

* For load (from pickledb file), use db object directly instead of locking database before (database is always locked before anything read/write/remove operation)

* Add benchmarks to tested databased for dump/load.

* Use storage instead of database to import.
BaseStorageProtocol:
- add new function delete_benchmark() and implement it in child class Legacy.

* Check progress callback messages in test_db_load:test_load_overwrite()

* Fix pylint

* Tru to set logging level in running import task

* Update docs/src/user/storage.rst

Co-authored-by: Xavier Bouthillier <[email protected]>

* Allow to not specify a resolve strategy. If not specified, an exception will be raised as soon as a conflict is detected.

* Just logging storage instead of `storage._db` in command lines `dump` and `load`.

* Storage export: add an option `--force` to explicitly overwrite dumped file if already exists.

* Import/export: if no specified, get latest instead of oldest version for specific experiment

* Use NamedTemporaryFile to generate temporary file in storage_resource.

* Rewrite docstring for class ImportTask in Numpy style.

* Fix a function name

* Rename test fixture used to test storage export.

* Add comment in test_dump_unknown_experiment() to explain why output file is created in any case.

* Remove unused logger in module storage_resource.

* Write generic checking functions for dump unit tests, that also verify expected number of children experiments or trials.

Discovering a corner case: imported/exported experiments keeps old `refers` links.
Currently remove refers links if only 1 specified experiment is dumped.

* Update TODO

* Regenerate experiment parent links in dst according to src when dumping, using correct dst IDs.

* Regenerate experiment parent links and experiment-to-trial links in dst according to src when loading, using correct dst IDs.

* Factorize tests in test_db_load and do not use prec-computed PKL files anymore (use pkl_* fixtures instead)

* Refactorize code for test_storage_resource.

* Use test_helpers for test_db_dump

* Remove useless tests/functional/commands/__init__.py
Add a test to check default storage content (we have more algos than expected)

* Check new data are ignored or indeed written when importing with ignore or overwrite resolution.

* Set and check trial links.

* Add tests/functional/conftest

* Regenerate links to root experiments in imported/exported experiments.

* Add common function to write experiment in a destination storage, to use for both dump and load features.

* Add a lock to ImportTask to prevent concurrent execution when updating task info.
Encapsulate ImportTask to make sure lock is used when necessary.

* Use Orion TreeNode to manage experiment and trial links.

* Try to fix CI failing tests. Tests seems to pass for python 3.8 but not for python 3.7.
- In python 3.8, we can clear previous logging handlers before setting new stream just by using new argument `force` in logging.basicConfig
- In python 3.7, we must clear previous handlers manually before setting new stream.

* Correctly update algorithm when setting deterministic experiment ID in tests/functional/commands/conftest

* Remove irrelevant calls to `logger.setLevel`

* Remove irrelevant calls to `logger.setLevel` that were added in this PR.

* [web api] make sure to transfer main root log level into import sub-process

* [test_storage_resource]
- Now that storage_resource logging depends on root logging, we must set client logging in unit tests to be sure expected logging messages are printed in import sub-process, using caplog.
- Using caplog, we can rewrite logging unit test using simulated client, instead of launching a real sub-process server.

* Clean dumped file if an error occurs when dumping, **except** if file already existed and *no* overwrite specified.

* Remove useless pylint in `orion.core.cli.db.load`

* Move module `orion.core.worker.storage_backup` into `orion.storage.backup`

* Remove fixed TODO

* Remove fixed TODO in test_db_load

* Clean dumped files only if no error occurs in storage_resource (files should have been deleted if an error occurred)

* Test dump using a temporary output path for most unit tests except `test_dump_default`

* - Work on temporary file when dumping and move it to output file only if no error occurred
- Move function _gen_host_file() from orion/serving/storage_resource to orion/core/utils and rename to generate_temporary_file().

* Use pytest fixture `tmp_path` instead of manually-created temp dir to test db dump.

---------

Co-authored-by: Xavier Bouthillier <[email protected]>
Co-authored-by: Setepenre <[email protected]>

docs/src/user/storage.rst

@@ -153,6 +153,46 @@ simply run the upgrade command.
 
 .. _storage_python_apis:
 
+``dump`` Export database content
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``dump`` command allows to export database content to a PickledDB PKL file.
+
+.. code-block:: sh
+
+   orion db dump -o backup.pkl
+
+You can also dump a specific experiment.
+
+.. code-block:: sh
+
+   orion db dump -n exp-name -v exp-version -o backup-exp.pkl
+
+``load`` Import database content
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``load`` command allows to import database content
+from any PickledDB PKL file (including files generated by ``dump`` command).
+
+You must specify a conflict resolution policy using ``-r/--resolve`` argument
+to apply when conflicts are detected during import. Available policies are:
+
+- ``ignore``, to ignore imported data
+- ``overwrite``, to replace old data with imported data
+- ``bump``, to bump version of imported data and then make import
+
+By default, whole PKL file will be imported.
+
+.. code-block:: sh
+
+   orion db load backup.pkl -r ignore
+
+You can also import a specific experiment.
+
+.. code-block:: sh
+
+   orion db load backup.pkl -r overwrite -n exp-name -v exp-version
+
 Python APIs
 ===========
 

docs/src/user/web_api.rst

@@ -393,7 +393,6 @@ visualize your experiments and their results.
 
    :statuscode 404: When the specified experiment doesn't exist in the database.
 
-
 Benchmarks
 ----------
 The benchmark resource permits the retrieval of in-progress and completed benchmarks. You can
@@ -487,6 +486,101 @@ retrieve individual benchmarks as well as a list of all your benchmarks.
                     or assessment, task or algorithms are not part of the existing benchmark
                     configuration.
 
+Database dumping
+----------------
+
+The database dumping resource allows to dump database content
+into a PickledDB and download it as PKL file.
+
+.. http:get:: /dump
+
+   Return a PKL file containing database content.
+
+   :query name: Optional name of experiment to export. It unspecified, whole database is dumped.
+   :query version: Optional version of the experiment to retrieve.
+      If unspecified and name is specified, the **latest** version of the experiment is exported.
+      If both name and version are unspecified, whole database is dumped.
+
+   :statuscode 404: When an error occurred during dumping.
+
+Database loading
+----------------
+
+The database loading resource allow to import data from a PKL file
+
+.. http:post:: /load
+
+   Import data into database from a PKL file.
+   This is a POST request, as a file must be uploaded.
+   Launch an import task in a separate process in backend and return task ID
+   which may be used to get task progress.
+
+   :query file: PKL file to import
+   :query resolve: policy to resolve conflicts during import. Either:
+
+    - ``ignore``: ignore imported data on conflict
+    - ``overwrite``: overwrite ancient data on conflict
+    - ``bump``: bump version of imported data before insertion on conflict
+
+   :query name: Optional name of experiment to import. If unspecified, whole data from PKL file is imported.
+   :query version: Optional version of experiment to import.
+      If unspecified and name is specified, the **latest** version of the experiment is imported.
+      If both name and version are unspceified, whole data from PKL file is imported.
+
+   **Example response**
+
+   .. sourcecode:: http
+
+      HTTP/1.1 200 OK
+      Content-Type: text/javascript
+
+   .. code-block:: json
+
+      {
+         "task": "e453679d-e36b-427a-a14d-58fe5e42ca19"
+      }
+
+   :>json task: The ID of the running task that are importing data.
+
+   :statuscode 400: When an invalid query parameter is passed in the request.
+   :statuscode 403: When an import task is already running.
+
+Import progression
+------------------
+
+The import progression resource allows to monitor an import task launched by ``/load`` entry.
+
+.. http:get:: /import-status/:name
+
+   Returns status of a running import task identified by given ``name``.
+   ``name`` is the task ID returned by ``/load`` entry.
+
+   **Example response**
+
+   .. sourcecode:: http
+
+      HTTP/1.1 200 OK
+      Content-Type: text/javascript
+
+   .. code-block:: json
+
+      {
+         "messages": ["latest", "logging", "lines", "from", "import", "process"],
+         "progress_message": "description of current import step",
+         "progress_value": 0.889,
+         "status": "active"
+      }
+
+   :>json messages: Latest logging lines printed in import process since last call to ``/import-status`` entry.
+   :>json progress_message: Description of current import process step.
+   :>json progress_value: Floating value (between 0 and 1 included) representing current import progression.
+   :>json status: Import process status. Either:
+       "active": still running
+       "error": terminated with an error (see latest messages for error info)
+       "finished": successfully terminated
+
+   :statuscode 400: When an invalid query parameter is passed in the request.
+
 Errors
 ------
 Oríon uses `conventional HTTP response codes <https://en.wikipedia.org/wiki/List_of_HTTP_status_codes>`_

src/orion/core/cli/db/dump.py

@@ -0,0 +1,61 @@
+#!/usr/bin/env python
+# pylint: disable=,protected-access
+"""
+Storage export tool
+===================
+
+Export database content into a file.
+
+"""
+import logging
+
+from orion.core.cli import base as cli
+from orion.core.io import experiment_builder
+from orion.storage.backup import dump_database
+from orion.storage.base import setup_storage
+
+logger = logging.getLogger(__name__)
+
+DESCRIPTION = "Export storage"
+
+
+def add_subparser(parser):
+    """Add the subparser that needs to be used for this command"""
+    dump_parser = parser.add_parser("dump", help=DESCRIPTION, description=DESCRIPTION)
+
+    cli.get_basic_args_group(dump_parser)
+
+    dump_parser.add_argument(
+        "-o",
+        "--output",
+        type=str,
+        default="dump.pkl",
+        help="Output file path (default: dump.pkl)",
+    )
+
+    dump_parser.add_argument(
+        "-f",
+        "--force",
+        action="store_true",
+        help="Whether to force overwrite if destination file already exists. "
+        "If specified, delete destination file and recreate a new one from scratch. "
+        "Otherwise (default), raise an error if destination file already exists.",
+    )
+
+    dump_parser.set_defaults(func=main)
+
+    return dump_parser
+
+
+def main(args):
+    """Script to dump storage"""
+    config = experiment_builder.get_cmd_config(args)
+    storage = setup_storage(config.get("storage"))
+    logger.info(f"Loaded src {storage}")
+    dump_database(
+        storage,
+        args["output"],
+        name=config.get("name"),
+        version=config.get("version"),
+        overwrite=args["force"],
+    )

src/orion/core/cli/db/load.py

@@ -0,0 +1,61 @@
+#!/usr/bin/env python
+"""
+Storage import tool
+===================
+
+Import database content from a file.
+
+"""
+import logging
+
+from orion.core.cli import base as cli
+from orion.core.io import experiment_builder
+from orion.storage.backup import load_database
+from orion.storage.base import setup_storage
+
+logger = logging.getLogger(__name__)
+
+DESCRIPTION = "Import storage"
+
+
+def add_subparser(parser):
+    """Add the subparser that needs to be used for this command"""
+    load_parser = parser.add_parser("load", help=DESCRIPTION, description=DESCRIPTION)
+
+    cli.get_basic_args_group(load_parser)
+
+    load_parser.add_argument(
+        "file",
+        type=str,
+        help="File to import",
+    )
+
+    load_parser.add_argument(
+        "-r",
+        "--resolve",
+        type=str,
+        choices=("ignore", "overwrite", "bump"),
+        help="Strategy to resolve conflicts: "
+        "'ignore', 'overwrite' or 'bump' "
+        "(bump version of imported experiment). "
+        "When overwriting, prior trials will be deleted. "
+        "If not specified, an exception will be raised on any conflict detected.",
+    )
+
+    load_parser.set_defaults(func=main)
+
+    return load_parser
+
+
+def main(args):
+    """Script to import storage"""
+    config = experiment_builder.get_cmd_config(args)
+    storage = setup_storage(config.get("storage"))
+    logger.info(f"Loaded dst {storage}")
+    load_database(
+        storage,
+        load_host=args["file"],
+        resolve=args["resolve"],
+        name=config.get("name"),
+        version=config.get("version"),
+    )

src/orion/core/cli/db/upgrade.py

@@ -135,7 +135,7 @@ def upgrade_documents(storage):
             )
 
         storage.update_experiment(uid=experiment, **experiment)
-        storage.initialize_algorithm_lock(uid, algorithm)
+        storage.write_algorithm_lock(uid, algorithm)
 
         for trial in storage.fetch_trials(uid=uid):
             # trial_config = trial.to_dict()

src/orion/core/cli/frontend.py

@@ -17,7 +17,6 @@
 from gunicorn.app.base import BaseApplication
 
 logger = logging.getLogger(__name__)
-logger.setLevel(logging.INFO)
 
 DESCRIPTION = "Starts Oríon Dashboard"
 

src/orion/core/utils/__init__.py

@@ -14,6 +14,7 @@
 from contextlib import contextmanager
 from glob import glob
 from importlib import import_module
+from tempfile import NamedTemporaryFile
 
 import pkg_resources
 
@@ -229,3 +230,13 @@ def sigterm_as_interrupt():
     yield None
 
     signal.signal(signal.SIGTERM, previous)
+
+
+def generate_temporary_file(basename="dump", suffix=".pkl"):
+    """Generate a temporary file where data could be saved.
+
+    Create an empty file without collision.
+    Return name of generated file.
+    """
+    with NamedTemporaryFile(prefix=f"{basename}_", suffix=suffix, delete=False) as tf:
+        return tf.name