Add storage import and export features to command line and web API

docs/src/user/storage.rst

@@ -153,6 +153,46 @@ simply run the upgrade command.
 
 .. _storage_python_apis:
 
+``dump`` Export database content
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``dump`` command allows to export database content to a PickledDB PKL file.
+
+.. code-block:: sh
+
+   orion db dump -o backup.pkl
+
+You can also dump a specific experiment.
+
+.. code-block:: sh
+
+   orion db dump -n exp-name -v exp-version -o backup-exp.pkl
+
+``load`` Import database content
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``load`` command allows to import database content
+from any PickledDB PKL file (including files generated by ``dump`` command).
+
+You must specify a conflict resolution policy using ``-r/--resolve`` argument
+to apply when conflicts are detected during import. Available policies are:
+
+- ``ignore``, to ignore imported data
+- ``overwrite``, to replace old data with imported data
+- ``bump``, to bump version of imported data and then make import
+
+By default, whole PKL file will be imported.
+
+.. code-block:: sh
+
+   orion db load backup.pkl -r ignore
+
+You can also import a specific experiment.
+
+.. code-block:: sh
+
+   orion db load backup.pkl -r overwrite -n exp-name -v exp-version
+
 Python APIs
 ===========
 

docs/src/user/web_api.rst

@@ -393,7 +393,6 @@ visualize your experiments and their results.
 
    :statuscode 404: When the specified experiment doesn't exist in the database.
 
-
 Benchmarks
 ----------
 The benchmark resource permits the retrieval of in-progress and completed benchmarks. You can
@@ -487,6 +486,101 @@ retrieve individual benchmarks as well as a list of all your benchmarks.
                     or assessment, task or algorithms are not part of the existing benchmark
                     configuration.
 
+Database dumping
+----------------
+
+The database dumping resource allows to dump database content
+into a PickledDB and download it as PKL file.
+
+.. http:get:: /dump
+
+   Return a PKL file containing database content.
+
+   :query name: Optional name of experiment to export. It unspecified, whole database is dumped.
+   :query version: Optional version of the experiment to retrieve.
+      If unspecified and name is specified, the **latest** version of the experiment is exported.
+      If both name and version are unspecified, whole database is dumped.
+
+   :statuscode 404: When an error occurred during dumping.
+
+Database loading
+----------------
+
+The database loading resource allow to import data from a PKL file
+
+.. http:post:: /load
+
+   Import data into database from a PKL file.
+   This is a POST request, as a file must be uploaded.
+   Launch an import task in a separate process in backend and return task ID
+   which may be used to get task progress.
+
+   :query file: PKL file to import
+   :query resolve: policy to resolve conflicts during import. Either:
+
+    - ``ignore``: ignore imported data on conflict
+    - ``overwrite``: overwrite ancient data on conflict
+    - ``bump``: bump version of imported data before insertion on conflict
+
+   :query name: Optional name of experiment to import. If unspecified, whole data from PKL file is imported.
+   :query version: Optional version of experiment to import.
+      If unspecified and name is specified, the **latest** version of the experiment is imported.
+      If both name and version are unspceified, whole data from PKL file is imported.
+
+   **Example response**
+
+   .. sourcecode:: http
+
+      HTTP/1.1 200 OK
+      Content-Type: text/javascript
+
+   .. code-block:: json
+
+      {
+         "task": "e453679d-e36b-427a-a14d-58fe5e42ca19"
+      }
+
+   :>json task: The ID of the running task that are importing data.
+
+   :statuscode 400: When an invalid query parameter is passed in the request.
+   :statuscode 403: When an import task is already running.
+
+Import progression
+------------------
+
+The import progression resource allows to monitor an import task launched by ``/load`` entry.
+
+.. http:get:: /import-status/:name
+
+   Returns status of a running import task identified by given ``name``.
+   ``name`` is the task ID returned by ``/load`` entry.
+
+   **Example response**
+
+   .. sourcecode:: http
+
+      HTTP/1.1 200 OK
+      Content-Type: text/javascript
+
+   .. code-block:: json
+
+      {
+         "messages": ["latest", "logging", "lines", "from", "import", "process"],
+         "progress_message": "description of current import step",
+         "progress_value": 0.889,
+         "status": "active"
+      }
+
+   :>json messages: Latest logging lines printed in import process since last call to ``/import-status`` entry.
+   :>json progress_message: Description of current import process step.
+   :>json progress_value: Floating value (between 0 and 1 included) representing current import progression.
+   :>json status: Import process status. Either:
+       "active": still running
+       "error": terminated with an error (see latest messages for error info)
+       "finished": successfully terminated
+
+   :statuscode 400: When an invalid query parameter is passed in the request.
+
 Errors
 ------
 Oríon uses `conventional HTTP response codes <https://en.wikipedia.org/wiki/List_of_HTTP_status_codes>`_

src/orion/core/cli/db/dump.py

@@ -0,0 +1,61 @@
+#!/usr/bin/env python
+# pylint: disable=,protected-access
+"""
+Storage export tool
+===================
+
+Export database content into a file.
+
+"""
+import logging
+
+from orion.core.cli import base as cli
+from orion.core.io import experiment_builder
+from orion.storage.backup import dump_database
+from orion.storage.base import setup_storage
+
+logger = logging.getLogger(__name__)
+
+DESCRIPTION = "Export storage"
+
+
+def add_subparser(parser):
+    """Add the subparser that needs to be used for this command"""
+    dump_parser = parser.add_parser("dump", help=DESCRIPTION, description=DESCRIPTION)
+
+    cli.get_basic_args_group(dump_parser)
+
+    dump_parser.add_argument(
+        "-o",
+        "--output",
+        type=str,
+        default="dump.pkl",
+        help="Output file path (default: dump.pkl)",
+    )
+
+    dump_parser.add_argument(
+        "-f",
+        "--force",
+        action="store_true",
+        help="Whether to force overwrite if destination file already exists. "
+        "If specified, delete destination file and recreate a new one from scratch. "
+        "Otherwise (default), raise an error if destination file already exists.",
+    )
+
+    dump_parser.set_defaults(func=main)
+
+    return dump_parser
+
+
+def main(args):
+    """Script to dump storage"""
+    config = experiment_builder.get_cmd_config(args)
+    storage = setup_storage(config.get("storage"))
+    logger.info(f"Loaded src {storage}")
+    dump_database(
+        storage,
+        args["output"],
+        name=config.get("name"),
+        version=config.get("version"),
+        overwrite=args["force"],
+    )

src/orion/core/cli/db/load.py

@@ -0,0 +1,61 @@
+#!/usr/bin/env python
+"""
+Storage import tool
+===================
+
+Import database content from a file.
+
+"""
+import logging
+
+from orion.core.cli import base as cli
+from orion.core.io import experiment_builder
+from orion.storage.backup import load_database
+from orion.storage.base import setup_storage
+
+logger = logging.getLogger(__name__)
+
+DESCRIPTION = "Import storage"
+
+
+def add_subparser(parser):
+    """Add the subparser that needs to be used for this command"""
+    load_parser = parser.add_parser("load", help=DESCRIPTION, description=DESCRIPTION)
+
+    cli.get_basic_args_group(load_parser)
+
+    load_parser.add_argument(
+        "file",
+        type=str,
+        help="File to import",
+    )
+
+    load_parser.add_argument(
+        "-r",
+        "--resolve",
+        type=str,
+        choices=("ignore", "overwrite", "bump"),
+        help="Strategy to resolve conflicts: "
+        "'ignore', 'overwrite' or 'bump' "
+        "(bump version of imported experiment). "
+        "When overwriting, prior trials will be deleted. "
+        "If not specified, an exception will be raised on any conflict detected.",
+    )
+
+    load_parser.set_defaults(func=main)
+
+    return load_parser
+
+
+def main(args):
+    """Script to import storage"""
+    config = experiment_builder.get_cmd_config(args)
+    storage = setup_storage(config.get("storage"))
+    logger.info(f"Loaded dst {storage}")
+    load_database(
+        storage,
+        load_host=args["file"],
+        resolve=args["resolve"],
+        name=config.get("name"),
+        version=config.get("version"),
+    )

src/orion/core/cli/db/upgrade.py

@@ -135,7 +135,7 @@ def upgrade_documents(storage):
             )
 
         storage.update_experiment(uid=experiment, **experiment)
-        storage.initialize_algorithm_lock(uid, algorithm)
+        storage.write_algorithm_lock(uid, algorithm)
 
         for trial in storage.fetch_trials(uid=uid):
             # trial_config = trial.to_dict()

src/orion/core/cli/frontend.py

@@ -17,7 +17,6 @@
 from gunicorn.app.base import BaseApplication
 
 logger = logging.getLogger(__name__)
-logger.setLevel(logging.INFO)
 
 DESCRIPTION = "Starts Oríon Dashboard"
 

src/orion/core/utils/__init__.py

@@ -14,6 +14,7 @@
 from contextlib import contextmanager
 from glob import glob
 from importlib import import_module
+from tempfile import NamedTemporaryFile
 
 import pkg_resources
 
@@ -229,3 +230,13 @@ def sigterm_as_interrupt():
     yield None
 
     signal.signal(signal.SIGTERM, previous)
+
+
+def generate_temporary_file(basename="dump", suffix=".pkl"):
+    """Generate a temporary file where data could be saved.
+
+    Create an empty file without collision.
+    Return name of generated file.
+    """
+    with NamedTemporaryFile(prefix=f"{basename}_", suffix=suffix, delete=False) as tf:
+        return tf.name