diff --git a/doc/_static/ingest-transaction-fsm.png b/doc/_static/ingest-transaction-fsm.png
new file mode 100755
index 000000000..29d4a076b
Binary files /dev/null and b/doc/_static/ingest-transaction-fsm.png differ
diff --git a/doc/_static/ingest-transaction-fsm.pptx b/doc/_static/ingest-transaction-fsm.pptx
new file mode 100644
index 000000000..7f2c2bbf0
Binary files /dev/null and b/doc/_static/ingest-transaction-fsm.pptx differ
diff --git a/doc/_static/ingest-transactions-aborted.png b/doc/_static/ingest-transactions-aborted.png
new file mode 100755
index 000000000..f5e7bcf81
Binary files /dev/null and b/doc/_static/ingest-transactions-aborted.png differ
diff --git a/doc/_static/ingest-transactions-aborted.pptx b/doc/_static/ingest-transactions-aborted.pptx
new file mode 100644
index 000000000..adfae1ad6
Binary files /dev/null and b/doc/_static/ingest-transactions-aborted.pptx differ
diff --git a/doc/_static/ingest-transactions-failed.png b/doc/_static/ingest-transactions-failed.png
new file mode 100755
index 000000000..d2a0d60d5
Binary files /dev/null and b/doc/_static/ingest-transactions-failed.png differ
diff --git a/doc/_static/ingest-transactions-failed.pptx b/doc/_static/ingest-transactions-failed.pptx
new file mode 100644
index 000000000..1ffe243f9
Binary files /dev/null and b/doc/_static/ingest-transactions-failed.pptx differ
diff --git a/doc/_static/ingest-transactions-resolved.png b/doc/_static/ingest-transactions-resolved.png
new file mode 100755
index 000000000..a321adbd1
Binary files /dev/null and b/doc/_static/ingest-transactions-resolved.png differ
diff --git a/doc/_static/ingest-transactions-resolved.pptx b/doc/_static/ingest-transactions-resolved.pptx
new file mode 100644
index 000000000..932f344e0
Binary files /dev/null and b/doc/_static/ingest-transactions-resolved.pptx differ
diff --git a/doc/admin/data-table-indexes.rst b/doc/admin/data-table-indexes.rst
new file mode 100644
index 000000000..39b98f8ca
--- /dev/null
+++ b/doc/admin/data-table-indexes.rst
@@ -0,0 +1,607 @@
+
+.. _admin-data-table-index:
+
+==================
+Data Table Indexes
+==================
+
+.. note::
+
+ All operations on indexes are only allowed on databases that are in the published state. The system will
+ refuse all requests on databases that are in a process of being ingested. This is done for three reasons:
+
+ #. To avoid interfering with the catalog ingest operations.
+ #. To prevent returning an inconsistent (or wrong) state of the indexes
+ #. To prevent transient errors due to potential race conditions since the overall state of the distributed catalogs
+ might be being changed by the Ingest system.
+
+.. _admin-data-table-index-intro:
+
+Introduction
+------------
+
+This document explains how to use the index management API of the Qserv Replication System. Services provided by the system
+are related to the following index management operations in MySQL:
+
+.. code-block:: sql
+
+ SHOW INDEX FROM
...
+ CREATE ... INDEX ON
...
+ DROP INDEX ON
...
+
+However, unlike the single table operations listed above, the services allow managing groups of related tables distributed
+across Qserv worker nodes. Hence, each request for any service refers to a single such group. In particular:
+
+- if a request refers to the *regular* (fully replicated) table then all instances of the table will be affected by the request.
+- otherwise (in the case of the *partitioned* tables) each replica of every chunk of the table will be included in an operation.
+ Note that the index management services differentiate between the *chunk* tables themselves and the corresponding *full overlap*
+ tables. When submitting a request, a user will have to choose which of those tables will be included in the operation.
+
+The latter would also be reflected in the result and error reports made by the services. The JSON objects returned by
+the services would return the names of the *final* tables affected by the operations, not the names of the corresponding
+*base* name of a table specified in service requests. This is done to facilitate investigating problems with Qserv should
+they occur. See more on the *base* and *final* table names in the section:
+
+- :ref:`ingest-general-base-table-names` (REST)
+
+Things to consider before creating indexes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The recommendations mentioned in this section are not complete. Please, get yourself familiarized with
+the MySQL documentation on indexes, and specifically with the index creation command:
+
+- https://dev.mysql.com/doc/refman/8.4/en/create-index.html
+
+syntax, before attempting the operation. Always keep in mind that there is the MySQL (or alike) machinery behind Qserv and
+that the data tables in Qserv are presently based on the MyISAM storage engine:
+
+- https://dev.mysql.com/doc/refman/8.4/en/myisam-storage-engine.html
+
+These are a few points to consider.
+
+General:
+
+- Indexes are created on tables, and not on views.
+- Indexes are created in the scope of the table, and not databases.
+- In the case of the *partitioned* tables, the *chunk* and *full* overlap may not need to have the same set of indexes.
+ In some cases, it may not be even possible, for instance, due to different ``UNIQUE`` constraints requirements.
+- Please, provide a reasonable comment for an index, even though, comments aren't mandatory. Your comments could be useful
+ for bookkeeping purposes to know why and what the index was created for.
+- Be aware that indexes take additional space on the Qserv workers' filesystems where the database tables are residing.
+ Potentially, if too many indexes were created, MySQL may run out of disk space and stop working. The rule of thumb for
+ estimating the amount of space to be taken by an index is based on the sizes of columns mentioned in an index
+ specification (explained in the next section) multiplied by the total number of rows in a table (on which the index
+ is being created). There are other factors here, such as the type of a table (regular or partitioned) as well as
+ the number of replicas for the partitioned tables. The index management service which is used for creating indexes
+ will also make the best attempt to prevent creating indexes if it will detect that the amount of available disk space
+ is falling too low. In this case, the service will refuse to create an index and an error will be reported back.
+
+When choosing a name for an index:
+
+- The name should be unique (don't confuse this with the UNIQUE keyword used in the index specifications) in the scope
+ of the table. It means it should not be already taken by some other index. Always check which indexes already exist
+ in a table before creating a new one.
+- Generally, the name must adhere to the MySQL requirements for identifiers as explained
+ in:
+
+ - https://dev.mysql.com/doc/refman/8.4/en/identifier-qualifiers.html
+
+- Keep in mind that names of identifiers (including indexes) in Qserv are case-insensitive. This is not the general
+ requirement in MySQL, where the case sensitivity of identifiers is configurable one way or another. It's because
+ of a design decision the original Qserv developers made to configure the underlying MySQL machinery.
+- To make things simple, restrain from using special characters (all but the underscore one ``_``).
+- The length of the name should not exceed 64 characters:
+
+ - https://dev.mysql.com/doc/refman/8.4/en/identifier-length.html
+
+Error reporting
+^^^^^^^^^^^^^^^^
+
+All services mentioned in the document are the Replication system's services, and they adhere to the same error reporting
+schema. The schema is explained in the document:
+
+- :ref:`ingest-general-error-reporting` (REST)
+
+In addition, the index managemnt services may return the service-specific error conditions in the ``error_ext`` attribute:
+
+.. code-block:
+
+ "error_ext" : {
+ "job_state" : ,
+ "workers" : {
+ : {
+
: {
+ "request_status" : ,
+ "request_error" :
+ },
+ ...
+ },
+ ...
+ }
+ }
+
+Where:
+
+``job_state`` : *string*
+ The completion status (state) of the corresponding C++ job classes:
+
+ - ``lsst::qserv::replica::SqlGetIndexesJob``
+ - ``lsst::qserv::replica::SqlCreateIndexesJob``
+ - ``lsst::qserv::replica::SqlDropIndexesJob``
+
+ The status will be serialized into a string. The explanation of the possible values could be found in
+ the following C++ header:
+
+ - https://github.com/lsst/qserv/blob/master/src/replica/jobs/Job.h
+
+ Look for this enum type:
+
+ .. code-block::
+
+ enum ExtendedState {
+ NONE, ///< No extended state exists at this time.
+ SUCCESS, ///< The job has been fully implemented.
+ CONFIG_ERROR, ///< Problems with job configuration found.
+ FAILED, ///< The job has failed.
+ ...
+ }
+
+ Also look for worker/table-specific errors in a JSON object explained below.
+
+``workers`` : *object*
+ The JSON object which has unique identifiers of workers (attribute ``worker``) as the keys, where the corresponding
+ value for the worker is another JSON object which has names of worker-side tables as the next-level keys for
+ descriptions of problems with managing indexes for the corresponding tables.
+
+ .. note:
+
+ Only the problematic tables (if any) would be mentioned in the report. If no problems were seen during
+ the index management operations then a collection of workers and tables will be empty.
+
+``request_status`` : *string*
+ The completion status (state) of the index creation C++ request classes:
+
+ - ``SqlGetIndexesRequest``
+ - ``SqlCreateIndexesRequest``
+ - ``SqlDropIndexesRequest``
+
+ The status will be serialized into a string. More information on possible values could be found in the
+ following C++ header:
+
+ - https://github.com/lsst/qserv/blob/main/src/replica/requests/Request.h
+
+ Look for this enum type:
+
+ .. code-block::
+
+ enum ExtendedState {
+ NONE, /// No extended state exists at this time
+ SUCCESS, /// The request has been fully implemented
+ CLIENT_ERROR, /// The request could not be implemented due to
+ /// an unrecoverable client-side error.
+ SERVER_BAD, /// Server reports that the request can not be implemented
+ /// due to incorrect parameters, etc.
+ SERVER_ERROR, /// The request could not be implemented due to
+ /// an unrecoverable server-side error.
+ ...
+ };
+
+``request_error`` : *string*
+ This string provides an expanded explanation of an error reported by the Replication system's worker (in case if the
+ request failed on the worker's side and is reported to the service).
+
+.. note::
+
+ **Reporting partial successes or failures**
+
+ Since the index management requests may (will) involve multiple tables, the corresponding operations may be potentially
+ partially successful and partially not successful. All failures for specific indexes which couldn't be managed (created,
+ queried, or deleted) would be reported as explained in the previous section. For example, that would be a case if a request
+ was made to drop a known to-exist index, and if no such index existed for some final tables. There may be various reasons
+ why this might happen. An explanation of the reasons is beyond a scope of this document. The best way a user should treat
+ this situation is to expect that the service would do the "best effort" of removing the index. It's also allowed to run
+ the index removal request multiple times. This won't make any harm. All subsequent requests will report failures for all
+ final tables in the specified group of tables.
+
+.. _admin-data-table-index-create:
+
+Creating
+--------
+
+To create a new index, a user must submit a request to the service:
+
+.. list-table::
+ :widths: 10 90
+ :header-rows: 0
+
+ * - ``POST``
+ - ``/replication/sql/index``
+
+Where the request object has the following schema:
+
+.. code-block::
+
+ { "database" : ,
+ "table" : ,
+ "overlap" : ,
+ "index" : ,
+ "spec" : ,
+ "comment" : ,
+ "columns" : [
+ { "column" : ,
+ "length " :,
+ "ascending" :
+ },
+ ..
+ ],
+ "auth_key" :
+ }
+
+Where:
+
+``database`` : *string*
+ The required name of the database where the table resides.
+
+``table`` : *string*
+ The required *base* name of the table where the index will be created.
+
+``overlap`` : *number* := ``0``
+ The optional *overlap* flagg indicating a sub-type of the *chunk* table. The value should be one of the following:
+
+ - ``1`` : *full overlap*
+ - ``0`` : *chunk*
+
+``index`` : *string*
+ The required name of the index to be created.
+
+``spec`` : *string*
+ The required index specification. The specification should adhere to the MySQL requirements for the index creation command:
+
+ - https://dev.mysql.com/doc/refman/8.4/en/create-index.html
+
+ Where a value of the parameter should be one of the following keywords: ``DEFAULT``, ``UNIQUE``, ``FULLTEXT``, or ``SPATIAL``.
+ All but the first one (``DEFAULT``) are mentioned in the MySQL documentation. The keywords map to the indfex
+ creation command:
+
+ .. code-block:: sql
+
+ CREATE [UNIQUE | FULLTEXT | SPATIAL] INDEX index_name ...
+
+ The REST service expects ``DEFAULT`` in those cases when none of the other three specifications are provided.
+ Any other value or a lack of ant will be considered as an error.
+
+``comment`` : *string* := ``""``
+ The optional comment for the index. The value will be passed via the ``COMMENT`` parameter of the MySQL index
+ creation command:
+
+ .. code-block:: sql
+
+ CREATE ... INDEX ... COMMENT 'string' ...
+
+``columns`` : *array*
+ The required non-empty array of JSON objects keys mentioned in the key_part of the index creation statements:
+
+ .. code-block:: sql
+
+ CREATE ... INDEX index_name ON tbl_name (key_part,...) ...
+
+ .. note::
+
+ The current implementation of the service doesn't support the extended-expression syntax of key_part introduced
+ in MySQL version 8.
+
+ These are the mandatory attributes of each key-specific object:
+
+ ``column`` : *string*
+ The required name of a column.
+ ``length`` : *number*
+ The required length of a substring used for the index. It only has a meaning for columns of
+ types: ``TEXT``, ``CHAR(N)``, ``VARCHAR(N)``, ``BLOB`` , etc. And it must be always 0 for other column
+ types (numeric, etc.). Otherwise, an index creation request will fail.
+
+ ``ascending`` : *number*
+ The required sorting order of the column in the index. It translates into ``ASC`` or ``DESC`` optiona
+ in the key definition in ``key_part``. A value of ``0`` will be interpreted as ``DESC``. Any other positive number
+ will be imterpreted as to ``ASC``.
+
+``auth_key`` : *string*
+ The required zauthorization key.
+
+Here is an example of the index creation request. Let's supposed we have the *regular* (fully replicated)
+table that has the following schema:
+
+.. code-block:: sql
+
+ CREATE TABLE `sdss_stripe82_01`.`Science_Ccd_Exposure_NoFile` (
+ `scienceCcdExposureId` BIGINT(20) NOT NULL,
+ `run` INT(11) NOT NULL,
+ `filterId` TINYINT(4) NOT NULL,
+ `camcol` TINYINT(4) NOT NULL,
+ `field` INT(11) NOT NULL,
+ `path` VARCHAR(255) NOT NULL
+ );
+
+And, suppose we are going to create the ``PRIMARY`` key index based on the very first column ``scienceCcdExposureId``.
+In this case the request object will look like this:
+
+.. code-block:: json
+
+ { "database" : "sdss_stripe82_01",
+ "table" : "Science_Ccd_Exposure_NoFile",
+ "index" : "PRIMARY",
+ "spec" : "UNIQUE",
+ "comment" : "This is the primary key index",
+ "columns" : [
+ { "column" : "scienceCcdExposureId",
+ "length" : 0,
+ "ascending" : 1
+ }
+ ],
+ "auth_key" : ""
+ }
+
+The request deliberately misses the optional ``overlap`` attribute since it won't apply to the regular tables.
+
+Here is how the request could be submitted to the service using ``curl``:
+
+.. code-block:: bash
+
+ curl 'http://localhost:25081/replication/sql/index' \
+ -X POST -H "Content-Type: application/json" \
+ -d '{"database":"sdss_stripe82_01","table":"Science_Ccd_Exposure_NoFile",
+ "index":"PRIMARY","spec":"UNIQUE","comment":"This is the primary key index",
+ "columns":[{"column":"scienceCcdExposureId","length":0,"ascending":1}],
+ "auth_key":""}'
+
+.. _admin-data-table-index-delete:
+
+Deleting
+--------
+
+To delete an existing index, a user must submit a request to the service:
+
+.. list-table::
+ :widths: 10 90
+ :header-rows: 0
+
+ * - ``DELETE``
+ - ``/replication/sql/index``
+
+Where the request object has the following schema:
+
+.. code-block::
+
+ { "database" : ,
+ "table" : ,
+ "overlap" : ,
+ "index" : ,
+ "auth_key" :
+ }
+
+Where:
+
+``database`` : *string*
+ The required name of the database where the table resides.
+
+``table`` : *string*
+ The required *base* name of the table where the index will be created.
+
+``overlap`` : *number* := ``0``
+ The optional *overlap* flagg indicating a sub-type of the *chunk* table. The value should be one of the following:
+
+ - ``1`` : *full overlap*
+ - ``0`` : *chunk*
+
+``index`` : *string*
+ The required name of the index to be dropped.
+
+Here is an example of the index deletion request. It's based on the same table that was mentioned in the previous section.
+The request object will look like this:
+
+.. code-block:: json
+
+ { "database" : "sdss_stripe82_01",
+ "table" : "Science_Ccd_Exposure_NoFile",
+ "index" : "PRIMARY",
+ "auth_key" : ""
+ }
+
+Here is how the request could be submitted to the service using ``curl``:
+
+.. code-block:: bash
+
+ curl 'http://localhost:25081/replication/sql/index' \
+ -X DELETE -H "Content-Type: application/json" \
+ -d '{"database":"sdss_stripe82_01","table":"Science_Ccd_Exposure_NoFile",
+ "index":"PRIMARY",
+ "auth_key":""}'
+
+.. _admin-data-table-index-inspect:
+
+Inspecting
+----------
+
+To inspect the existing indexes, a user must submit a request to the service:
+
+.. list-table::
+ :widths: 10 90
+ :header-rows: 0
+
+ * - ``GET``
+ - ``/replication/sql/index/:database/:table[?overlap={0|1}]``
+
+Where the service path has the following parameters:
+
+``database`` : *string*
+ The name of a database affected by the operation.
+
+``table`` : *string*
+ The name of the table for which the indexes are required to be collected.
+
+The optional query parameyter is
+
+``overlap`` : *number* := ``0``
+ The optional *overlap* flagg indicating a sub-type of the *chunk* table. The value should be one of the following:
+
+ - ``1`` : *full overlap*
+ - ``0`` : *chunk*
+
+In case of successful completion of the request the JSON object returned by the service will have the following schema:
+
+.. code-block::
+
+ { "status": {
+ "database" : ,
+ "table" : ,
+ "overlap" : ,
+ "indexes" : [
+ { "name" : ,
+ "unique" : ,
+ "type" : ,
+ "comment" : ,
+ "status" : ,
+ "num_replicas_total" : ,
+ "num_replicas" : ,
+ "columns" : [
+ { "name" : ,
+ "seq" : ,
+ "sub_part" : ,
+ "collation" :
+ },
+ ...
+ ]
+ },
+ ...
+ ]
+ }
+ }
+
+Where:
+
+``name`` : *string*
+ The name of the index (the key).
+
+``unique`` : *number*
+ The numeric flag indicates of the index's keys are unique, where a value of ``0`` means they're unique. Any other
+ value would mean the opposite.
+
+``type`` : *string*
+ The type of index, such as ``BTREE``, ``SPATIAL``, ``PRIMARY``, ``FULLTEXT``, etc.
+
+``comment`` : *string*
+ An optional explanation for the index passed to the index creation statement:
+
+ .. code-block:: sql
+
+ CREATE ... INDEX ... COMMENT 'string' ...
+
+``status`` : *string*
+ The status of the index. This parameter considers the aggregate status of the index across all replicas of the table.
+ Possible values here are:
+
+ - ``COMPLETE`` : the same index (same type, columns) is present in all replicas of the table (or its chunks in the case
+ of the partitioned table)
+ - ``INCOMPLETE`` : the same index is present in a subset of replicas of the table, where all indexes have the same
+ definition.
+ - ``INCONSISTENT`` : instances of the index that have the same name have different definitions in some replicas
+
+ .. warning::
+
+ The result object reported by the service will not provide any further details on the last status ``INCONSISTENT``
+ apart from indicating the inconsistency. It will be up to the data administrator to investigate which replicas have
+ unexpected index definitions.
+
+``num_replicas_total`` : *number*
+ The total number of replicas that exist for the table. This is the target number of replicas where the index is expected
+ to be present.
+
+``num_replicas`` : *number*
+ The number of replicas where the index was found to be present. If this number is not the same as the one reported
+ in the attribute
+ ``num_replicas_total`` then the index will be ``INCOMPLETE``.
+
+``columns`` : *array*
+ The collection of columns that were included in the index definition. Each entry of the collection has:
+
+ ``name`` : *string*
+ The name of the column
+ ``seq`` : *number*
+ The 1-based position of the column in the index.
+ ``sub_part`` : *number*
+ The index prefix. That is, the number of indexed characters if the column is only partly indexed 0 if
+ the entire column is indexed.
+ ``collation`` : *string*
+ How the column is sorted in the index. This can have values ``ASC``, ``DESC``, or ``NOT_SORTED``.
+
+The following request will report indexes from the fully-replicated table ``ivoa.ObsCore``:
+
+.. code-block:: bash
+
+ curl http://localhost:25081/replication/sql/index/ivoa/ObsCore -X GET
+
+The (truncated and formatted for readability) result of an operation performed in a Qserv deployment with 6-workers may
+look like this:
+
+.. code-block:: json
+
+ { "status" : {
+ "database" : "ivoa"
+ "indexes" : [
+ { "name" : "idx_dataproduct_subtype",
+ "type" : "BTREE",
+ "unique" : 0,
+ "status" : "COMPLETE",
+ "num_replicas" : 6,
+ "num_replicas_total" : 6,
+ "comment" : "The regular index on the column dataproduct_subtype",
+ "columns" : [
+ { "collation" : "ASC",
+ "name" : "dataproduct_subtype",
+ "seq" : 1,
+ "sub_part" : 0
+ }
+ ]
+ },
+ { "name" : "idx_s_region_bounds",
+ "type" : "SPATIAL",
+ "unique" : 0,
+ "status" : "COMPLETE",
+ "num_replicas" : 6,
+ "num_replicas_total" : 6,
+ "comment" : "The spatial index on the geometric region s_region_bounds",
+ "columns" : [
+ { "collation" : "ASC",
+ "name" : "s_region_bounds",
+ "seq" : 1,
+ "sub_part" : 32
+ }
+ ]
+ },
+ { "name" : "idx_lsst_tract_patch",
+ "type" : "BTREE",
+ "unique" : 0,
+ "status" : "COMPLETE",
+ "num_replicas" : 6,
+ "num_replicas_total" : 6,
+ "comment" : "The composite index on the columns lsst_tract and lsst_patch",
+ "columns" : [
+ { "collation" : "ASC",
+ "name" : "lsst_tract",
+ "seq" : 1,
+ "sub_part" : 0
+ },
+ { "collation" : "ASC",
+ "name" : "lsst_patch",
+ "seq" : 2,
+ "sub_part" : 0
+ }
+ ]
+ },
+ }
+
+.. note::
+
+ - The second index ``idx_s_region_bounds`` is spatial. It's based on the binary column of which only
+ the first 32 bytes are indexed.
+
+ - The third index ``idx_lsst_tract_patch`` is defined over two columns.
diff --git a/doc/admin/index.rst b/doc/admin/index.rst
index 89a01074d..c47817e72 100644
--- a/doc/admin/index.rst
+++ b/doc/admin/index.rst
@@ -1,7 +1,5 @@
-.. warning::
- **Information in this guide is known to be outdated.** A documentation sprint is underway which will
- include updates and revisions to this guide.
+.. _admin:
#####################
Administrator's Guide
@@ -11,3 +9,5 @@ Administrator's Guide
:maxdepth: 4
k8s
+ row-counters
+ data-table-indexes
diff --git a/doc/admin/row-counters.rst b/doc/admin/row-counters.rst
new file mode 100644
index 000000000..a1d75f7bb
--- /dev/null
+++ b/doc/admin/row-counters.rst
@@ -0,0 +1,176 @@
+
+.. _admin-row-counters:
+
+=========================
+Row counters optimization
+=========================
+
+.. _admin-row-counters-intro:
+
+Introduction
+------------
+
+Shortly after the first public instances of Qserv were made available to the users, it was observed that many users
+were launching the following query:
+
+.. code-block:: sql
+
+ SELECT COUNT(*) FROM .
+
+Normally, Qserv would process the query by broadcasting the query to all workers to count rows at each chunk
+table and aggregate results into a single number at Czar. This is basically the very same mechanism that is found
+behind the *shared scan* (or just *scan*) queries. The performnce of the *scan* queries is known to vary depending on
+the following factors:
+
+- the number of chunks in the table of interest
+- the number of workers
+- and the presence of other competing queries (especially the slow ones)
+
+In the best case scenario such scan would take seconds, in the worst one - many minutes or even hours.
+This could quickly cause (and has caused) frustration among users since this query looks like (and in reality is)
+the very trivial non-scan query.
+
+To address this situation, Qserv has a built-in optimization that is targeting exactly this class of queries.
+Here is how it works. For each data table Qserv Czar would have an optional metadata table to store the number
+of rows for each chunk. The table is populated and managed by the Qserv Replication system.
+
+Note that this optimization is presently an option. And these are the reasons:
+
+- Counter collection requires scanning all chunk tables, which would take time. Doing this during
+ the catalog *publishing* time would prolong the ingest time and increase the chances of instabilities
+ for the workflows (in general, the longer some operation is going - the higher the probability of runing into
+ the infrastructure-related faulures).
+- The counters are not needed for the purposes of the data ingest *per se*. These are just optimizations for the queries.
+- Building the counters before the ingested data have been Q&A-ed may not be a good idea.
+- The counters may need to be rebuilt if the data have been changed (after fix ups to the ingested catalogs)
+
+The rest of this section along with the formal description of the corresponding REST services explains how to build
+and manage the counters.
+
+.. note::
+
+ In the future, the per-chunk counters will be used for optimizing another class of the unconditional queries
+ presented below:
+
+ .. code-block:: sql
+
+ SELECT * FROM .
LIMIT
+ SELECT `col`,`col2` FROM .
LIMIT
+
+ For these "indiscriminate" data probes Qserv would dispatch chunk queries to a subset of random chunks that have enough
+ rows to satisfy the requirements specified in ``LIMIT ``.
+
+
+.. _admin-row-counters-build:
+
+Building and deploying
+----------------------
+
+.. warning::
+
+ Depending on a scale of a catalog (data size of the affected table), it may take a while before this operation
+ will be complete.
+
+.. note::
+
+ Please, be advised that the very same operation could be performed at the catalog publishing time as explained in:
+
+ - :ref:`ingest-db-table-management-publish-db` (REST)
+
+ The choice of doing this at the catalog publishing time, or doing this as a separate operation explained in this document
+ is left to the Qserv administrators or developers of the ingest workflows. The general recommendation is to make it
+ a separate stage of the ingest workflow. In this case, the overall transition time of a catalog to the final published
+ state would be faster. In the end, the row counters optimization is optional, and it doesn't affect the overall
+ functionality of Qserv or query results seen by users.
+
+To build and deploy the counters one would need to use the following REST service:
+
+- :ref:`ingest-row-counters-deploy` (REST)
+
+The service needs to be invoked for every table of the ingested catalog. This is the typical example of using this service
+that would work regardless if the very same operation was already done before:
+
+.. code-block:: bash
+
+ curl http://localhost:25080/ingest/table-stats \
+ -X POST -H "Content-Type: application/json" \
+ -d '{"database":"test101",
+ "table":"Object",
+ "overlap_selector":"CHUNK_AND_OVERLAP",
+ "force_rescan":1,
+ "row_counters_state_update_policy":"ENABLED",
+ "row_counters_deploy_at_qserv":1,
+ "auth_key":""}'
+
+This would work for tables of any type: *director*, *dependent*, *RefMatch*, or *regular* (fully replicated). If the counters
+already existed in the Replication system's database, they would still be rescanned and redeployed in there.
+
+It may be a good idea to compare the performance of Qserv for executing the above-mentioned queries before and after running
+this operation. Normally, if the table statistics are available at Qserv, it should take a small fraction of
+a second (about 10 milliseconds) to see the result on the lightly loaded Qserv.
+
+.. _admin-row-counters-delete:
+
+Deleting
+--------
+
+Sometimes, if there is doubt that the row counters were incorrectly scanned, or when Q&A-in the ingested catalog,
+a data administrator may want to remove the counters and let Qserv do the full scan of the table instead. This can be done
+by using the following REST service:
+
+
+- :ref:`ingest-row-counters-delete` (REST)
+
+Likewise, the previously explained service, this one should also be invoked for each table needing attention. Here is
+an example:
+
+.. code-block:: bash
+
+ curl http://localhost:25080/ingest/table-stats/test101/Object \
+ -X DELETE -H "Content-Type: application/json" \
+ -d '{"overlap_selector":"CHUNK_AND_OVERLAP","qserv_only":1,"auth_key":""}'
+
+Note that with a combination of the parameters shown above, the statistics will be removed from Qserv only.
+So, the system would not need to rescan the tables again should the statistics need to be rebuilt. The counters could be simply
+redeployed later at Qserv. To remove the counters from the Replication system's persistent state as well
+the request should have ``qserv_only=0``.
+
+An alternative technique explained in the next section is to tell Qserv not to use the counters for optimizing queries.
+
+
+.. _admin-row-counters-disable:
+
+Disabling the optimization at run-time
+---------------------------------------
+
+.. warning::
+
+ This is a global setting that affects all users of Qserv. All new quries will be ru w/o the optimization.
+ It should be used with caution. Normally, it is meant to be used by the Qserv data administrator to investigate
+ suspected issues with Qserv or the catalogs it serves.
+
+To complement the previously explained methods for scanning, deploying, or deleting row counters for query optimization,
+Qserv also supports the run-time switch. The switch is turned on or off by the following statement to be submitted via
+the front-ends of Qserv:
+
+.. code-block:: sql
+
+ SET GLOBAL QSERV_ROW_COUNTER_OPTIMIZATION = 1
+ SET GLOBAL QSERV_ROW_COUNTER_OPTIMIZATION = 0
+
+
+The default behavior of Qserv when the variable is not set is to enable the optimization for tables where the counters
+are available.
+
+.. _admin-row-counters-retrieve:
+
+Inspecting
+----------
+
+It's also possible to retrieve the counters from the Replication system's state using the following REST service:
+
+- :ref:`ingest-row-counters-inspect` (REST)
+
+The information obtained in this way could be used for various purposes, such as investigating suspected issues with
+the counters, monitoring data placement in the chunks, or making visual representations of the chunk density maps.
+See the description of the REST service for further details on this subject.
diff --git a/doc/conf.py b/doc/conf.py
index 1c35a00ea..e481eb922 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -42,6 +42,7 @@
r"^https://rubinobs.atlassian.net/wiki/",
r"^https://rubinobs.atlassian.net/browse/",
r"^https://www.slac.stanford.edu/",
+ r".*/_images/",
]
html_additional_pages = {
diff --git a/doc/ingest/api/advanced/index.rst b/doc/ingest/api/advanced/index.rst
new file mode 100644
index 000000000..8a79cfb8e
--- /dev/null
+++ b/doc/ingest/api/advanced/index.rst
@@ -0,0 +1,44 @@
+
+.. _ingest-api-advanced:
+
+==================
+Advanced Scenarios
+==================
+
+.. _ingest-api-advanced-multiple-transactions:
+
+Multiple transactions
+----------------------
+
+.. _ingest-api-advanced-unpublishing-databases:
+
+Ingesting tables into the published catalogs
+--------------------------------------------
+
+.. _ingest-api-advanced-transaction-abort:
+
+Aborting transactions
+----------------------
+
+When should a transaction be aborted?
+What happens when a transaction is aborted?
+How long does it take to abort a transaction?
+How to abort a transaction?
+What to do if a transaction cannot be aborted?
+What happens if a transaction is aborted while contributions are being processed?
+Will the transaction be aborted if the Master Replication Controller is restarted?
+Is it possible to have unfinished transactions in the system when publishing a catalog?
+
+.. _ingest-api-advanced-contribution-requests:
+
+Options for making contribution requests
+----------------------------------------
+
+.. _ingest-api-advanced-server-config:
+
+Ingest server configuration options
+------------------------------------
+
+Explain what configuration options are available for the Ingest server:
+
+- safe
\ No newline at end of file
diff --git a/doc/ingest/api/appendix/index.rst b/doc/ingest/api/appendix/index.rst
new file mode 100644
index 000000000..8faeb11e9
--- /dev/null
+++ b/doc/ingest/api/appendix/index.rst
@@ -0,0 +1,247 @@
+.. _ingest-api-appendix:
+
+========
+APPENDIX
+========
+
+.. _ingest-api-appendix-indexes:
+
+Managing indexes of MySQL tables at Qserv workers
+-------------------------------------------------
+
+.. _ingest-api-appendix-partitioning:
+
+Partitioning data of the RefMatch tables
+-----------------------------------------
+
+Introduction
+^^^^^^^^^^^^
+
+The ``RefMatch`` tables are a special class of tables that are designed to match rows between two independent *director* tables
+belonging to the same or different catalogs. In this case, there is no obvious 1-to-1 match between rows of the director tables.
+Instead, the pipelines compute a (spatial) match table between the two that provides a many-to-many relationship between both tables.
+The complication is that a match record might reference a row (an *object*) and reference row that fall on opposite sides of
+the partition boundary (into different chunks). Qserv deals with this by taking advantage of the overlap that must be stored
+alongside each partition (this overlap is stored so that Qserv can avoid inter-worker communication when performing
+spatial joins on the fly).
+
+Since the ``RefMatch`` tables are also partitioned tables the input data (CSV) of the tables have to be partitioned into chunks.
+In order to partition the RefMatch tables one would have to use a special version of the partitioning tool sph-partition-matches.
+A source code of the tool is found in the source tree of Qserv: https://github.com/lsst/qserv/blob/main/src/partition/sph-partition-matches.cc.
+The corresponding binary is built and placed into the binary Docker image of Qserv.
+
+Here is an example illustrating how to launch the tool from the container:
+
+.. code-block:: bash
+
+ % docker run -it qserv/lite-qserv:2022.9.1-rc1 sph-partition-matches --help
+
+ sph-partition-matches [options]
+
+ The match partitioner partitions one or more input CSV files in
+ preparation for loading by database worker nodes. This involves assigning
+ both positions in a match pair to a location in a 2-level subdivision
+ scheme, where a location consists of a chunk and sub-chunk ID, and
+ outputting the match pair once for each distinct location. Match pairs
+ are bucket-sorted by chunk ID, resulting in chunk files that can then
+ be distributed to worker nodes for loading.
+ A partitioned data-set can be built-up incrementally by running the
+ partitioner with disjoint input file sets and the same output directory.
+ Beware - the output CSV format, partitioning parameters, and worker
+ node count MUST be identical between runs. Additionally, only one
+ partitioner process should write to a given output directory at a
+ time. If any of these conditions are not met, then the resulting
+ chunk files will be corrupt and/or useless.
+ \_____________________ Common:
+ -h [ --help ] Demystify program usage.
+ -v [ --verbose ] Chatty output.
+ -c [ --config-file ] arg The name of a configuration file
+ containing program option values in a
+ ...
+
+The tool has two parameters specifying the locations of the input (CSV) file and the output folder where
+the partitioned products will be stored:
+
+.. code-block:: bash
+
+ % sph-partition-matches --help
+ ..
+ \_____________________ Output:
+ --out.dir arg The directory to write output files to.
+ \______________________ Input:
+ -i [ --in.path ] arg An input file or directory name. If the
+ name identifies a directory, then all
+ the files and symbolic links to files
+ in the directory are treated as inputs.
+ This option must be specified at least
+ once.
+
+.. hint::
+
+ If the tool is launched via the docker command as was shown above, one would have to mount the corresponding
+ host paths into the container.
+
+All tables, including both *director* tables and the ``RefMatch`` table itself, have to be partitioned using
+the same values of the partitioning parameters, including:
+
+- The number of stripes
+- The number of sub-stripes
+- The overlap radius
+
+Values of the partitioning parameters should be specified using the following options (the default values shown below are meaningless
+for any production scenario):
+
+.. code-block:: bash
+
+ --part.num-stripes arg (=18) The number of latitude angle stripes to
+ divide the sky into.
+ --part.num-sub-stripes arg (=100) The number of sub-stripes to divide
+ each stripe into.
+ --part.overlap arg (=0.01) Chunk/sub-chunk overlap radius (deg).
+
+The next sections present two options for partitioning the input data.
+
+The spatial match within the given overlap radius
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This is the most reliable way of partitioning the input data of the match tables. It is available when
+the input rows of the match table carry the exact spatial coordinates of both matched rows (from the corresponding
+*director* tables).
+
+In this scenario, the input data file (``CSV``) is expected to have 4 columns representing the spatial coordinates
+of the matched rows from the *director* tables on the 1st ("left") and on the 2nd ("righ"). Roles and sample names
+of the columns are presented in the table below:
+
+``dir1_ra``
+ The *right ascent* coordinate (*longitude*) of the 1st matched entity (from the 1st *director* table).
+``dir1_dec``
+ The *declination* coordinate (*latitude*) of the 1st matched entity (from the 1st director table).
+``dir2_ra``
+ The *right ascent* coordinate (*longitude*) of the 2nd matched entity (from the 2nd *director* table).
+``dir2_dec``
+ The *declination* coordinate (*latitude*) of the 2nd matched entity (from the 2nd director table).
+
+The names of these columns need to be passed to the partitioning tool using two special parameters:
+
+.. code-block:: bash
+
+ % sph-partition-matches \
+ --part.pos1="dir1_ra,dir1_dec" \
+ --part.pos2="dir2_ra,dir2_dec"
+
+.. note:
+
+ The order of the columns in each packed pair pf columns is important. The names must be separated by commas.
+
+When using this technique for partitioning the match tables, it's required that the input CSV file(s) had at least those 4 columns
+mentioned above. The actual number of columns could be larger. Values of all additional will be copied into the partitioned
+products (the chunk files). The original order of the columns will be preserved.
+
+Here is an example of a sample ``CSV`` file that has values of the above-described spatial coordinates in the first 4 columns
+and the object identifiers of the corresponding rows from the matched *director* tables in the last 2 columns:
+
+.. code-block::
+
+ 10.101,43.021,10.101,43.021,123456,6788404040
+ 10.101,43.021,10.102,43.023,123456,6788404041
+
+The last two columns are meant to store values of the following columns:
+
+``dir1_objectId``
+ The unique object identifier of the 1st *director* table.
+``dir2_objectId``
+ The unique object identifier of the 2nd *director* table.
+
+The input CSV file shown above could be also presented in the tabular format:
+
+.. list-table::
+ :widths: 10 10 10 10 10 10
+ :header-rows: 1
+
+ * - ``dir1_ra``
+ - ``dir1_dec``
+ - ``dir2_ra``
+ - ``dir2_dec``
+ - ``dir1_objectId``
+ - ``dir2_objectId``
+ * - 0.101
+ - 43.021
+ - 10.101
+ - 43.021
+ - 123456
+ - 6788404040
+ * - 0.101
+ - 43.021
+ - 10.102
+ - 43.023
+ - 123456
+ - 6788404041
+
+Note that this is actually a 1-to-2 match, in which a single object (``123456``) of the 1st director has two matched
+objects (``6788404040`` and ``6788404041``) in the 2nd director. Also, note that the second matched object has slightly
+different spatial coordinates than the first one. If the value of the overlap parameter is bigger than the difference
+between the coordinates then the tool will be able to match the objects successfully. For example, this would work if
+a value of the overlap was set to ``0.01``. Otherwise, no match will be made and the row will be ignored by the tool.
+
+.. _warning:
+
+ It is assumed that the input data of the ``RefMatch`` tables are correctly produced by the data processing
+ pipelines. Verifying the quality of the input data is beyond the scope of this document. However, one might
+ consider writing a special tool for pre-scanning the input files and finding problems in the files.
+
+Here is the complete practical example of how to run the tool with the assumptions made above:
+
+.. code-block:: bash
+
+ % cat in.csv
+ 10.101,43.021,10.101,43.021,123456,6788404040
+ 10.101,43.021,10.102,43.023,123456,6788404041
+
+ % cat config.json
+ {
+ "part":{
+ "num-stripes":340.
+ "num-sub-stripes":3,
+ "overlap":0.01,
+ "pos1":"dir1_ra,dir1_dec",
+ "pos2":"dir2_ra,dir2_dec"
+ },
+ "in":{
+ "csv":{
+ "null":"\\N",
+ "delimiter":",",
+ "field":[
+ "dir1_ra",
+ "dir1_dec"
+ "dir2_ra",
+ "dir2_dec",
+ "dir1_objectId",
+ "dir2_objectId"
+ ]
+ }
+ },
+ "out":{
+ "csv":{
+ "null":"\\N",
+ "delimiter":",",
+ "escape":"\\",
+ "no-quote":true
+ }
+ }
+ }
+
+ % mkdir chunks
+ % sph-partition-matches -c config.json --in.path=in.csv --out.dir=chunks/
+
+Partitioning using index maps
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. note::
+
+ This section is under construction. Only the basic idea is presented here.
+
+This is an alternative way of partitioning the input data of the match tables. It is available when the input rows of the match table
+do not carry the exact spatial coordinates of both matched rows (from the corresponding *director* tables). Instead, the input data
+has to carry the unique object identifiers of the matched rows. The tool will use the object identifiers to find the spatial coordinates
+of the matched rows in the *director* tables. The tool will use the index maps of the *director* tables to find the spatial coordinates
+of the matched rows.
diff --git a/doc/ingest/api/concepts/index.rst b/doc/ingest/api/concepts/index.rst
new file mode 100644
index 000000000..2007f9b7b
--- /dev/null
+++ b/doc/ingest/api/concepts/index.rst
@@ -0,0 +1,362 @@
+
+.. _ingest-api-concepts:
+
+=============
+Main concepts
+=============
+
+.. hint::
+
+ This section of the document starts with the high-level overview of the ingest workflow in Qserv.
+ Please read this section carefully to understand the main concepts and the order of the tasks that
+ have to be accomplished in order to ingest data into Qserv.
+
+ After finishing with the overview, a reader has two options on how to proceed:
+
+ - continue studing the concepts of the API in depth by visiting subsections on :ref:`ingest-api-concepts-table-types`,
+ :ref:`ingest-api-concepts-transactions`, :ref:`ingest-api-concepts-publishing-data`, etc.
+ - or, jumping straight to the practical example of the :ref:`ingest-api-simple`
+
+.. _ingest-api-concepts-overview:
+
+Overview of the ingest workflow
+-------------------------------
+
+In the nutshell, any ingest workflow has to accomlish quite a few tasks in order to ingest data into Qserv.
+These tasks are presented in the correct order below:
+
+**Plan the ingest**
+
+.. hint::
+
+ You may also contact Qserv experts or Qserv administrators to get help on the planning stage.
+
+There is a number of important decisions to be made ahead of time in the following areas before starting ingesting data
+into Qserv. Knowing these facts allows to organize the ingest activities in the most efficient way.
+
+- Creating a new database or adding tables to the existing one?
+
+ - **Concepts**: :ref:`ingest-api-concepts-publishing-data`
+
+- What are the types of tables to be ingested?
+
+ - **Concepts**: :ref:`ingest-api-concepts-table-types`
+
+- What should be the values of the partitioning parameters of the partitioned tables?
+
+ - **Concepts**: :ref:`ingest-api-concepts-database-families`
+
+- What is a scale of the planned ingest effort?
+
+ - The amount of data (rows, bytes) to be ingested in each table
+ - The number of the ``CSV`` files to be ingested
+ - Sizes of the files
+ - The number of the workers that are available in Qserv
+ - **Advaced**: :ref:`ingest-api-advanced-multiple-transactions`
+
+- Where the ready to ingest data will be located?
+
+ - Are there any data staging areas available?
+ - **Advaced**: :ref:`ingest-api-advanced-contribution-requests`
+
+**Prepare the input data**
+
+Data files (table *contributions*) to be ingested into Qserv need to be in the ``CSV`` format. It's up to the workflow
+to ensure that the data is in the right format and that it's sanitized to ensure the values of the columns
+are compatible with the MySQL expectations.
+
+- **Concepts**: :ref:`ingest-api-concepts-data-preparation`
+
+**Prepare configuration files**
+
+The configurations of the ingested entities (databases, tables, etc.) are presented to the Ingest system
+in a form of thw JSON objects.
+
+**Register or un-publish a database**
+
+- **TODO** a link to the REST service
+
+**Register tables**
+
+- name
+- type
+
+ - type-specific attributes, some which are referring to other tables (names and the foreign keys)
+ - schema (including column names and types)
+
+- **TODO** a link to the REST service
+
+**Configure the Ingest service**
+
+- **TODO** a link to the REST service
+
+**Start transactions**
+
+- :ref:`ingest-api-concepts-transactions` (Concepts)
+- :ref:`ingest-trans-management` (API)
+
+**Figure out locations of tables and chunks**
+
+- Connection parameters of the workers where the table and chunk data (*contributions*) will be sent.
+- Note that it's a responsibility of the workflow to contact the workers and to provide them with the data
+ or a reference to the data.
+- The API provides the REST services for obtaining the desired information for the data products
+ that are being ingested.
+
+**Send the data to the workers**
+
+- initiate the ingest activities
+
+**Monitor the progress of the ingest activities**
+
+- The workflow can query the REST services to get the status of databases, tables, transactions
+ and the data contribution requests.
+
+**Commit the transactions**
+
+- **TODO** a link to the REST service
+- also mention transaction abort
+
+**Publish the database**
+
+- the table would be published automatically
+- when the stage is fiished the database and the tables will be visible to the users
+- **TODO** a link to the REST service
+
+**Verify the ingested data products**
+
+- the data can be queried
+- the data can be compared to the original data
+
+**Perform the optional post-ingest data management operation on the ingested tables**
+
+- :ref:`ingest-api-post-ingest`
+
+.. _ingest-api-concepts-table-types:
+
+Types of tables in Qserv
+------------------------
+
+.. note:
+
+ Consider moving this section to the general documentatin on Qserv and refer to it from where
+ it's needed.
+
+Explain the tables and drow diagrams for:
+
+- regular (fully replicated) tables
+- partitioned tables, including sub-types
+
+ - director tables
+ - dependent tables
+
+ - simple (1 director)
+ - ref-match (2 directors)
+
+Design of the partitioned tables. Draw a diagram that includes workers and chunk.
+
+.. _ingest-api-concepts-database-families:
+
+Database families
+-----------------
+
+Explain roles and implications of the database families in Qserv.
+
+- to allow joins between the tables accros the same family
+- required by the Ingest workflow
+
+.. _ingest-api-concepts-transactions:
+
+Transactions
+------------
+
+The distributed transaction mechanism (also known as *super-transactions*) is one of the key technologies that was
+implemented in the Qserv Ingest system to allow for the incremental updates of the overall state of the data and metadata
+while ensuring the consistency of the ingested catalogs. Transactions also play an important role in allowing
+the high-performance ingest activities to be performed in a distributed environment. Transactions if used correct may
+significantly increase the level of parallelism of the ingest workflows. Transactions are not visible to end users.
+
+It's a responsibility of the workflows to manage transactions as needed for the ingest activities:
+
+- :ref:`ingest-trans-management` (REST)
+
+Row tagging
+^^^^^^^^^^^
+
+Transactions implement the tagging mechanism for the ingested data (rows). All rows ingested into to the tables are tagged with
+the transaction identifiers that is unique for each transaction. Hence, all contribution requests made into the tables via
+this API are associated with a specific identifier. The identifiers are usually sent in the service request and response
+objects in an attribute ``transaction_id``. For example, see a description of the following REST service:
+
+- :ref:`ingest-worker-contrib-by-ref` (REST)
+
+The tagging can be seen as a special column called ``qserv_trans_id`` that is automatically added by the Ingest system
+into the table schema of each table. In the current implementation of the system, this is the very first column of the table.
+The column is of the ``UNSIGNED INT`` type and is not nullable. The column is visible to the users and is queriable.
+Here is an illustration of a query and the corresponding result set illustrating the concept:
+
+.. code-block:: sql
+
+ SELECT `qserv_trans_id`, `objectId`,`chunkId`
+ FROM `dp02_dc2_catalogs`.`Object`
+ WHERE `qserv_trans_id` IN (860, 861);
+
+ +----------------+---------------------+---------+
+ | qserv_trans_id | objectId | chunkId |
+ +----------------+---------------------+---------+
+ | 860 | 1249546586455828954 | 57871 |
+ | 860 | 1249546586455828968 | 57871 |
+ . . . .
+ | 861 | 1252546054176403713 | 57891 |
+ | 861 | 1252546054176403722 | 57891 |
+ +----------------+---------------------+---------+
+
+Checkpointing
+^^^^^^^^^^^^^
+
+Transactions provide the checkpointing mechanism that allows rolling back to a prior consistent state of the affected tables
+should any problem occur during the ingest activities. Transactions may spans across many workers and tables located
+at the workers. It's up to the workflow to decide what contrubutions to ingest and in what order to ingest those in
+a scope of each transaction.
+
+The following diagram illustrates the concept of the transactions in Qserv. There are 3 transactions that are started and
+commited sequentially (in the real life scenarios the transactions can be and should be started and commited in parallel,
+and indepedently of each other). Data are ingested into two separate table located at 2 workers. The diagram also shows
+a failure to ingest the data into table ``Table-A`` at ``Worker-X`` in a scope of ``Transaction-2``:
+
+.. image:: /_static/ingest-transactions-failed.png
+ :target: ../../../_images/ingest-transactions-failed.png
+ :alt: Failed Transaction Contribution
+
+At this point the table ``Table-A`` at ``Worker-X`` is found in an inconsistent state. The workflow can decide to roll back
+the transaction and to re-try the ingest activities for the failed transaction. The rollback is performed by
+the worker ingest service:
+
+- :ref:`ingest-trans-management-end` (REST)
+
+Also read the following document to learn more about the transaction abort:
+
+- :ref:`ingest-api-advanced-transaction-abort` (ADVANCED)
+
+This resulted in the following state of the tables, which is *clean* and consistent:
+
+.. image:: /_static/ingest-transactions-aborted.png
+ :target: ../../../_images/ingest-transactions-aborted.png
+ :alt: Aborted Transaction
+
+After that, the workflow can re-try **ALL** ingest activities that were meant to be done in a scope of the previously
+failed transaction by starting another transaction. If the ingest activities are successful, the tables will be in the
+consistent state:
+
+.. image:: /_static/ingest-transactions-resolved.png
+ :target: ../../../_images/ingest-transactions-resolved.png
+ :alt: Another Transaction
+
+Implementation Details
+^^^^^^^^^^^^^^^^^^^^^^
+
+The Qserv transactions are quite different from the one in the typical RDBMS implementations. Firstly, they are not designed
+as an an isolation mechanis for executing user queries, and the ar enot visible to Qserv users. In Qserv, tables that are being ingested are not seen or queriable by
+the users anyway. The main purpose of the transactions in Qserv is to allow for the incremental updates of the distributed
+state of data in Qserv across many (potentially - the hundreds of) workers. Each worker runs its own instance of
+the MySQL/MariaDB server which is not aware of the of the others.
+
+The second problem to be addressed by the transactions is a lack of the transaction support in the MyISAM table engine that
+is used in Qserv for the data tables. The MyISAM engine is used in Qserv due to it ssimplicity and high performance.
+Unfortunately, failuires while ingesting data into the MyISAM tables can leave the table in a corrupted state.
+The transactions provide the mechanism to roll back the tables to a consistent state in case of the failures. The current
+implementation of the transactions in Qserv is based on the MySQL/MariaDB partitions:
+
+- https://mariadb.com/kb/en/partitioning-overview/
+
+
+.. warning::
+
+ When the catalog is being published, the partitioned MyISAM tables are converted to the regular format.
+ This operation is performed by the Qserv Ingest system.
+ The conversion is a time-consuming operation and may take a long time to complete for
+ a single table. An observed performance of the operation per table is on a scale of ``20 MB/s`` to ``50 MB/s``.
+ However, a typical catalog will have thousands of such chunk tables which would be processed in parallel
+ at all workers of the Qserv cluster.
+
+From a prospective of the workflows, these are the most important limitations of the transactions:
+
+- Transaction identifiers are the 32-bit unsignd integer numbers. The maximum number of the transactions that can be
+ started in the system is 2^32 - 1 = 4,294,967,295. The transactions are not re-used, so the number of the transactions
+ that can be started in the system is limited by the number of the unique transaction identifiers that can be generated
+ by the system.
+
+- The transaction with the identifier ``0`` is reserved for the system for the so called *default* transaction.
+ The workflows can't ingest any contributions in a context of that transaction, or manage this special transaction.
+
+- MySQL tables only allow up to ``8000`` partitions per table. This is a limitation of the MySQL/MariaDB partitioning mechanism.
+ And there is a certain oerhead in MySQL for each partition. Hence, it's not recommended to start more than ``1000`` transactions
+ during the ingest.
+
+Transaction numbers directly map to the partition identifiers of the MySQL/MariaDB partitioned tables. Here is an example
+of a few chunk tables of a catalog that is still being ingested:
+
+.. code-block:: bash
+
+ -rw-rw----+ 1 rubinqsv gu 4868 Sep 10 20:48 gaia_source_1012.frm
+ -rw-rw----+ 1 rubinqsv gu 48 Sep 10 20:48 gaia_source_1012.par
+ -rw-rw----+ 1 rubinqsv gu 0 Sep 10 20:46 gaia_source_1012#P#p0.MYD
+ -rw-rw----+ 1 rubinqsv gu 2048 Sep 10 20:46 gaia_source_1012#P#p0.MYI
+ -rw-rw----+ 1 rubinqsv gu 0 Sep 10 20:46 gaia_source_1012#P#p1623.MYD
+ -rw-rw----+ 1 rubinqsv gu 2048 Sep 10 20:46 gaia_source_1012#P#p1623.MYI
+ -rw-rw----+ 1 rubinqsv gu 31000308 Sep 10 20:48 gaia_source_1012#P#p1628.MYD
+ -rw-rw----+ 1 rubinqsv gu 2048 Sep 11 19:49 gaia_source_1012#P#p1628.MYI
+ -rw-rw----+ 1 rubinqsv gu 4868 Sep 10 20:48 gaia_source_1020.frm
+ -rw-rw----+ 1 rubinqsv gu 48 Sep 10 20:48 gaia_source_1020.par
+ -rw-rw----+ 1 rubinqsv gu 0 Sep 10 20:46 gaia_source_1020#P#p0.MYD
+ -rw-rw----+ 1 rubinqsv gu 2048 Sep 10 20:46 gaia_source_1020#P#p0.MYI
+ -rw-rw----+ 1 rubinqsv gu 51622084 Sep 10 20:48 gaia_source_1020#P#p1624.MYD
+ -rw-rw----+ 1 rubinqsv gu 2048 Sep 11 19:49 gaia_source_1020#P#p1624.MYI
+ -rw-rw----+ 1 rubinqsv gu 0 Sep 10 20:46 gaia_source_1020#P#p1630.MYD
+ -rw-rw----+ 1 rubinqsv gu 2048 Sep 10 20:46 gaia_source_1020#P#p1630.MYI
+ -rw-rw----+ 1 rubinqsv gu 4868 Sep 10 20:47 gaia_source_1028.frm
+ -rw-rw----+ 1 rubinqsv gu 48 Sep 10 20:47 gaia_source_1028.par
+ -rw-rw----+ 1 rubinqsv gu 0 Sep 10 20:46 gaia_source_1028#P#p0.MYD
+ -rw-rw----+ 1 rubinqsv gu 2048 Sep 10 20:46 gaia_source_1028#P#p0.MYI
+ -rw-rw----+ 1 rubinqsv gu 739825104 Sep 10 20:48 gaia_source_1028#P#p1625.MYD
+ -rw-rw----+ 1 rubinqsv gu 2048 Sep 11 19:49 gaia_source_1028#P#p1625.MYI
+ -rw-rw----+ 1 rubinqsv gu 0 Sep 10 20:46 gaia_source_1028#P#p1629.MYD
+ -rw-rw----+ 1 rubinqsv gu 2048 Sep 10 20:46 gaia_source_1028#P#p1629.MYI
+
+This snapshot was taken by looking at the MariaDB data directory at one of the Qserv workers. Note that the tables
+are partitioned by the transaction numbers, where the transaction number is the number after the ``#P#`` in the table name.
+
+
+.. _ingest-api-concepts-publishing-data:
+
+Publishing databases and tables
+-------------------------------
+
+Explain the concept of the published databases and tables in Qserv.
+
+Also mention:
+
+- :ref:`ingest-api-advanced-unpublishing-databases`
+
+
+.. _ingest-api-concepts-data-preparation:
+
+Data preparation
+----------------
+
+More details on the data preparation steps.
+
+In general there are two types of operations that have to be performed on the input data:
+
+- If the input files are in the Parquet format they have to be converted to the CSV format:
+- For the *partitioned* tables artition the ``CSV`` files into chunks
+- Place the input files in the right location where the workflow will be
+
+- ensure that the data is in the right format (CSV is the only supported format) and that it's sanitized
+ to ensure the values of the columns are compatible with the MySQL expectations:
+
+ - one good example is the ``BOOL`` type that maps to the ``TINYINT`` type in MySQL. Values like ``true``
+ and ``false`` are not supported by MySQL. Hence they need to be converted to ``1`` and ``0`` respectively.
+
+ - another example are ``-inf`` and ``+inf`` values that are not supported by MySQL. They need to be
+ converted to the ``-1.7976931348623157E+308`` and ``1.7976931348623157E+308`` respectively.
\ No newline at end of file
diff --git a/doc/ingest/api/index.rst b/doc/ingest/api/index.rst
index 75b56ddd4..88ba5b74a 100644
--- a/doc/ingest/api/index.rst
+++ b/doc/ingest/api/index.rst
@@ -1,20 +1,22 @@
+
+.. note::
+
+ Information in this guide corresponds to the version **38** of the Qserv REST API. Keep in mind
+ that each implementation of the API has a specific version. The version number will change
+ if any changes to the implementation or the API that might affect users will be made.
+ The current document will be kept updated to reflect the latest version of the API.
+
#####################################
The Ingest Workflow Developer's Guide
#####################################
-TBC
-
-
-.. list-table:: Title
- :widths: 25 25 50
- :header-rows: 1
+.. toctree::
+ :maxdepth: 4
- * - Heading row 1, column 1
- - Heading row 1, column 2
- - Heading row 1, column 3
- * - Row 1, column 1
- -
- - Row 1, column 3
- * - Row 2, column 1
- - Row 2, column 2
- - Row 2, column 3
+ introduction
+ concepts/index
+ simple/index
+ advanced/index
+ post-ingest/index
+ reference/index
+ appendix/index
diff --git a/doc/ingest/api/introduction.rst b/doc/ingest/api/introduction.rst
new file mode 100644
index 000000000..b860129d0
--- /dev/null
+++ b/doc/ingest/api/introduction.rst
@@ -0,0 +1,91 @@
+Introduction
+============
+
+This document presents an API that is available in Qserv for constructing the data ingest applications (also mentioned
+in the document as *ingest workflows*). The API is designed to provide a high-performance and reliable mechanism for
+ingesting large quantities of data where the high performance or reliability of the ingests is at stake.
+The document is intended to be a practical guide for the developers who are building those applications.
+It provides a high-level overview of the API, its main components, and the typical workflows that can be built using the API.
+
+At the very high level, the Qserv Ingest system is comprised of:
+
+- The REST server that is integrated into the Master Replication Controller. The server provides a collection
+ of services for managing metadata and states of the new catalogs to be ingested. The server also coordinates
+ its own operations with Qserv itself and the Qserv Replication System to prevent interferences with those
+ and minimize failures during catalog ingest activities.
+- The Worker Ingest REST service run at each Qserv worker node alongside the Qserv worker itself and the worker MariaDB server.
+ The role of these services is to actually ingest the client's data into the corresponding MySQL tables.
+ The services would also do an additional (albeit, minimal) preprocessing and data transformation (where or when needed)
+ before ingesting the input data into MySQL. Each worker server also includes its own REST server for processing
+ the "by reference" ingest requests as well as various metadata requests in the scope of the workers.
+
+Implementation-wise, the Ingest System heavily relies on services and functions of the Replication System including
+the Replication System's Controller Framework, various (including the Configuration) services, and the worker-side
+server infrastructure of the Replication System.
+
+Client workflows interact with the system's services via open interfaces (based on the HTTP protocol, REST services,
+JSON data format, etc.) and use ready-to-use tools to fulfill their goals of ingesting catalogs.
+
+Here is a brief summary of the Qserv Ingest System's features:
+
+- It introduces the well-defined states and semantics into the ingest process. With that, a process of ingesting a new catalog
+ now has to go through a sequence of specific steps maintaining a progressive state of the catalog within Qserv
+ while it's being ingested. The state transitions and the corresponding enforcements made by the system would
+ always ensure that the catalog would be in a consistent state during each step of the process.
+ Altogether, this model increases the robustness of the process, and it also makes it more efficient.
+
+- To facilitate and implement the above-mentioned state transitions the new system introduces a distributed
+ *tagging* and *checkpointing* mechanism called *super-transactions*. The transactions allow for incremental
+ updates of the overall state of the data and metadata while allowing to safely roll back to a prior consistent
+ state should any problem occur during data loading within such transactions.
+
+ - The data tagging capability of the transactions can be also used by the ingest workflows and by
+ the Qserv administrators for bookkeeping of the ingest activities and for the quality control of
+ the ingested catalogs.
+
+- In its very foundation, the system has been designed for constructing high-performance and parallel ingest
+ workflows w/o compromising the consistency of the ingested catalogs.
+
+- For the actual data loading, the system offers plenty of options, inluding pushing data into Qserv directly
+ via a proprietary binary protocol using :ref:`ingest-tools-qserv-replica-file`, :ref:`ingest-worker-contrib-by-val`
+ in the HTTP request body, or :ref:`ingest-worker-contrib-by-ref`. In the latter case, the input data (so called table
+ *contributions*) will be pulled by the worker services from remote locations as instructed by the ingest workflows.
+ The presently supported sources include the object stores (via the HTTP/HTTPS protocols) and the locally mounted
+ distributed filesystems (via the POSIX protocol).
+
+ - The ongoing work on the system includes the development of the support for the ingesting contributions
+ from the S3 object stores.
+
+- The data loading services also collect various information on the ongoing status of the ingest activities,
+ abnormal conditions that may occur during reading, interpreting, or loading the data into Qserv, as well
+ as the metadata for the data that is loaded. The information is retained within the persistent
+ state of the Replication/Ingest System for the monitoring and debugging purposes. A feedback is provided
+ to the workflows on various aspects of the ingest activities. The feedback is useful for the workflows to adjust their
+ behavior and to ensure the quality of the data being ingested.
+
+ - To get further info on this subject, see sections :ref:`ingest-general-error-reporting` and
+ :ref:`ingest-worker-contrib-descriptor-warnings`.
+ In addition, the API provides REST services for obtaining metadata on the state of catalogs, tables, distributed
+ transactions, contribution requests, the progress of the requested operations, etc.
+
+**What the Ingest System does NOT do**:
+
+- As per its current implementation (which may change in the future) it does not automatically partition
+ input files. This task is expected to be a responsibility of the ingest workflows. The only data format
+ is is presently supported for the table payload are ``CSV`` and ``JSON`` (primarily for ingesting
+ user-generated data products as explained in :ref:`http-frontend-ingest`).
+
+- It does not (with an exception of adding an extra leading column ``qserv_trans_id`` required by
+ the implementation of the previously mentioned *super-transactions*) pre-process the input ``CSV``
+ payload sent to the Ingest Data Servers by the workflows for loading into tables.
+ It's up to the workflows to sanitize the input data and to make them ready to be ingested into Qserv.
+
+More information on the requirements and the low-level technical details of its implementation (unless it's
+needed for the purposes of this document's goals) can be found elsewhere.
+
+It's recommended to read the document sequentially. Most ideas presented in the document are introduced in
+a section :ref:`ingest-api-concepts` and illustrated with a simple practical example in :ref:`ingest-api-simple`.
+The section is followed by a few more sections covering :ref:`ingest-api-advanced` and :ref:`ingest-api-post-ingest`.
+The :ref:`ingest-api-reference` section of the document provides complete descriptions of the REST services and tools
+mentioned in the document. The last section :ref:`ingest-api-appendix` offers the supplementary material that may
+be useful for the developers building the ingest workflows.
diff --git a/doc/ingest/api/post-ingest/index.rst b/doc/ingest/api/post-ingest/index.rst
new file mode 100644
index 000000000..e07b57187
--- /dev/null
+++ b/doc/ingest/api/post-ingest/index.rst
@@ -0,0 +1,11 @@
+
+.. _ingest-api-post-ingest:
+
+=================================
+Post-Ingest Data Management Tasks
+=================================
+
+The following optional steps are performed after the data has been ingested:
+
+- :ref:`admin-row-counters`
+- :ref:`admin-data-table-index`
diff --git a/doc/ingest/api/reference/index.rst b/doc/ingest/api/reference/index.rst
new file mode 100644
index 000000000..24a58e83d
--- /dev/null
+++ b/doc/ingest/api/reference/index.rst
@@ -0,0 +1,12 @@
+
+.. _ingest-api-reference:
+
+######################
+Ingest API Reference
+######################
+
+.. toctree::
+ :maxdepth: 4
+
+ rest/index
+ tools
diff --git a/doc/ingest/api/reference/rest/controller/config.rst b/doc/ingest/api/reference/rest/controller/config.rst
new file mode 100644
index 000000000..6967fa9dc
--- /dev/null
+++ b/doc/ingest/api/reference/rest/controller/config.rst
@@ -0,0 +1,235 @@
+.. _ingest-config:
+
+Configuring parameters of the ingests
+=====================================
+
+.. _ingest-config-set:
+
+Setting configuration parameters
+--------------------------------
+
+Parameters are set for a database (regardless of the *published* status) using the following service:
+
+.. list-table::
+ :widths: 10 90
+ :header-rows: 0
+
+ * - ``PUT``
+ - ``/ingest/config``
+
+The request object has the following schema:
+
+.. code-block::
+
+ { "database" : ,
+ "SSL_VERIFYHOST" : ,
+ "SSL_VERIFYPEER" : ,
+ "CAPATH" : ,
+ "CAINFO" : ,
+ "CAINFO_VAL" : ,
+ "PROXY_SSL_VERIFYHOST" : ,
+ "PROXY_SSL_VERIFYPEER" : ,
+ "PROXY_CAPATH" : ,
+ "PROXY_CAINFO" : ,
+ "PROXY_CAINFO_VAL" : ,
+ "CURLOPT_PROXY" : ,
+ "CURLOPT_NOPROXY" : ,
+ "CURLOPT_HTTPPROXYTUNNEL" : ,
+ "CONNECTTIMEOUT" : ,
+ "TIMEOUT" : ,
+ "LOW_SPEED_LIMIT" : ,
+ "LOW_SPEED_TIME" : ,
+ "ASYNC_PROC_LIMIT" :
+ }
+
+Where:
+
+``database`` : *string* : **required**
+ The required name of a database affected by the operation.
+
+``SSL_VERIFYHOST`` : *number* = ``2``
+ The optional flag that tells the system to verify the host of the peer. If the value is set
+ to ``0`` the system will not check the host name against the certificate. Any other value would tell the system
+ to perform the check.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_SSL_VERIFYHOST.html.
+
+``SSL_VERIFYPEER`` : *number* = ``1``
+ The optional flag that tells the system to verify the peer's certificate. If the value is set
+ to ``0`` the system will not check the certificate. Any other value would tell the system to perform the check.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html.
+
+``CAPATH`` : *string* = ``/etc/ssl/certs``
+ The optional path to a directory holding multiple CA certificates. The system will use the certificates
+ in the directory to verify the peer's certificate. If the value is set to an empty string the system will not use
+ the certificates.
+
+ Putting the empty string as a value of the parameter will effectively turn this option off as if it has never been
+ configured for the database.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_CAPATH.html.
+
+``CAINFO`` : *string* = ``/etc/ssl/certs/ca-certificates.crt``
+ The optional path to a file holding a bundle of CA certificates. The system will use the certificates
+ in the file to verify the peer's certificate. If the value is set to an empty string the system will not use
+ the certificates.
+
+ Putting the empty string as a value of the parameter will effectively turn this option off as if it has never been
+ configured for the database.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_CAINFO.html.
+
+``CAINFO_VAL`` : *string* = ``""``
+ The optional value of a certificate bundle for a peer. This parameter is used in those cases when it's
+ impossible to inject the bundle directly into the Ingest workers' environments. If a non-empty value of the parameter
+ is provided then ingest servers will use it instead of the one mentioned (if any) in the above-described
+ attribute ``CAINFO``.
+
+ **Attention**: Values of the attribute are the actual certificates, not file paths like in the case of ``CAINFO``.
+
+``PROXY_SSL_VERIFYHOST`` : *number* = ``2``
+ The optional flag that tells the system to verify the host of the proxy. If the value is set
+ to ``0`` the system will not check the host name against the certificate. Any other value would tell the system
+ to perform the check.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_PROXY_SSL_VERIFYHOST.html.
+
+``PROXY_SSL_VERIFYPEER`` : *number* = ``1``
+ The optional flag that tells the system to verify the peer's certificate. If the value is set
+ to ``0`` the system will not check the certificate. Any other value would tell the system to perform the check.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_PROXY_SSL_VERIFYPEER.html.
+
+``PROXY_CAPATH`` : *string* = ``""``
+ The optional path to a directory holding multiple CA certificates. The system will use the certificates
+ in the directory to verify the peer's certificate. If the value is set to an empty string the system will not use
+ the certificates.
+
+ Putting the empty string as a value of the parameter will effectively turn this option off as if it has never been
+ configured for the database.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_PROXY_CAPATH.html.
+
+``PROXY_CAINFO`` : *string* = ``""``
+ The optional path to a file holding a bundle of CA certificates. The system will use the certificates
+ in the file to verify the peer's certificate. If the value is set to an empty string the system will not use
+ the certificates.
+
+ Putting the empty string as a value of the parameter will effectively turn this option off as if it has never been
+ configured for the database.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_PROXY_CAINFO.html.
+
+``PROXY_CAINFO_VAL`` : *string* = ``""``
+ The optional value of a certificate bundle for a proxy. This parameter is used in those cases when it's
+ impossible to inject the bundle directly into the Ingest workers' environments. If a non-empty value of the parameter
+ is provided then ingest servers will use it instead of the one mentioned (if any) in the above-described
+ attribute ``PROXY_CAINFO``.
+
+ **Attention**: Values of the attribute are the actual certificates, not file paths like in the case of ``PROXY_CAINFO``.
+
+``CURLOPT_PROXY`` : *string* = ``""``
+ Set the optional proxy to use for the upcoming request. The parameter should be a null-terminated string
+ holding the host name or dotted numerical IP address. A numerical IPv6 address must be written within ``[brackets]``.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_PROXY.html.
+
+``CURLOPT_NOPROXY`` : *string* = ``""``
+ The optional string consists of a comma-separated list of host names that do not require a proxy
+ to get reached, even if one is specified.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_NOPROXY.html.
+
+``CURLOPT_HTTPPROXYTUNNEL`` : *number* = ``0``
+ Set the optional tunnel parameter to ``1`` to tunnel all operations through the HTTP proxy
+ (set with ``CURLOPT_PROXY``).
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_HTTPPROXYTUNNEL.html.
+
+``CONNECTTIMEOUT`` : *number* = ``0``
+ The optional maximum time in seconds that the system will wait for a connection to be established.
+ The default value means that the system will wait indefinitely.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_CONNECTTIMEOUT.html
+
+``TIMEOUT`` : *number* = ``0``
+ The optional maximum time in seconds that the system will wait for a response from the server.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_TIMEOUT.html
+
+``LOW_SPEED_LIMIT`` : *number* = ``0``
+ The optional transfer speed in bytes per second that the system considers too slow and will abort the transfer.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_LIMIT.html
+
+``LOW_SPEED_TIME`` : *number* = ``0``
+ The optional time in seconds that the system will wait for the transfer speed to be above the limit
+ set by ``LOW_SPEED_LIMIT``.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html
+
+``ASYNC_PROC_LIMIT`` : *number* = ``0``
+ The optional maximum concurrency limit for the number of contributions to be processed in a scope of
+ the database. The actual number of parallel requests may be further lowered by the hard limit specified by
+ the Replication System worker's configuration parameter (``worker``, ``num-async-loader-processing-threads``).
+ The parameter can be adjusted in real time as needed. It gets into effect immediately. Putting ``0`` as a value of
+ the parameter will effectively turn this option off as if it has never been configured for the database.
+
+ This attribute directly maps to https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html
+
+ **Note**: The parameter is available as of API version ``14``.
+
+If a request is successfully finished it returns the standard JSON object w/o any additional data but
+the standard completion status.
+
+.. _ingest-config-get:
+
+Retrieving configuration parameters
+-----------------------------------
+
+.. warning::
+ As of version ``14`` of the API, the name of the database is required to be passed in the request's query instead of
+ passing it in the JSON body. The older implementation was wrong.
+
+
+.. list-table::
+ :widths: 10 25 65
+ :header-rows: 1
+
+ * - method
+ - service
+ - query parameters
+ * - ``GET``
+ - ``/ingest/config``
+ - ``database=``
+
+Where the mandatory query parameter ``database`` specifies the name of a database affected by the operation.
+
+If the operation is successfully finished it returns an extended JSON object that has the following schema (in addition
+to the standard status and error reporting attributes):
+
+.. code-block::
+
+ { "database" : ,
+ "SSL_VERIFYHOST" : ,
+ "SSL_VERIFYPEER" : ,
+ "CAPATH" : ,
+ "CAINFO" : ,
+ "CAINFO_VAL" : ,
+ "PROXY_SSL_VERIFYHOST" : ,
+ "PROXY_SSL_VERIFYPEER" : ,
+ "PROXY_CAPATH" : ,
+ "PROXY_CAINFO" : ,
+ "PROXY_CAINFO_VAL" : ,
+ "CURLOPT_PROXY" : ,
+ "CURLOPT_NOPROXY" : ,
+ "CURLOPT_HTTPPROXYTUNNEL" : ,
+ "CONNECTTIMEOUT" : ,
+ "TIMEOUT" : ,
+ "LOW_SPEED_LIMIT" : ,
+ "LOW_SPEED_TIME" : ,
+ "ASYNC_PROC_LIMIT" :
+ }
+
+The attributes of the response object are the same as the ones described in the section :ref:`ingest-config-set`.
diff --git a/doc/ingest/api/reference/rest/controller/db-table-management.rst b/doc/ingest/api/reference/rest/controller/db-table-management.rst
new file mode 100644
index 000000000..32da34884
--- /dev/null
+++ b/doc/ingest/api/reference/rest/controller/db-table-management.rst
@@ -0,0 +1,559 @@
+Database and table management
+=============================
+
+.. _ingest-db-table-management-config:
+
+Finding existing databases and database families
+------------------------------------------------
+
+The following service pulls all configuration information of of the Replication/Ingest System, including info
+on the known database families, databases and tables:
+
+.. list-table::
+ :widths: 10 90
+ :header-rows: 0
+
+ * - ``GET``
+ - ``/replication/config``
+
+Upon successful (see :ref:`ingest-general-error-reporting`) completion of the request, the service will return an object
+that has the following schema (of which only the database and database family-related fields are shown):
+
+.. code-block:: json
+
+ {
+ "config": {
+ "database_families" : [
+ {
+ "overlap" : 0.01667,
+ "min_replication_level" : 3,
+ "num_sub_stripes" : 3,
+ "name" : "production",
+ "num_stripes" : 340
+ }
+ ],
+ "databases" : [
+ {
+ "database" : "dp01_dc2_catalogs_02",
+ "create_time" : 0,
+ "is_published" : 1,
+ "publish_time" : 1662688661000,
+ "family_name" : "production",
+ "tables" : [
+ {
+ "ang_sep" : 0,
+ "is_director" : 1,
+ "latitude_key" : "coord_dec",
+ "create_time" : 1662774817703,
+ "unique_primary_key" : 1,
+ "flag" : "",
+ "name" : "Source",
+ "director_database_name" : "",
+ "is_ref_match" : 0,
+ "is_partitioned" : 1,
+ "longitude_key" : "coord_ra",
+ "database" : "dp02_dc2_catalogs",
+ "director_table" : "",
+ "director_key2" : "",
+ "director_database_name2" : "",
+ "director_key" : "sourceId",
+ "director_table2" : "",
+ "director_table_name2" : "",
+ "is_published" : 1,
+ "director_table_name" : "",
+ "publish_time" : 1663033002753,
+ "columns" : [
+ {
+ "name" : "qserv_trans_id",
+ "type" : "INT NOT NULL"
+ },
+ {
+ "type" : "BIGINT NOT NULL",
+ "name" : "sourceId"
+ },
+ {
+ "type" : "DOUBLE NOT NULL",
+ "name" : "coord_ra"
+ },
+ {
+ "type" : "DOUBLE NOT NULL",
+ "name" : "coord_dec"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ }
+ }
+
+**Notes**:
+
+- The sample object was truncated for brevity. The actual number of families, databases, tables and columns were
+ much higher in the real response.
+- The number of attributes varies depending on a particular table type. The example above shows
+ attributes for the table ``Source``. This table is *partitioned* and is a *director* (all *director*-type tables
+ are partitioned in Qserv).
+
+
+.. _ingest-db-table-management-register-db:
+
+Registering databases
+----------------------
+
+Each database has to be registered in Qserv before one can create tables and ingest data. The following
+service of the Replication Controller allows registering a database:
+
+.. list-table::
+ :widths: 10 90
+ :header-rows: 0
+
+ * - ``POST``
+ - ``/ingest/database``
+
+The service requires a JSON object of the following schema:
+
+.. code-block::
+
+ {
+ "database" : ,
+ "num_stripes" : ,
+ "num_sub_stripes" : ,
+ "overlap" : ,
+ "auto_build_secondary_index" : ,
+ "local_load_secondary_index" :
+ }
+
+Where:
+
+``database`` : *string*
+ The required name of the database to be created.
+
+``num_stripes`` : *number*
+ The required number of stripes that was used when partitioning data of all tables to be ingested in a scope of the database.
+
+``num_sub_stripes`` : *number*
+ The required number of sub-stripes that was used when partitioning data of all tables to be ingested in a scope of the database.
+
+``overlap`` : *number*
+ The required overlap between the stripes.
+
+``auto_build_secondary_index`` : *number* = ``1``
+ The flag that specifies the desired mode for building the *director* (used to be known as the *secondary*)
+ indexes of the director tables of the catalog. The flag controls the automatic building of the indexes, where:
+
+ - ``1``: Build the index automatically during transaction commit time.
+ - ``0``: Do not build the index automatically during transaction commit time. In this case, it will be up to a workflow
+ to trigger the index building as a separated "post-ingest" action using the corresponding service:
+
+ - :ref:`ingest-director-index-build`
+
+ **Note**: Catalogs in Qserv may have more than one director table. This option applies to all such tables.
+
+.. warning::
+
+ - The service will return an error if the database with the same name already exists in the system.
+ - Values of attributes ``num_stripes``, ``num_sub_stripes`` and ``overlap`` are expected to match
+ the corresponding partitioning parameters used when partitioning all partitioned tables of the new database.
+ Note that the current implementation of the Qserv Ingest system will not validate contributions to the partitioned
+ tables to enforce this requirement. Only the structural correctness will be checked. It's up to a workflow
+ to ensure the data ingested into tables are correct.
+ - Building the *director* index during transaction commit time (for the relevant tables) may have a significant
+ impact on the performance of the transaction commit operation. The impact is proportional to the size of the
+ contributions made into the table during the transaction. This may orotolng the transaction commit time.
+ An alternative option is to build the indexes as a separated "post-ingest" action using the corresponding service:
+
+ - :ref:`ingest-director-index-build`
+
+If the operation is successfully finished (see :ref:`ingest-general-error-reporting`) a JSON object returned by the service
+will have the following attribute:
+
+.. code-block::
+
+ {
+ "database": {
+ ...
+ }
+ }
+
+The object containing the database configuration information has the same schema as it was explained earlier in section:
+
+- :ref:`ingest-db-table-management-config`
+
+
+.. _ingest-db-table-management-register-table:
+
+Registering tables
+------------------
+
+All tables, regardless if they are *partitioned* or *regular* (fully replicated on all worker nodes), have to be registered
+using the following Replication Controller's service:
+
+.. list-table::
+ :widths: 10 90
+ :header-rows: 0
+
+ * - ``POST``
+ - ``/ingest/table``
+
+The service requires a JSON object of the following schema:
+
+Where a JSON object sent to the service with the request shall describe that table. This is a schema of the object for
+the **partitioned** tables is presented below:
+
+.. code-block::
+
+ {
+ "database" : ,
+ "table" : ,
+ "is_partitioned" : ,
+ "schema" : [
+ { "name" : ,
+ "type" :
+ },
+ ...
+ ],
+ "director_table" : ,
+ "director_key" : ,
+ "director_table2" : ,
+ "director_key2" : ,
+ "latitude_key" : ,
+ "longitude_key" : ,
+ "flag" : ,
+ "ang_sep" : ,
+ "unique_primary_key" :
+ }
+
+A description of the *regular* tables has a fewer number of attributes (attributes that which are specific to the *partitioned*
+tables are missing):
+
+.. code-block::
+
+ {
+ "database" : ,
+ "table" : ,
+ "is_partitioned" : ,
+ "schema": [
+ {
+ "name" : ,
+ "type" :
+ },
+ ...
+ ]
+ }
+
+Where the attributes are:
+
+``database`` : *string*
+ The required name of the existing database.
+
+``table`` : *string*
+ The required name of a table to be created.
+
+``is_partitioned`` : *number*
+ The required type of table. Allowed values:
+
+ - ``1`` for partitioned tables (including any subtypes)
+ - ``0`` for the regular tables.
+
+``schema`` : *array*
+ The required definition of the table schema, where each entry of the array is an object with the following attributes:
+
+ - ``name``: The name of the column.
+ - ``type``: The type of the column. The type must adhere to the MySQL requirements for column types.
+
+``director_table`` : *string*
+ The name of the corresponding first (or left) *director* table. The name is required to be not empty for
+ the *dependent* tables and it has to be empty for the *director* tables. This is the only way to differentiate between
+ two types of *partitioned* tables.
+
+ **Note**: The *ref-match* tables are considered as the *dependent* tables since they have columns that are pointing
+ to the corresponding *director* tables. See attributes: ``director_key``, ``director_table2``, and ``director_key2``.
+
+``director_key`` : *string*
+ The required name of a column in a *partitioned* table. A role of the column depends on a subtype of
+ the table:
+
+ - *director*: the primary key of the table
+ - *dependent*: the foreign key pointing to the corresponding column of the *director* table
+
+``director_table2`` : *string*
+ The name of the corresponding second (or right) *director* table. The non-empty value
+ name is required for the *ref-match* tables and it has to be empty for the *director* and *dependent* tables.
+
+ **Note**: The very presence of this attribute in the input configuration would imply an intent to register
+ a "ref-match* table. In this case, non-empty values of the attributes ``director_key2`` , ``flag`` and ``ang_sep``
+ will be required in order to succeed with the registration.
+
+``director_key2`` : *string*
+ The name of a column that is associated (AKA *foreign key*) with corresponding column of the second *director* table.
+ A value of this attribute is required for and it must not be empty when registering the *ref-match* tables.
+ It will be ignored for other table types. See a description of the attribute ``director_table2``.
+
+``latitude_key`` : *string*
+ The required name of a column in a *director* table represents latitude. It's optional for the *dependent* tables.
+
+``longitude_key`` : *string*
+ The required name of a column in a *director* table represents longitude. It's optional for the *dependent* tables.
+
+``flag`` : *string*
+ The name of the special column that is required to be present on the *ref-match* tables.
+ Values of the column are populated by the tool ``sph-partition-matches`` when partitioning the input files
+ of the *ref-match* tables. The data type of this column is usually:
+
+ .. code-block:: sql
+
+ INT UNSIGNED
+
+``ang_sep`` : *double*
+ The value of the angular separation for the matched objects that is used by Qserv to process queries which
+ involve the *ref-match* tables. The value is in radians. The value is required to be non-zero for the *ref-match* tables.
+
+``unique_primary_key`` : *number* = ``0``
+ The optional flag allows to drop the uniqueness requirement for the *director* keys of the table. The parameter
+ is meant to be used for testing new table products, or for the *director* tables that won't have any dependants (child tables).
+ Allowed values:
+
+ - ``0``: The primary key is not unique.
+ - ``1``: The primary key is unique.
+
+.. warning::
+
+ - The table schema does not include definitions of indexes. Those are managed separately after the table is published.
+ The index management interface is documented in a dedicated document
+
+ - **TODO**: Managing indexes of MySQL tables at Qserv workers.
+
+ - The service will return an error if the table with the same name already exists in the system, or
+ if the database didn' exist at a time when teh request was delivered to the service.
+
+ - The service will return an error if the table schema is not correct. The schema will be checked for the correctness.
+
+.. note:: Requirements for the table schema:
+
+ - The variable-length columns are not allowed in Qserv for the *director* and *ref-match* tables. All columns of these
+ tables must have fixed lengths. These are the variable length types: ``VARCHAR``, ``VARBINARY``, ``BLOB``, ``TEXT``,
+ ``GEOMETRY`` and ``JSON``.
+
+ - The *partitioned* tables are required to have parameters ``director_key``, ``latitude_key`` and ``longitude_key``.
+ - The *director* tables are required to have non-empty column names in the parameters ``director_key``, ``latitude_key`` and ``longitude_key``.
+ - The *dependent* tables are required to have a non-empty column name specified in the parameter ``director_key``.
+ - The *dependent* tables are allowed to have empty values in the parameters ``latitude_key`` and ``longitude_key``.
+
+ - For tables where the attributes ``latitude_key`` and ``longitude_key`` are provided (either because they are required
+ of if they are optional), values must be either both non-empty or empty. An attempt to specify only one of the attribute
+ or have a non-empty value in an attribute while the other one has it empty will result in an error.
+
+ - All columns mentioned in attributes ``director_key``, ``director_key2``, ``flag``, ``latitude_key`` and ``longitude_key``
+ must be present in the table schema.
+
+ - Do not use quotes around the names or type specifications.
+
+ - Do not start the columm names with teh reserved prefix ``qserv``. This prefix is reserved for the Qserv-specific columns.
+
+An example of the schema definition for the table ``Source``:
+
+.. code-block:: json
+
+ [
+ {
+ "name" : "sourceId"
+ "type" : "BIGINT NOT NULL",
+ },
+ {
+ "name" : "coord_ra"
+ "type" : "DOUBLE NOT NULL",
+ },
+ {
+ "name" : "coord_dec"
+ "type" : "DOUBLE NOT NULL",
+ }
+ ]
+
+If the operation is successfully finished (see :ref:`ingest-general-error-reporting`) a JSON object returned by the service
+will have the following attribute:
+
+.. code-block::
+
+ {
+ "database": {
+ ...
+ }
+ }
+
+The object will contain the updated database configuration information that will also include the new table.
+The object will have the same schema as it was explained earlier in section:
+
+- :ref:`ingest-db-table-management-config`
+
+**Notes on the table names**:
+
+- Generally, the names of the tables must adhere to the MySQL requirements for identifiers
+ as explained in:
+
+ - https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html
+
+- The names of identifiers (including tables) in Qserv are case-insensitive. This is not the general requirement
+ in MySQL, where the case sensitivity of identifiers is configurable one way or another. This requirement
+ is enforced by the configuration of MySQL in Qserv.
+
+- The length of the name should not exceed 64 characters as per:
+
+ - https://dev.mysql.com/doc/refman/8.0/en/identifier-length.html
+
+- The names should **not** start with the prefix ``qserv``. This prefix is reserved for the Qserv-specific tables.
+
+
+.. _ingest-db-table-management-publish-db:
+
+Publishing databases
+--------------------
+
+Databases are published (made visible to Qserv users) by calling this service:
+
+.. list-table::
+ :widths: 10 90
+ :header-rows: 0
+
+ * - ``PUT``
+ - ``/ingest/database/:database``
+
+The name of the database is provided as a parameter ``database`` of the resource path. There are a few optional
+parameters to be sent in the JSON body of the request:
+
+.. code-block::
+
+ {
+ "consolidate_secondary_index" : ,
+ "row_counters_deploy_at_qserv" :
+ }
+
+Where:
+
+``consolidate_secondary_index`` : *number* = ``0``
+ The optional parameter that controls the final format of all the *director* index tables of the database.
+ Normally, the *director* indexes are MySQL-partitioned tables. If the value of this optional parameter is
+ not ``0`` then the Ingest System will consolidate the MySQL partitions and turn the tables into the monolitical form.
+
+ .. warning::
+
+ Depending on the scale of the catalog (sizes of the affected tables), this operation may be quite lengthy (up to many hours).
+ Besides, based on the up to the date experience with using the MySQL-partitioned director indexes, the impact of the partitions
+ on the index's performance is rather negligible. So, it's safe to ignore this option in most but very special cases that are not
+ discussed by the document.
+
+ One can find more info on the MySQL partitioning at:
+
+ - https://dev.mysql.com/doc/refman/8.0/en/partitioning.html
+
+``row_counters_deploy_at_qserv`` : *number* = ``0``
+ This optional flag that triggers scanning and deploying the row counters as explained at:
+
+ - :ref:`admin-row-counters` (ADMIN)
+ - :ref:`ingest-row-counters-deploy` (REST)
+
+ To trigger this operation the ingest workflow should provide a value that is not ``0``. In this case the row counters
+ collection service will be invoked with the following combination of parameters:
+
+ .. list-table::
+ :widths: 50 50
+ :header-rows: 1
+
+ * - attr
+ - value
+ * - ``overlap_selector``
+ - ``CHUNK_AND_OVERLAP``
+ * - ``force_rescan``
+ - ``1``
+ * - ``row_counters_state_update_policy``
+ - ``ENABLED``
+ * - ``row_counters_deploy_at_qserv``
+ - ``1``
+
+.. warning::
+
+ The row counters deployment is a very resource-consuming operation. It may take a long time to complete
+ depending on the size of the catalog. This will also delay the catalog publiushing stage of an ingest compaign.
+ A better approach is to deploy the row counters as the "post-ingest" operation as explained in:
+
+ - (**TODO** link) Deploying row counters as a post-ingest operation
+
+.. note::
+
+ The catalogs may be also unpublished to add more tables. The relevant REST service is documented in:
+
+ - (**TODO** link) Un-publishing databases to allow adding more tables
+
+
+.. _ingest-db-table-management-unpublish-db:
+
+Un-publishing databases to allow adding more tables
+---------------------------------------------------
+
+Unpublished databases as well as previously ingested tables will be still visible to users of Qserv.
+The main purpose of this operation is to allow adding new tables to the existing catalogs.
+The new tables won't be seen by users until the catalog is published back using the following REST service:
+
+- :ref:`ingest-db-table-management-publish-db`
+
+Databases are un-published by calling this service:
+
+.. list-table::
+ :widths: 10 90
+ :header-rows: 0
+
+ * - ``PUT``
+ - ``/replication/config/database/:database``
+
+The name of the database is provided in a parameter ``database`` of the resource. The only mandatory parameter
+to be sent in the JSON body of the request is:
+
+``admin_auth_key`` : *string*
+ The administrator-level authentication key that is required to publish the database.
+ The key is used to prevent unauthorized access to the service.
+
+ **Note**: The key is different from the one used to publish the database. The eleveated privileges
+ are needed to reduce risks of disrupting user access to the previously loaded and published databases.
+
+
+.. _ingest-db-table-management-delete:
+
+Deleting databases and tables
+-----------------------------
+
+These services can be used for deleting non-*published* (the ones that are still ingested) as well as *published* databases,
+or tables, including deleting all relevant persistent structures from Qserv:
+
+.. list-table::
+ :widths: 10 90
+ :header-rows: 0
+
+ * - ``DELETE``
+ - | ``/ingest/database/:database``
+ | ``/ingest/table/:database/:table``
+
+To delete a non-*published* database (or a table from such database) a client has to provide the normal level authentication
+key ``auth_key`` in a request to the service:
+
+.. code-block::
+
+ { "auth_key" :
+ }
+
+The name of the databases affected by the operation is specified at the resource's path.
+
+Deleting databases (or tables from those databases) that have already been published requires a user to have
+elevated administrator-level privileges. These privileges are associated with the authentication key ``admin_auth_key``
+to be sent with a request instead of ``auth_key``:
+
+.. code-block::
+
+ { "admin_auth_key" :
+ }
+
+Upon successful completion of the request (for both above-mentioned states of the database), the service will return the standard
+response as explained in the section mentoned below. After that, the database (or the table, depending on a scope of a request)
+name can be reused for further ingests if needed.
+
+- :ref:`ingest-general-error-reporting`
+
diff --git a/doc/ingest/api/reference/rest/controller/director-index.rst b/doc/ingest/api/reference/rest/controller/director-index.rst
new file mode 100644
index 000000000..edb0224b0
--- /dev/null
+++ b/doc/ingest/api/reference/rest/controller/director-index.rst
@@ -0,0 +1,139 @@
+Director Index Management
+=========================
+
+.. _ingest-director-index-build:
+
+(Re-)building the Index
+-----------------------
+
+.. note:: API version notes:
+
+ - As of version ``21``, the service can no longer be used to (re-)build indexes of all *director*
+ tables of a catalog. It's not required to provide the name of the affected table in the parameter ``director_table``.
+
+ - As of version ``22``, the service no longer support the option ``allow_for_published``. Any attempts to specify
+ the option will result in a warning reported by the service back to a client. The service will ignore the option.
+
+.. warning::
+ Be advised that the amount of time needed to build an index of a large-scale catalog may be quite large.
+ The current implementation of the secondary index is based on MySQL's InnoDB table engine. The insert
+ time into this B-Tree table has logarithmic performance. It may take many hours to build catalogs of
+ billions of objects. In some earlier tests, the build time was 20 hours for a catalog of 20 billion objects.
+
+
+The service of the **Master Replication Controller** builds or rebuilds (if needed) the *director* (used to be known as
+the *secondary*) index table of a database. The target table must be *published* at the time of this operation.
+
+.. list-table::
+ :widths: 10 90
+ :header-rows: 0
+
+ * - ``POST``
+ - ``/ingest/index/secondary``
+
+The request object has the following schema:
+
+.. code-block::
+
+ { "database" : ,
+ "director_table" : ,
+ "rebuild" : ,
+ "local" :
+ }
+
+Where:
+
+``database`` : *string*
+ The required name of a database affected by the operation.
+
+``director_table`` : *string*
+ The required name of the *director* table for which the index is required to be (re-)built.
+
+``rebuild`` : *number* = ``0``
+ The optional flag that allows recreating an existing index. If the value is set to ``0`` the service
+ will refuse to proceed with the request if the index already exists. Any other value would tell the service
+ to drop (if exists) the index table before re-creating and re-populating it with entries.
+
+``local`` : *number* = ``0``
+ The optional flag that tells the service how to ingest data into the index table, where:
+
+ - ``0``: Index contributions are required to be directly placed by the Replication/Ingest System at a location
+ that is directly accessible by the MySQL server hosting the index table. This could be either some local folder
+ of a host where the service is being run or a folder located at a network filesystem mounted on the host.
+ Once a file is in place, it would be ingested into the destination table using this protocol:
+
+ .. code-block:: sql
+
+ LOAD DATA INFILE ...
+
+ **Note**: Be aware that this option may not be always possible (or cause complications) in Kubernetes-based
+ deployments of Qserv.
+
+ - ``1`` (or any other numeric value): Index contributions would be ingested into the table using this protocol:
+
+ .. code-block:: sql
+
+ LOAD DATA LOCAL INFILE ...
+
+ **Note**: Files would be first copied by MySQL at some temporary folder owned by the MySQL service before being
+ ingested into the table. This option has the following caveats:
+
+ - The protocol must be enabled in the MySQL server configuration by setting a system variable: ``local_infile=1``.
+ - The temporary folder of the MySQL server is required to have sufficient space to temporarily accommodate index
+ contribution files before they'd be loaded into the table. In the worst-case scenario, there should be enough
+ space to accommodate all contributions of a given catalog. One could make a reasonable estimate for the latter
+ by knowing the total number of rows in the director table of the catalog, the size of the primary
+ key (typically the ``objectId`` column) of the table, as well as types of the ``chunk`` and ``subChunk``
+ columns (which are usually the 32-bit integer numbers in Qserv).
+ - This ingest option would also affect (lower) the overall performance of the operation due to additional
+ data transfers required for copying file contributions from a location managed by the **Master Replication Controller**
+ to the temporary folder of the MySQL server.
+
+If the operation succeeded, the service will respond with the default JSON object which will not carry any additional
+attributes on top of what's mandated in :ref:`ingest-general-error-reporting`.
+
+In case of errors encountered during an actual attempt to build the index was made, the object may have a non-trivial
+value of the ``error_ext``. The object wil carry specific reasons for the failures. The schema of the object
+is presented below:
+
+.. code-block::
+
+ "error_ext" : {
+
: {
+ : {
+ : ,
+ ...
+ },
+ },
+ ...
+ }
+
+Where:
+
+``table`` : *string*
+ The placeholder for the name of the director table.
+
+``worker`` : *string*
+ The placeholder for the name of the worker service that failed to build the index.
+
+``chunk`` : *number*
+ The placeholder for the chunk number.
+
+``error`` : *string*
+ The placeholder for the error message.
+
+Here is an example of how this object might look like:
+
+.. code-block::
+
+ "error_ext" : {
+ "object" : {
+ "qserv-db01" : {
+ 122 : "Failed to connect to the worker service",
+ 3456 : "error: Table 'tes96__Object' already exists, errno: 1050",
+ },
+ "qserv-db23" : {
+ 123 : "Failed to connect to the worker service"
+ }
+ }
+ }
diff --git a/doc/ingest/api/reference/rest/controller/index.rst b/doc/ingest/api/reference/rest/controller/index.rst
new file mode 100644
index 000000000..427d79ace
--- /dev/null
+++ b/doc/ingest/api/reference/rest/controller/index.rst
@@ -0,0 +1,14 @@
+#############################
+Master Replication Controller
+#############################
+
+.. toctree::
+ :maxdepth: 4
+
+ config
+ db-table-management
+ trans-management
+ table-location
+ info
+ director-index
+ row-counters
diff --git a/doc/ingest/api/reference/rest/controller/info.rst b/doc/ingest/api/reference/rest/controller/info.rst
new file mode 100644
index 000000000..7194a567f
--- /dev/null
+++ b/doc/ingest/api/reference/rest/controller/info.rst
@@ -0,0 +1,137 @@
+Information services
+====================
+
+.. _ingest-info-chunks:
+
+Chunk disposition
+-----------------
+
+.. warning::
+ Do not use this service for the chunk placement decisions during catalog ingestion. The service is for
+ informational purposes only.
+
+The service of the **Master Replication Controller** return information about the chunk *replicas* in a scope of a given database:
+
+.. list-table::
+ :widths: 10 15 75
+ :header-rows: 1
+
+ * - method
+ - service
+ - query parameters
+ * - ``GET``
+ - ``/ingest/chunks``
+ - ``database=``
+
+Where:
+
+``name`` : *string*
+ The required name of a database affected by the operation.
+
+The resulting object has the following schema:
+
+.. code-block::
+
+ {
+ "replica": [
+ {
+ "chunk" : ,
+ "worker": ,
+ "table" : {
+ : {
+ "overlap_rows" : ,
+ "overlap_data_size" : ,
+ "overlap_index_size" : ,
+ "rows" : ,
+ "data_size" : ,
+ "index_size" :
+ },
+ ...
+ }
+ },
+ ...
+ ]
+ }
+
+Where:
+
+``replica`` : *array*
+ A collection of chunk **replicas**, where each object representes a chunk replica. Replicas of a chunk
+ are essentially the same chunk, but placed on different workers.
+
+``chunk`` : *number*
+ The chunk number.
+
+``worker`` : *string*
+ The unique identifier of a worker where the chunk replica is located.
+
+``table`` : *object*
+ The object with the information about the chunk replica in the scope of
+ a particular *partitioned* table.
+
+ **Attention**: The current implementation is incomplete. It will return ``0`` for all attributes
+ of the table object.
+
+``overlap_rows`` : *number*
+ The number of rows in the chunk's overlap table.
+
+``overlap_data_size`` : *number*
+ The number of bytes in the chunk's overlap table (measured by the size of the corresponding file).
+
+``overlap_index_size`` : *number*
+ The number of bytes in the index of the chunk's overlap table (measured by the size
+ of the corresponding file).
+
+``rows`` : *number*
+ The number of rows in the chunk table.
+
+``data_size`` : *number*
+ The number of bytes in the chunk table (measured by the size of the corresponding file).
+
+``index_size`` : *number*
+ The number of bytes in the index of the chunk table (measured by the size of
+ the corresponding file).
+
+.. _ingest-info-contrib-requests:
+
+Status of the contribution request
+----------------------------------
+
+The service of the **Master Replication Controller** returns information on a contribution request:
+
+.. list-table::
+ :widths: 10 15 75
+ :header-rows: 1
+
+ * - method
+ - service
+ - query parameters
+ * - ``GET``
+ - ``/ingest/trans/contrib/:id``
+ - | ``include_warnings=<0|1>``
+ | ``include_retries=<0|1>``
+
+Where:
+
+``id`` : *number*
+ The required unique identifier of the contribution request that was submitted
+ to a Worker Ingest service earlier.
+
+``include_warnings`` : *number* = ``0``
+ The optional flag telling the service to include warnings into the response. Any value
+ that is not ``0`` is considered as ``1``, meaning that the warnings should be included.
+
+``include_retries`` : *number* = ``0``
+ The optional flag telling the service to include retries into the response. Any value
+ that is not ``0`` is considered as ``1``, meaning that the retries should be included.
+
+The resulting object has the following schema:
+
+.. code-block::
+
+ { "contribution" :