Feature/metax integration

.env.example

@@ -31,3 +31,9 @@ DOI_API=http://mockdoi:8001/dois
 DOI_PREFIX=10.xxxx
 DOI_USER=user
 DOI_KEY=key
+DISCOVERY_URL=https://etsin.demo.fairdata.fi/dataset/
+
+# metax
+METAX_USER=sd
+METAX_PASS=demo_pass
+METAX_URL=http://mockmetax:8002

.github/config/.wordlist.txt

@@ -165,6 +165,7 @@ dockerfile
 docstrings
 doi
 doiinfo
+dois
 dt
 dzongkha
 ean
@@ -196,6 +197,7 @@ experimentref
 experimenttype
 externalid
 extrainfo
+fairdata
 faire
 familyname
 faroese
@@ -342,6 +344,9 @@ metagenomic
 metagenomics
 metatranscriptome
 metatranscriptomic
+metax
+metaxidentifier
+metaxservicehandler
 methylation
 methylcytidine
 mf
@@ -609,6 +614,7 @@ txt
 ui
 ujson
 umi
+uncomment
 unencryptedchecksum
 uniqueitems
 unlocalised

.github/workflows/int.yml

@@ -43,10 +43,7 @@ jobs:
       env:
         BASE_URL: http://localhost:5430
         OIDC_URL: http://localhost:8000
-        MONGO_HOST: localhost:27017
-        MONGO_DATABASE: default
-        MONGO_AUTHDB: admin
-
+
     - name: Collect logs from docker
       if: ${{ failure() }}
       run: docker-compose logs --no-color -t > tests/dockerlogs || true
@@ -78,6 +75,7 @@ jobs:
         MONGO_HOST: localhost:27017
         MONGO_DATABASE: default
         MONGO_AUTHDB: admin
+        MONGO_SSL: True
 
     - name: Run Integration test
       run: |
@@ -86,7 +84,7 @@ jobs:
         BASE_URL: http://localhost:5430
         OIDC_URL: http://localhost:8000
         MONGO_SSL: True
-        
+
     - name: Collect logs from docker
       if: ${{ failure() }}
       run: docker-compose logs --no-color -t > tests/dockerlogs || true

CHANGELOG.md

@@ -9,15 +9,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- Add patching of folders after object save and update operations
+- Submission endpoint update #371
+  - Adds mandatory query parameter `folder` for submit endpoint POST 
+  - On actions add and modify object is added or updated to folder(submission) where it belongs with it's accession ID, schema, submission type, title and filename
+  - Adds metax integration to submit endpoint
+- Integration with Metax service #356
+  - Adds new local container for testing against mocked Metax API
+  - Introduces new env vars: METAX_USER, METAX_PASS, METAX_URL for connection to Metax service
+  - Introduces new env var DISCOVERY_URL for creating link to dataset inside Fairdata SD catalog
+  - Adds new key metaxIdentifier to Study and Dataset collections containing metax id returned from Metax API
+  - Adds new handler MetaxServiceHandler to take care of mapping Submitter metadata to Metax metadata and to connect to Metax API
+- Add patching of folders after object save and update operations #354
+  - Adds mandatory query parameter `folder` for objects endpoint POST 
   - Object is added or updated to folder(submission) where it belongs with it's accession ID, schema, submission type, title and filename in the case of CSV and XML upload
   - Adds configuration for mypy linting to VScode devcontainer setup
 - Templates API #256
   - use `ujson` as default json library
-- Creating draft Datacite DOI for folders #257
+- Creating draft Datacite DOI for folders #257 #332
   - created a mock web app, which would act similarly to DataCite REST API
   - altered `publish_folder` endpoint so that `extraInfo` containing the DOI data is added upon publishing
   - added `datePublished` key to folders which takes in the date/time, when folder is published
+- DOI Publishing and deletion to Datacite #332 #369
+  - create draft DOIs for both Study and Datasets and add them to the folder `extraInfo` when published
+  - delete draft DOIs on object delete
+  - update DOI info at Datacite when folder is published
 - VScode Dev environment #287
   - Add VS Code development container
   - Update docker for development
@@ -46,6 +61,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - new endpoint `GET /templates` to replace `GET /users/current` `{"templates":[...]}`
   - new JSON keys `index` and `tags` to `PATCH /templates/schema/templateId`, same values as were previously used in `PATCH /user` which is now removed
   - WARNING: breaking change that requires fresh database, because "project" is new information that did not exist before, and it can't be migrated to existing user-owned hierarchy
+- Multilevel add patch objects to support `/extraInfo/datasetIdentifiers/-` which needs dot notation for mongodb to work e.g. `extraInfo.datasetIdentifiers` #332
 
 ### Changed
 
@@ -59,7 +75,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - README updated with tox command, development build instructions, and prettify Dockerfile.
 - Update ENA XML and JSON schemas #299
 - Github actions changed the use of https://git.io/misspell to rojopolis/spellcheck-github-actions #316
-- Separated most of the handlers to own files inside the handlers folder #319
+- Separated most of the handlers to own files inside the handlers folder #319 
+- allow inserting only one study in folder #332
+- JSON schemas #332
+   - introduce `keywords` required for Metax in `doiInfo`
+   - dataset `description` and study `studyAbstract` are now mandatory
+- `keywords` will be comma separated values, that will require splitting when adding to Metax API
 
 ### Fixed
 

README.md

@@ -35,7 +35,7 @@ Server can then be found from `http://localhost:5430`.
   - Suitable mongodb instance can be launched with Docker by running `docker-compose up database`
 - After installing and setting up database, server can be launched with `metadata_submitter`
 
-If you also need frontend for development, check out [frontend repository](https://github.com/CSCfi/metadata-submitter-frontend/).
+If you also need frontend for development, check out [frontend repository](https://github.com/CSCfi/metadata-submitter-frontend/). You will also need to uncomment `REDIRECT_URL` environment variable from .env file.
 
 ## Tests
 

docker-compose-tls.yml

@@ -4,6 +4,7 @@ services:
     build:
       dockerfile: Dockerfile-dev
       context: .
+      target: develop
     image: cscfi/metadata-submitter-dev
     container_name: "metadata_submitter_backend_dev"
     volumes:
@@ -14,6 +15,7 @@ services:
       - database
       - mockauth
       - mockdoi
+      - mockmetax
     restart: on-failure
     environment:
       - "MONGO_HOST=${MONGO_HOST}"
@@ -34,6 +36,10 @@ services:
       - "DOI_PREFIX=${DOI_PREFIX}"
       - "DOI_USER=${DOI_USER}"
       - "DOI_KEY=${DOI_KEY}"
+      - "DISCOVERY_URL=${DISCOVERY_URL}"
+      - "METAX_USER=${METAX_USER}"
+      - "METAX_PASS=${METAX_PASS}"
+      - "METAX_URL=${METAX_URL}"
   database:
     image: "mongo"
     container_name: "metadata_submitter_database_dev"
@@ -54,6 +60,7 @@ services:
     build:
       dockerfile: Dockerfile-dev
       context: .
+      target: develop
     image: cscfi/metadata-submitter-dev
     environment:
       - "LOG_LEVEL=${LOG_LEVEL}"
@@ -71,6 +78,7 @@ services:
     build:
       dockerfile: Dockerfile-dev
       context: .
+      target: develop
     image: cscfi/metadata-submitter-dev
     hostname: mockdoi
     expose:
@@ -80,5 +88,19 @@ services:
     volumes:
       - ./tests/integration/mock_doi_api.py:/mock_doi_api.py
     entrypoint: ["python", "/mock_doi_api.py", "0.0.0.0", "8001"]
+  mockmetax:
+    build:
+      dockerfile: Dockerfile-dev
+      context: .
+      target: develop
+    image: cscfi/metadata-submitter-dev
+    hostname: mockmetax
+    expose:
+      - 8002
+    ports:
+      - 8002:8002
+    volumes:
+      - ./tests/integration/mock_metax_api.py:/mock_metax_api.py
+    entrypoint: ["python", "/mock_metax_api.py", "0.0.0.0", "8002"]
 volumes:
-  data: 
+  data:

docker-compose.yml

@@ -13,6 +13,7 @@ services:
       - database
       - mockauth
       - mockdoi
+      - mockmetax
     restart: on-failure
     environment:
       - "MONGO_HOST=${MONGO_HOST}"
@@ -29,6 +30,10 @@ services:
       - "DOI_PREFIX=${DOI_PREFIX}"
       - "DOI_USER=${DOI_USER}"
       - "DOI_KEY=${DOI_KEY}"
+      - "DISCOVERY_URL=${DISCOVERY_URL}"
+      - "METAX_USER=${METAX_USER}"
+      - "METAX_PASS=${METAX_PASS}"
+      - "METAX_URL=${METAX_URL}"
   database:
     image: "mongo"
     container_name: "metadata_submitter_database_dev"
@@ -60,7 +65,7 @@ services:
       - 8000:8000
     volumes:
       - ./tests/integration/mock_auth.py:/mock_auth.py
-    entrypoint: [ "python", "/mock_auth.py", "0.0.0.0", "8000" ]
+    entrypoint: ["python", "/mock_auth.py", "0.0.0.0", "8000"]
   mockdoi:
     build:
       dockerfile: Dockerfile-dev
@@ -74,6 +79,20 @@ services:
       - 8001:8001
     volumes:
       - ./tests/integration/mock_doi_api.py:/mock_doi_api.py
-    entrypoint: [ "python", "/mock_doi_api.py", "0.0.0.0", "8001" ]
+    entrypoint: ["python", "/mock_doi_api.py", "0.0.0.0", "8001"]
+  mockmetax:
+    build:
+      dockerfile: Dockerfile-dev
+      context: .
+      target: develop
+    image: cscfi/metadata-submitter-dev
+    hostname: mockmetax
+    expose:
+      - 8002
+    ports:
+      - 8002:8002
+    volumes:
+      - ./tests/integration/mock_metax_api.py:/mock_metax_api.py
+    entrypoint: ["python", "/mock_metax_api.py", "0.0.0.0", "8002"]
 volumes:
   data:

docs/specification.yml

@@ -21,6 +21,14 @@ paths:
       tags:
         - Submission
       summary: XML submission endpoint, will also trigger validation.
+      parameters:
+        - name: folder
+          in: query
+          schema:
+            type: string
+          description: The folder ID where object belongs to.
+          required: true
+          example: "folder=12345"
       requestBody:
         content:
           multipart/form-data:
@@ -293,11 +301,6 @@ paths:
         - Submission
       summary: Submit data to a specific schema
       parameters:
-        - in: query
-          name: folder
-          schema:
-            type: string
-          description: The folder ID where object belongs to.
         - name: schema
           in: path
           description: Name of the Metadata schema.

metadata_backend/api/handlers/common.py

@@ -15,7 +15,7 @@
 
 async def multipart_content(
     req: Request, extract_one: bool = False, expect_xml: bool = False
-) -> Tuple[List[Tuple[Any, str]], str, str]:
+) -> Tuple[List[Tuple[Any, str, str]], str]:
     """Get content(s) and schema type(s) of a multipart request (from either csv or xml format).
 
     Note: for multiple files support check: https://docs.aiohttp.org/en/stable/multipart.html#hacking-multipart
@@ -26,8 +26,8 @@ async def multipart_content(
     :raises: HTTPBadRequest for multiple different reasons
     :returns: content and schema type for each uploaded file and file type of the upload
     """
-    xml_files: List[Tuple[str, str]] = []
-    csv_files: List[Tuple[Dict, str]] = []
+    xml_files: List[Tuple[str, str, str]] = []
+    csv_files: List[Tuple[Dict, str, str]] = []
     try:
         reader = await req.multipart()
     except AssertionError:
@@ -59,20 +59,20 @@ async def multipart_content(
         if expect_xml or part.headers[hdrs.CONTENT_TYPE] == "text/xml":
             content, schema_type = await _extract_upload(part)
             _check_xml(content)
-            xml_files.append((content, schema_type))
+            xml_files.append((content, schema_type, filename))
         elif part.headers[hdrs.CONTENT_TYPE] == "text/csv":
             content, schema_type = await _extract_upload(part)
             _check_csv(content)
             csv_content = CSVToJSONParser().parse(schema_type, content)
             for row in csv_content:
-                csv_files.append((row, schema_type))
+                csv_files.append((row, schema_type, filename))
         else:
             reason = "Submitted file was not proper XML nor CSV."
             LOG.error(reason)
             raise web.HTTPBadRequest(reason=reason)
 
     # Return extracted content
-    return _get_content_with_type(xml_files, csv_files) + (filename,)
+    return _get_content_with_type(xml_files, csv_files)
 
 
 async def _extract_upload(part: BodyPartReader) -> Tuple[str, str]:
@@ -137,8 +137,8 @@ def _check_xml(content: str) -> bool:
 
 
 def _get_content_with_type(
-    xml_files: List[Tuple[str, str]], csv_files: List[Tuple[Dict, str]]
-) -> Tuple[List[Tuple[Any, str]], str]:
+    xml_files: List[Tuple[str, str, str]], csv_files: List[Tuple[Dict, str, str]]
+) -> Tuple[List[Tuple[Any, str, str]], str]:
     """Return either list of XML or CSV files with the file type info.
 
     :param xml_files: List of xml contents with schema types