Merge pull request #64 from sanger/develop

Prepare release of version 1.0.0

.github/workflows/check_release_version.yml

@@ -14,7 +14,7 @@ jobs:
 
       - name: Get specific changed files
         id: changed-files-specific
-        uses: tj-actions/changed-files@v23
+        uses: tj-actions/changed-files@v41
         with:
           files: |
             .release-version

.release-version

@@ -1 +1 @@
-0.3.0
+1.0.0

Pipfile

@@ -20,7 +20,7 @@ responses = "*"
 colorlog = "~=6.7"
 more-itertools = "~=9.0"
 python-dotenv = "~=0.21"
-requests = "~=2.28"
+requests = "~=2.31"
 slackclient = "~=2.9"
 lab-share-lib = {editable = false,git = 'https://github.com/sanger/lab-share-lib.git',ref = 'v0.1.6'}
 requests-mock = "*"

README.md

@@ -2,7 +2,6 @@
 
 Rabbitmq consumer for TOL data input
 
-
 ## Getting Started
 
 The following tools are required for development:
@@ -13,48 +12,80 @@ Use pyenv or something similar to install the version of python
 defined in the `Pipfile`:
 
 ```bash
-    brew install pyenv
-    pyenv install <python_version>
+brew install pyenv
+pyenv install <python_version>
 ```
-        
+
 Use pipenv to install the required python packages for the application and development:
 
 ```bash
-     pipenv install --dev
+pipenv install --dev
 ```
 
-
 ### Setting up with Docker
 
 If you want to setup a local development environment with Docker please check
 the instructions in [SETUP_DOCKER.md](SETUP_DOCKER.md)
 
-
 ## Running
 
 1. Enter the python virtual environment using:
-```bash
+
+    ```bash
     pipenv shell
-```
+    ```
 
 1. Run the app using:
 
-```bash
+    ```bash
     python main.py
-```
+    ```
 
 ## Testing
 
 Run the tests using pytest (flags are for verbose and exit early):
 
 ```bash
-    python -m pytest -vx
+python -m pytest -vx
 ```
 
-
 ## Deployment
 
 This project uses a Docker image as the unit of deployment. Update `.release-version` with
 major/minor/patch. On merging a pull request into *develop* or *master*, a release will be created
 along with the Docker image associated to that release.
 
+## Snappy
+
+If when you install the dependencies and you see the following error:
+
+```stdout
+[pipenv.exceptions.InstallError]:       src/snappy/snappymodule.cc:33:10: fatal error: 'snappy-c.h' file not found
+[pipenv.exceptions.InstallError]:       #include <snappy-c.h>
+[pipenv.exceptions.InstallError]:                ^~~~~~~~~~~~
+[pipenv.exceptions.InstallError]:       1 error generated.
+[pipenv.exceptions.InstallError]:       error: command '/usr/bin/clang' failed with exit code 1
+[pipenv.exceptions.InstallError]:       [end of output]
+[pipenv.exceptions.InstallError]:
+[pipenv.exceptions.InstallError]:   note: This error originates from a subprocess, and is likely not a problem with pip.
+[pipenv.exceptions.InstallError]:   ERROR: Failed building wheel for python-snappy
+[pipenv.exceptions.InstallError]: ERROR: Could not build wheels for python-snappy, which is required to install pyproject.toml-based projects
+ERROR: Couldn't install package: {}
+ Package installation failed...
+```
+
+You need to install snappy:
+
+```bash
+brew install snappy
+```
+
+Ensure the `include` and `lib` directories of `homebrew` are set in environment variables.
+You might want to add these to your `~/.zshrc` file:
+
+```bash
+export CPPFLAGS="-I$(brew --prefix)/include"
+export LDFLAGS="-L$(brew --prefix)/lib"
+```
+
+![TOL Labware Production Flow - Architecture](https://github.com/sanger/tol-lab-share/assets/519327/5356846a-6d9b-4b8d-8ffb-af26d0776222)

SETUP_DOCKER.md

@@ -1,47 +1,53 @@
 ## Setting up a complete development environment from scratch
 
 1. Start dependent services: RabbitMQ and RedPanda
-```bash
-  ./docker/dependencies/up.sh
-```
 
-2. Setup RabbitMQ configuration (queues, etc). You may need to wait 30 seconds from the previous command
+    ```bash
+    ./docker/dependencies/up.sh
+    ```
+
+1. Setup RabbitMQ configuration (queues, etc). You may need to wait 30 seconds from the previous command
 to run this one as it requires Rabbitmq to have started completely:
-```bash
+
+    ```bash
     python setup_dev_rabbit.py
-```
+    ```
 
-3. Load Redpanda schemas:
-```bash
+1. Load Redpanda schemas:
+
+    ```bash
     ./schemas/push.sh http://localhost:8081 redpanda-test
-```
+    ```
+
+1. Build docker image
 
-4. Build docker image
-```bash
+    ```bash
     docker build . -t tol-lab-share:develop
-```
+    ```
 
-5. Create .env file with contents
-```
-SETTINGS_MODULE=tol_lab_share.config.defaults
-LOCALHOST=host.docker.internal
-```
+1. Create .env file with contents
 
-6. Start interactive bash in docker container
-```bash
+    ```text
+    SETTINGS_MODULE=tol_lab_share.config.defaults
+    LOCALHOST=host.docker.internal
+    ```
+
+1. Start interactive bash in docker container
+
+    ```bash
     docker run -ti -v $(pwd):/code --env-file=.env --entrypoint bash tol-lab-share:develop
-```
+    ```
 
-7. Start the consumer service (inside the previous bash)
-```bash
+1. Start the consumer service (inside the previous bash)
+
+    ```bash
     pipenv run python main.py
-```
+    ```
 
 After this you should have:
 
 * Consumer (python main.py) running connected to Rabbitmq queue
 * Rabbitmq service running in http://localhost:8080/ with user/password: admin/development
 * Redpanda API service running in local in http://localhost:8081/
 
-
-If you want to perform any changes in code, you can kill the consumer with Control-C, modify the code in local and then restart the consumer again using the same command
+If you want to perform any changes in code, you can kill the consumer with Control-C, modify the code in local and then restart the consumer again using the same command

schemas/README.md

@@ -1,32 +1,37 @@
 How to publish schemas
 ----------------------
 
+1. Ensure `jq` is installed:
+
+    ```bash
+    brew install jq
+    ```
 
 1. Run the command:
 
-```bash
-   push.sh <REDPANDA_URL> <API_KEY>
-```
+    ```bash
+    push.sh <REDPANDA_URL> <API_KEY>
+    ```
 
-where:
+    where:
 
-```
-  <REDPANDA_URL>: URL to connect to RedPanda where the schemas will be uploaded
-  <API_KEY>: secret key with write permission for redpanda
-```
+    ```text
+    <REDPANDA_URL>: URL to connect to RedPanda where the schemas will be uploaded
+    <API_KEY>: secret key with write permission for redpanda
+    ```
 
 How to remove last schemas created
 ----------------------------------
 
-1. Run the command:
+Run the command:
 
 ```bash
-   remove_all.sh <REDPANDA_URL> <API_KEY>
+remove_all.sh <REDPANDA_URL> <API_KEY>
 ```
 
 where:
 
-```
-  <REDPANDA_URL>: URL to connect to RedPanda where the schemas will be uploaded
-  <API_KEY>: secret key with write permission for redpanda
+```text
+<REDPANDA_URL>: URL to connect to RedPanda where the schemas will be uploaded
+<API_KEY>: secret key with write permission for redpanda
 ```

tests/conftest.py

@@ -138,7 +138,7 @@ def valid_sample():
         "countryOfOrigin": "United Kingdom",
         "genomeSize": "14",
         "accessionNumber": "EE1234",
-        "sampleCollectionDateUtc": datetime.now(),
+        "sampleCollectionDateUtc": datetime.utcnow(),
         "costCode": "S1234",
         "shearedFemtoFragmentSize": "8",
         "postSPRIConcentration": "9",
@@ -147,7 +147,7 @@ def valid_sample():
         "finalNanoDrop230": "200",
         "finalNanoDrop": "150",
         "shearingAndQCComments": "",
-        "dateSubmittedUTC": datetime.now(),
+        "dateSubmittedUTC": datetime.utcnow(),
         "priorityLevel": "Medium",
     }
 
@@ -171,15 +171,15 @@ def invalid_sample():
         "genomeSize": "14",
         "accessionNumber": "EE1234",
         "costCode": 1234,
-        "sampleCollectionDateUtc": datetime.now(),
+        "sampleCollectionDateUtc": datetime.utcnow(),
         "shearedFemtoFragmentSize": "8",
         "postSPRIConcentration": "9",
         "postSPRIVolume": "10",
         "finalNanoDrop280": 200,
         "finalNanoDrop230": 200,
         "finalNanoDrop": "150",
         "shearingAndQCComments": "",
-        "dateSubmittedUTC": datetime.now(),
+        "dateSubmittedUTC": datetime.utcnow(),
         "priorityLevel": "Medium",
     }
 

tests/data/examples_create_labware_messages.py

@@ -1,9 +1,9 @@
 from datetime import datetime
-from typing import Any, Dict
+from typing import Any
 
-TEST_CREATE_LABWARE_MSG_OBJECT: Dict[str, Any] = {
+TEST_CREATE_LABWARE_MSG_OBJECT: dict[str, Any] = {
     "messageUuid": "b01aa0ad-7b19-4f94-87e9-70d74fb8783c".encode(),
-    "messageCreateDateUtc": datetime.now(),
+    "messageCreateDateUtc": datetime.utcnow(),
     "labware": {
         "labwareType": "Plate12x8",
         "barcode": "BARCODE001",
@@ -24,7 +24,7 @@
                 "countryOfOrigin": "United Kingdom",
                 "genomeSize": "14",
                 "accessionNumber": "EE3383",
-                "sampleCollectionDateUtc": datetime.now(),
+                "sampleCollectionDateUtc": datetime.utcnow(),
                 "costCode": "S1234",
                 "shearedFemtoFragmentSize": "5",
                 "postSPRIConcentration": "10",
@@ -33,7 +33,7 @@
                 "finalNanoDrop230": "6",
                 "finalNanoDrop": "7",
                 "shearingAndQCComments": "Comments",
-                "dateSubmittedUTC": datetime.now(),
+                "dateSubmittedUTC": datetime.utcnow(),
                 "priorityLevel": "Medium",
             },
             {
@@ -52,7 +52,7 @@
                 "countryOfOrigin": "United Kingdom",
                 "genomeSize": "14",
                 "accessionNumber": "EE3383",
-                "sampleCollectionDateUtc": datetime.now(),
+                "sampleCollectionDateUtc": datetime.utcnow(),
                 "costCode": "S1234",
                 "shearedFemtoFragmentSize": "5",
                 "postSPRIConcentration": "10",
@@ -61,17 +61,17 @@
                 "finalNanoDrop230": "6",
                 "finalNanoDrop": "7",
                 "shearingAndQCComments": "Comments",
-                "dateSubmittedUTC": datetime.now(),
+                "dateSubmittedUTC": datetime.utcnow(),
                 "priorityLevel": "Medium",
             },
         ],
     },
 }
 
 
-TEST_INVALID_CREATE_LABWARE_MSG_OBJECT: Dict[str, Any] = {
+TEST_INVALID_CREATE_LABWARE_MSG_OBJECT: dict[str, Any] = {
     "messageUuid": "b01aa0ad7b19-4f94-87e9-70d74fb8783c".encode(),
-    "messageCreateDateUtc": datetime.now(),
+    "messageCreateDateUtc": datetime.utcnow(),
     "labware": {
         "labwareType": "Plate12x8",
         "barcode": "BARCODE001",
@@ -92,7 +92,7 @@
                 "countryOfOrigin": "United Kingdom",
                 "genomeSize": "14",
                 "accessionNumber": "EE3383",
-                "sampleCollectionDateUtc": datetime.now(),
+                "sampleCollectionDateUtc": datetime.utcnow(),
                 "costCode": "S1234",
                 "shearedFemtoFragmentSize": "5",
                 "postSPRIConcentration": "10",
@@ -101,7 +101,7 @@
                 "finalNanoDrop230": "6",
                 "finalNanoDrop": 7,
                 "shearingAndQCComments": "Comments",
-                "dateSubmittedUTC": datetime.now(),
+                "dateSubmittedUTC": datetime.utcnow(),
                 "priorityLevel": "Medium",
             },
             {
@@ -120,7 +120,7 @@
                 "countryOfOrigin": "United Kingdom",
                 "genomeSize": "14",
                 "accessionNumber": "EE3383",
-                "sampleCollectionDateUtc": datetime.now(),
+                "sampleCollectionDateUtc": datetime.utcnow(),
                 "costCode": "S1234",
                 "shearedFemtoFragmentSize": "5",
                 "postSPRIConcentration": "10",
@@ -129,7 +129,7 @@
                 "finalNanoDrop230": "6",
                 "finalNanoDrop": "7",
                 "shearingAndQCComments": "Comments",
-                "dateSubmittedUTC": datetime.now(),
+                "dateSubmittedUTC": datetime.utcnow(),
                 "priorityLevel": "Medium",
             },
         ],

tests/message_properties/definitions/test_dict_input.py

@@ -23,12 +23,6 @@ def test_dict_input_can_validate_when_invalid():
     input = DictInput({}, "name")
     assert not input.validate()
 
-    input = DictInput(None, "name")  # type: ignore
-    assert not input.validate()
-
-    input = DictInput([], "name")  # type: ignore
-    assert not input.validate()
-
     input = DictInput({"other": 1}, "name")
     assert not input.validate()