Merge pull request #27 from bag-cnag/dev

0.7.5

.gitignore

@@ -4,7 +4,6 @@
 .coverage
 
 # Envs
-.env.cnag
 venv*/
 
 # Caches
@@ -19,3 +18,7 @@ docs/biodm/
 
 # Minikube
 cxg_on_k8/
+
+# Internal
+*.cnag*
+omicsdm_server_v2/

README.md

@@ -23,13 +23,16 @@ BioDM is a fast, modular, stateless and asynchronous REST API framework with the
   - Storage leveraging _S3_ protocol
   - Jobs & Visualization leveraging _Kubernetes_ cluster
 
--> Act as an API gateway and log relevant data.
-
 - Also sets up essentials:
   - Liveness endpoint
   - Login and token retrieval system
   - OpenAPI schema generation through [apispec](https://github.com/marshmallow-code/apispec)
 
+
+It sits on the **F**indability and **A**ccessibility part of the **F.A.I.R** principles,
+while remaining flexible for the remainder to be implemented.
+
+
 ## Quickstart
 ### Install
 ```bash

compose.example.yml

@@ -0,0 +1,45 @@
+include:
+  - ./compose.yml
+
+services:
+  example:
+    build:
+      context: ./
+      dockerfile: docker/Dockerfile.biodm-test-api
+      args:
+        - PYTHON__V=3.11
+        - KEEPENV=1
+    extra_hosts:
+      - "host.minikube.internal:10.10.0.3"
+    healthcheck:
+      test: python3 -c "import requests; exit(requests.get('http://0.0.0.0:8000/live').text != 'live\n');"
+      interval: 5s
+      timeout: 5s
+      retries: 10
+    depends_on:
+      api-db:
+        condition: service_healthy
+      keycloak:
+        condition: service_healthy
+      s3bucket:
+        condition: service_healthy
+    stdin_open: true
+    ports:
+    - 8000:8000
+    links:
+    - api-db
+    - keycloak
+    - s3bucket
+
+  example-swagger-ui:
+    image: swaggerapi/swagger-ui:v5.17.14
+    depends_on:
+      example:
+        condition: service_healthy
+    environment:
+      PORT: 9080
+      API_URL: http://localhost:8000/schema
+    ports:
+    - 9080:9080
+    links:
+    - example

compose.yml

@@ -100,14 +100,15 @@ services:
   swagger-ui:
     image: swaggerapi/swagger-ui:v5.17.14
     ports:
-      - "9080:8080"
+      - 9080:8080
     environment:
       API_URL: http://localhost:8000/schema
 
 networks:
   biodm-dev:
     driver: bridge
     ipam:
+      driver: default
       config:
         - subnet: 10.10.0.0/16
           gateway: 10.10.0.1

docker/Dockerfile.biodm-test-api

@@ -1,9 +1,11 @@
 ARG PYTHON__V=3.11
 ARG PIP_CACHE_DIR=/root/.cache/pip
+ARG KEEPENV
 
 FROM python:${PYTHON__V}-slim-bookworm
 
 ARG PIP_CACHE_DIR
+ARG KEEPENV
 
 COPY ./src/requirements /biodm/src/requirements
 WORKDIR /biodm
@@ -16,9 +18,9 @@ RUN --mount=type=cache,target=${PIP_CACHE_DIR} \
 COPY ./pyproject.toml /biodm/pyproject.toml
 COPY ./src/biodm /biodm/src/biodm
 COPY ./src/example /biodm/src/example
-# Remove .env to replace it with environment variables in compose file.
-RUN find /biodm/src/example -name '.env' | xargs rm -rf
 
+# conditionally remove .env to replace it with environment variables in compose file.
+RUN if [ -z "$KEEPENV" ] ; then find /biodm/src/example -name '.env' | xargs rm -rf ; else : ; fi
 
 RUN pip3 install .
 

docs/developer_manual/advanced_use.rst

@@ -122,7 +122,7 @@ insert/update statement building and execution.
             if not path.parent.parts == ('/',):
                 parent = await self.kc.get_group_by_path(str(path.parent))
                 if not parent:
-                    raise ValueError("Input path does not match any parent group.")
+                    raise DataError("Input path does not match any parent group.")
                 parent_id = parent['id']
 
             data['id'] = await self.kc.create_group(path.name, parent_id)
@@ -147,6 +147,7 @@ insert/update statement building and execution.
                     await User.svc.read_or_create(user, [group["path"]], [group["id"]],)
 
             # Send to DB
+            # Not passing user_info, which gives unrestricted permissions as check happens above.
             return await super().write(data, stmt_only=stmt_only, **kwargs)
 
 
@@ -316,7 +317,7 @@ A lot of that code has to do with retrieving async SQLAlchemy objects attributes
             key = await self.gen_key(file, session=session)
             parts.append(
                 UploadPart(
-                    id_upload=file.upload.id,
+                    upload_id=file.upload.id,
                     form=str(
                         self.s3.create_presigned_post(
                             object_name=key,
@@ -369,18 +370,26 @@ A lot of that code has to do with retrieving async SQLAlchemy objects attributes
             return url
 
         @DatabaseManager.in_session
-        async def _insert(self, stmt: Insert, session: AsyncSession) -> Base:
+        async def _insert(
+            self,
+            stmt: Insert,
+            user_info: UserInfo | None,
+            session: AsyncSession
+        ) -> (Any | None):
             """INSERT special case for file: populate url after getting entity id."""
-            file = await super()._insert(stmt, session=session)
+            file = await super()._insert(stmt, user_info=user_info, session=session)
             await self.gen_upload_form(file, session=session)
             return file
 
         @DatabaseManager.in_session
         async def _insert_list(
-            self, stmts: Sequence[Insert], session: AsyncSession
+            self,
+            stmts: Sequence[Insert],
+            user_info: UserInfo | None,
+            session: AsyncSession
         ) -> Sequence[Base]:
-            """INSERT many objects into the DB database."""
-            files = await super()._insert_list(stmts, session=session)
+            """INSERT many objects into the DB database, check token write permission before commit."""
+            files = await super()._insert_list(stmts, user_info=user_info, session=session)
             for file in files:
                 await self.gen_upload_form(file, session=session)
             return files

docs/developer_manual/demo.rst

@@ -30,8 +30,8 @@ we will go over the following minimal example.
 
     class File(bd.components.S3File, bd.components.Base):
         id                         = sa.Column(sa.Integer,                    primary_key=True)
-        id_dataset                 = sa.Column(sa.ForeignKey("DATASET.id"),   nullable=False)
-        dataset: sao.Mapped["Dataset"] = sao.relationship(back_populates="files", single_parent=True, foreign_keys=[id_dataset])
+        dataset_id                 = sa.Column(sa.ForeignKey("DATASET.id"),   nullable=False)
+        dataset: sao.Mapped["Dataset"] = sao.relationship(back_populates="files", single_parent=True, foreign_keys=[dataset_id])
 
     # Schemas
     class DatasetSchema(ma.Schema):
@@ -50,7 +50,7 @@ we will go over the following minimal example.
         size           = mf.Integer(required=True)
         url            = mf.String(                 dump_only=True)
         ready          = mf.Bool(                   dump_only=True)
-        id_dataset     = mf.Integer(required=True,  load_only=True)
+        dataset_id     = mf.Integer(required=True,  load_only=True)
         dataset        = mf.Nested("DatasetSchema")
 
     # Controllers

docs/developer_manual/index.rst

@@ -22,7 +22,7 @@ metadata and relationships, to setup standard RESTful endpoints.
 
 Furthermore, it is providing a structure and toolkit in order to manage common Data Management problems
 such as: s3 protocol remote file storage, group based permissions access (on both resources and
-endpoints), resource versioning (coming up), cluster jobs and so on.
+endpoints), resource versioning, cluster jobs and so on.
 
 Moreover, the modular and flexible architecture allows you to easily extend base features for
 instance specific use cases.

docs/developer_manual/permissions.rst

@@ -99,7 +99,7 @@ In our example:
         files         : sao.Mapped[List["File"]] = sao.relationship(back_populates="dataset")
 
         __permissions__ = (
-            Permission(files, write=True, read=False, download=True),
+          Permission(files, write=True, read=False, download=True),
         )
 
 The latter enables ``File`` permissions at the ``Dataset`` level.
@@ -111,6 +111,52 @@ with a nested collection and its elements.
 
     Those permissions will be taken into account when directly accessing ``/files`` API routes. 
 
+.. note::
+
+    You always need a top level resource. This system is thought to be combined with decorator
+    based permission for such resources.
+
+
+Nesting and propagation
+~~~~~~~~~~~~~~~~~~~~~~~
+
+This tool offers flexible options. Imagine a case with one more level of collections with a
+``Project`` table, containing a collection of ``Dataset`` such as showcased in ``example``.
+
+.. code-block:: python
+
+  class Project(Base):
+      id = Column(Integer, nullable=False, primary_key=True)
+      ...
+
+      datasets: Mapped[List["Dataset"]] = relationship(back_populates="project")
+
+
+Then you may use a string selector to apply that top level permission directly on a lower level
+resource, skipping the mid level.
+
+
+.. code-block:: python
+
+  class Project(Base):
+      ...
+    __permissions__ = (
+      Permission("datasets.files", download=True),
+    )
+
+
+Moreover, you have the option of propagating that top level permission to the lower nested
+collections. Sharing those permissions between intermediate level and lower level.
+
+
+.. code-block:: python
+
+  class Project(Base):
+      ...
+    __permissions__ = (
+      Permission(datasets, read=True, write=True, download=True, propagates_to=["files"]),
+    )
+
 
 Strict composition
 ~~~~~~~~~~~~~~~~~~