Public API files resources

[bisgaard-itis]

@bisgaard-itis

Baklava issues

Beta Give feedback

1. Complete delete file entrypoint
2. Make sure query params are correctly exposed in /v0/files/{file_id}:complete and /v0/files/{file_id}:abort
3. When uploading to system - check if resource already exists
4. Ensure only relative links are passed back from get_upload_links
5. Change name of ClientFileUploadSchema so it doesn't have "Schema" in it

Tasks

Beta Give feedback

1. Hash/FileId which can be computed in client (nice to have)

[pcrespov]

files

• release 0.6.0 we will include files resources
• user story on files resource
• how to:
  • auto-delete files
  • limit maximum upload (user-based?)
  • collection : pagination & filtering & search
    • search by checksum
    • search by file-name
  • hash?
  • upload w/o intermediate storing in api-server
  • metadata

list_files

sequenceDiagram
    participant API
    participant AS as api-server
    participant SS as storage
    link API: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/api-server/openapi.json
    link SS: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/storage/src/simcore_service_storage/api/v0/openapi.yaml

    API->>+AS: list_files
    AS->>+SS: search_files_starting_with("api/", user_id)
    SS-->>-AS: list[StorageFileMetadata] 
    AS-->>-API: list[File]

Loading

• uses models_library/api_schemas_storage

get_files_page NOT IMPLEMENTED

sequenceDiagram
    participant API
    participant AS as api-server
    participant SS as storage
    link API: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/api-server/openapi.json
    link SS: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/storage/src/simcore_service_storage/api/v0/openapi.yaml

    API->>+AS: get_files_page
    AS->>+SS: ?
    SS-->>-AS: ?
    AS-->>-API: Page[File]

Loading

upload_file

sequenceDiagram
    participant API
    participant AS as api-server
    participant SS as storage
    participant S3 as s3
    link API: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/api-server/openapi.json
    link SS: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/storage/src/simcore_service_storage/api/v0/openapi.yaml

    API->>+AS: upload_file
    AS->>+SS: ?
    AS->>+S3: ?
    S3-->>-AS: ?
    SS-->>-AS: ?    
    AS-->>-API: Page[File]

Loading

• uses simcore-sdk.upload_path
• Should upload directly to s3 w/o api-server: uploaded files are temporarily located in the server #3223

get_upload_links

sequenceDiagram
    participant API
    participant AS as api-server
    participant SS as storage
    participant S3 as s3
    link API: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/api-server/openapi.json
    link SS: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/storage/src/simcore_service_storage/api/v0/openapi.yaml

    API->>+AS: get_upload_links
    AS->>+SS: get_upload_file_links
    SS->>+S3: ?
    S3-->>-SS: file_links
    SS-->>-AS: (store_id, file_links)    
    AS-->>-API: ClientFileUploadSchema

Loading

• uses simcore_sdk.filemanager.get_upload_links_from_s3
• rename ClientFileUploadSchema to avoid word Schema

get_file

sequenceDiagram
    participant API
    participant AS as api-server
    participant SS as storage
    participant S3 as s3
    link API: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/api-server/openapi.json
    link SS: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/storage/src/simcore_service_storage/api/v0/openapi.yaml

    API->>+AS: storage.search_files_file
    AS->>+SS: search_files_starting_with(f"api/{file_id}", user_id)
    SS-->>-AS: list[StorageFileMetadata][0]
    AS-->>-API: list[File]

Loading

• implement single file get in storage

delete_file NOT IMPLEMENTED

sequenceDiagram
    participant API
    participant AS as api-server
    participant SS as storage
    participant S3 as s3
    link API: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/api-server/openapi.json
    link SS: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/storage/src/simcore_service_storage/api/v0/openapi.yaml

    API->>+AS: storage.delete_file
    AS->>+SS: delete_file(location_id, file_id, user_id)
    SS->>+S3: delete
    S3-->>-SS: ?
    SS-->>-AS: 204
    AS-->>-API: 204

Loading

• sim4life.io - WP6: API osparc-issues#952

complete_multipart_upload, abort_multipart_upload

sequenceDiagram
    participant API
    participant AS as api-server
    participant SS as storage
    participant S3 as s3
    link API: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/api-server/openapi.json
    link SS: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/storage/src/simcore_service_storage/api/v0/openapi.yaml

    API->>+AS: create_complete_upload_link
    AS->>+SS: complete_upload_file(f"api/{file_id}/file_name", user_id)
    SS-->>-AS: Envelope[FileUploadSchema]
    AS-->>-API: RedirectResponse to AnyUrl

Loading

• uses simcore_sdk

download_file

sequenceDiagram
    participant API
    participant AS as api-server
    participant SS as storage
    participant S3 as s3
    link API: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/api-server/openapi.json
    link SS: OAS @ https://editor.swagger.io/?url=https://raw.githubusercontent.com/ITISFoundation/osparc-simcore/master/services/storage/src/simcore_service_storage/api/v0/openapi.yaml

    API->>+AS: file_file
    AS->>+SS: search_files_starting_with(f"api/{file_id}", user_id)
    SS-->>-AS: list[StorageFileMetadata][0]
    AS->>+SS: download_file("api/{file_id}/{file_name}", user_id)
    SS->>+S3: ?
    S3-->>-SS: presigned_download_link: ? AnyUrl 
    SS-->>-AS: Enveloped[LinkType]
    AS-->>-API: RedirectResponse to AnyUrl

Loading