BREAKING CHANGE : Auxiliary Files stage out

[wphyojpl]

Closes #375
Closes #405

[wphyojpl]

• Files with same names in different directory, they are overwritten as uploaded location is s3://bucket/collection/granule-id/filename. Should they retain their original folder structure?
• Still need an output directory to write successful and failed json files
• if OUTPUT_FILE is provided, a catalog file is also written there.

[ngachung]

Since we use the filename as the granule ID, I don't think we can avoid overwriting.

[ngachung]

@wphyojpl As I was working on the ASIPS data ingest, I was thinking we should look at unitypy's support for creating stac (it uses pystac behind the scenes) so that we reuse our own libraries. UDS will need to maintain all of unitypy that interacts with data (DAPA, STAC).

[wphyojpl]

2024-08-26T13:23:50.335Z 003d5b13-2be5-4f56-8caf-c9a9774a7693 ERROR {"level":"error","message":"Attempt 1 failed. API invoke error: /providers/unity returned 400: {\"statusCode\":400,\"error\":\"Bad Request\",\"message\":\"Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?\",\"name\":\"KnexTimeoutError\"}.","sender":"@api-client/cumulusApiClient","timestamp":"2024-08-26T13:23:50.335Z"}
--
  | 2024-08-26T13:23:50.753Z | 2024-08-26T13:23:50.753Z 003d5b13-2be5-4f56-8caf-c9a9774a7693 ERROR {"level":"error","message":"Attempt 1 failed. API invoke error: /collections/URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION/2408071015 returned 404: {\"statusCode\":404,\"error\":\"Not Found\",\"message\":\"Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?\"}.","sender":"@api-client/cumulusApiClient","timestamp":"2024-08-26T13:23:50.753Z"}

• https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fuds-sbx-cumulus-sqsMessageConsumer/log-events/2024$252F08$252F26$252F$255B$2524LATEST$255D22aa072d1d814635a547741d4ee118f0
• Failing to start step function at this moment.. probably due to 18.3.3 update

[ngachung]

Yes, will need to restore RDS first.

[wphyojpl]

2024-08-26T23:56:47.975Z | 2024-08-26T23:56:47.975Z e873fa96-2434-4b41-946f-d802fb02aba1 ERROR {"level":"error","message":"Attempt 1 failed. API invoke error: /providers/unity returned 400: {\"statusCode\":400,\"error\":\"Bad Request\",\"message\":\"Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?\",\"name\":\"KnexTimeoutError\"}.","sender":"@api-client/cumulusApiClient","timestamp":"2024-08-26T23:56:47.975Z"}
-- | --
  | 2024-08-26T23:56:48.473Z | 2024-08-26T23:56:48.473Z e873fa96-2434-4b41-946f-d802fb02aba1 ERROR {"level":"error","message":"Attempt 1 failed. API invoke error: /collections/URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION/2408071015 returned 404: {\"statusCode\":404,\"error\":\"Not Found\",\"message\":\"Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?\"}.","sender":"@api-client/cumulusApiClient","timestamp":"2024-08-26T23:56:48.473Z"}

Still the same error.

[wphyojpl]

Hi @ngachung

I think this is ready. I managed to upload arbitrary files, and get the results with file size and checksum

{
    "numberMatched": {
        "total_size": 6
    },
    "numberReturned": 6,
    "stac_version": "1.0.0",
    "type": "FeatureCollection",
    "links": [
        {
            "rel": "self",
            "href": "https://d3vc8w9zcq658.cloudfront.net/data-sbx/collections/URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900/items?limit=20"
        },
        {
            "rel": "root",
            "href": "https://d3vc8w9zcq658.cloudfront.net"
        }
    ],
    "features": [
        {
            "type": "Feature",
            "stac_version": "1.0.0",
            "id": "URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900:test_file_0",
            "properties": {
                "datetime": "2024-08-29T16:08:09.777000Z",
                "start_datetime": "2024-08-29T16:06:53.214000Z",
                "end_datetime": "2024-08-29T16:06:53.214000Z",
                "created": "1970-01-01T00:00:00Z",
                "updated": "2024-08-29T16:08:49.557000Z",
                "status": "completed",
                "provider": "unity"
            },
            "geometry": {
                "type": "Point",
                "coordinates": [
                    0.0,
                    0.0
                ]
            },
            "links": [
                {
                    "rel": "collection",
                    "href": "."
                },
                {
                    "rel": "self",
                    "href": "https://d3vc8w9zcq658.cloudfront.net/data-sbx/collections/URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900/items/URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900:test_file_0",
                    "type": "application/json",
                    "title": "URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900:test_file_0"
                }
            ],
            "assets": {
                "URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900:test_file_0.cmr.xml": {
                    "href": "s3://uds-sbx-cumulus-staging/URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900/URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900:URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900:test_file_0/URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900:test_file_0.cmr.xml",
                    "title": "URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900:test_file_0.cmr.xml",
                    "description": "size=1860;checksumType=md5;checksum=823d3f29bbaf4e954ac2050dce005e79;",
                    "file:size": 1860,
                    "file:checksum": "823d3f29bbaf4e954ac2050dce005e79",
                    "roles": [
                        "metadata"
                    ]
                },
                "test_file_0.json.stac.json": {
                    "href": "s3://uds-sbx-cumulus-staging/URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900/URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900:URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900:test_file_0/test_file_0.json.stac.json",
                    "title": "test_file_0.json.stac.json",
                    "description": "size=-1;checksumType=md5;checksum=unknown;",
                    "file:size": -1,
                    "file:checksum": "unknown",
                    "roles": [
                        "metadata"
                    ]
                },
                "test_file_0.json": {
                    "href": "s3://uds-sbx-cumulus-staging/URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900/URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900:URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900:test_file_0/test_file_0.json",
                    "title": "test_file_0.json",
                    "description": "size=107;checksumType=md5;checksum=86165135d27f4c96dc3ddea2344e84a8;",
                    "file:size": 107,
                    "file:checksum": "86165135d27f4c96dc3ddea2344e84a8",
                    "roles": [
                        "data"
                    ]
                }
            },
            "bbox": [
                -180.0,
                -90.0,
                180.0,
                90.0
            ],
            "stac_extensions": [
                "https://stac-extensions.github.io/file/v2.1.0/schema.json"
            ],
            "collection": "URN:NASA:UNITY:UDS_MY_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2408290900"
        },

It does need a new index because previous stac_extensions definition in elastic search index is invalid.