v11.0.0

Release v11.0.0

Please note - this release has been superseded by release v11.1.0.

Users upgrading to V11 should upgrade instead to at least v11.1.0, but should follow the migration instructions noted here.

v9.9->v11.0 MIGRATION NOTES

Release v11.0 is a maintenance release series, replacing v9.9. If you are
upgrading to or past v11 from v9.9.x to this release, please pay attention to the following
migration notes from prior releases:

Migration steps

After deploying the data-persistence module, but before deploying the main cumulus module

• Due to a bug in the PUT /rules/<name> endpoint, the rule records in PostgreSQL may be
  out of sync with records in DynamoDB. In order to bring the records into sync, re-deploy and re-run the
  data-migration1 Lambda with a payload of
  {"forceRulesMigration": true}:

aws lambda invoke --function-name $PREFIX-data-migration1 \
  --payload $(echo '{"forceRulesMigration": true}' | base64) $OUTFILE

As part of the cumulus deployment

• Please read the documentation on the updates to the granule files schema for our Cumulus workflow tasks and how to upgrade your deployment for compatibility.
• (Optional) Update the task-config for all workflows that use the sync-granule task to include workflowStartTime set to
  {$.cumulus_meta.workflow_start_time}. See here for an example.

After the cumulus deployment

As part of the work on the RDS Phase 2 feature, it was decided to re-add the
granule file type property on the file table (detailed reasoning
https://wiki.earthdata.nasa.gov/pages/viewpage.action?pageId=219186829). This
change was implemented as part of CUMULUS-2672/CUMULUS-2673, however granule
records ingested prior to v11 will not have the file.type property stored in the
PostGreSQL database, and on installation of v11 API calls to get granule.files
will not return this value. We anticipate most users are impacted by this issue.

Users that are impacted by these changes should re-run the granule migration
lambda to only migrate granule file records:

PAYLOAD=$(echo '{"migrationsList": ["granules"], "granuleMigrationParams": {"migrateOnlyFiles": "true"}}' | base64)
aws lambda invoke --function-name $PREFIX-postgres-migration-async-operation \
--payload $PAYLOAD $OUTFILE

You should note that this will only move files for granule records in
PostgreSQL. If you have not completed the phase 1 data migration or
have granule records in dynamo that are not in PostgreSQL, the migration will
report failure for both the DynamoDB granule and all the associated files and the file
records will not be updated.

If you prefer to do a full granule and file migration, you may instead
opt to run the migration with the migrateAndOverwrite option instead, this will re-run a
full granule/files migration and overwrite all values in the PostgreSQL database from
what is in DynamoDB for both granules and associated files:

PAYLOAD=$(echo '{"migrationsList": ["granules"], "granuleMigrationParams": {"migrateAndOverwrite": "true"}}' | base64)
aws lambda invoke --function-name $PREFIX-postgres-migration-async-operation \
--payload $PAYLOAD $OUTFILE

Please note: Since this data migration is copying all of your granule data
from DynamoDB to PostgreSQL, it can take multiple hours (or even days) to run,
depending on how much data you have and how much parallelism you configure the
migration to use. In general, the more parallelism you configure the migration
to use, the faster it will go, but the higher load it will put on your
PostgreSQL database. Excessive database load can cause database outages and
result in data loss/recovery scenarios. Thus, the parallelism settings for the
migration are intentionally set by default to conservative values but are
configurable. If this impacts only some of your data products you may want
to consider using other granuleMigrationParams.

Please see the second data migration
docs
for more on this tool if you are unfamiliar with the various options.

Notable changes

• CUMULUS-2703
  • ORCA Backup is now a supported reportType for the POST /reconciliationReports endpoint

Added

• CUMULUS-2311 - RDS Migration Epic Phase 2
  • CUMULUS-2208
    • Added @cumulus/message/utils.parseException to parse exception objects
    • Added helpers to @cumulus/message/Granules:
      • getGranuleProductVolume
      • getGranuleTimeToPreprocess
      • getGranuleTimeToArchive
      • generateGranuleApiRecord
    • Added @cumulus/message/PDRs/generatePdrApiRecordFromMessage to generate PDR from Cumulus workflow message
    • Added helpers to @cumulus/es-client/indexer:
      • deleteAsyncOperation to delete async operation records from Elasticsearch
      • updateAsyncOperation to update an async operation record in Elasticsearch
    • Added granules PUT endpoint to Cumulus API for updating a granule.
      Requests to this endpoint should be submitted without an action
      attribute in the request body.
    • Added @cumulus/api-client/granules.updateGranule to update granule via the API
  • CUMULUS-2303
    • Add translatePostgresProviderToApiProvider method to @cumulus/db/translate/providers
  • CUMULUS-2306
    • Updated API execution GET endpoint to read individual execution records
      from PostgreSQL database instead of DynamoDB
    • Updated API execution-status endpoint to read execution records from
      PostgreSQL database instead of DynamoDB
  • CUMULUS-2302
    • Added translatePostgresCollectionToApiCollection method to
      @cumulus/db/translate/collections
    • Added searchWithUpdatedAtRange method to
      @cumulus/db/models/collections
  • CUMULUS-2301
    • Created API asyncOperations POST endpoint to create async operations.
  • CUMULUS-2307
    • Updated API PDR GET endpoint to read individual PDR records from
      PostgreSQL database instead of DynamoDB
    • Added deletePdr to @cumulus/api-client/pdrs
  • CUMULUS-2782
    • Update API granules endpoint move action to update granules in the index
      and utilize postgres as the authoritative datastore
  • CUMULUS-2769
    • Update collection PUT endpoint to require existance of postgresql record
      and to ignore lack of dynamoDbRecord on update
  • CUMULUS-2767
    • Update provider PUT endpoint to require existence of PostgreSQL record
      and to ignore lack of DynamoDB record on update
  • CUMULUS-2759
    • Updates collection/provider/rules/granules creation (post) endpoints to
      primarily check for existence/collision in PostgreSQL database instead of DynamoDB
  • CUMULUS-2714
    • Added @cumulus/db/base.deleteExcluding method to allow for deletion of a
      record set with an exclusion list of cumulus_ids
  • CUMULUS-2317
    • Added @cumulus/db/getFilesAndGranuleInfoQuery() to build a query for searching file
      records in PostgreSQL and return specified granule information for each file
    • Added @cumulus/db/QuerySearchClient library to handle sequentially fetching and paging
      through results for an arbitrary PostgreSQL query
    • Added insert method to all @cumulus/db models to handle inserting multiple records into
      the database at once
    • Added @cumulus/db/translatePostgresGranuleResultToApiGranule helper to
      translate custom PostgreSQL granule result to API granule
  • CUMULUS-2672
    • Added migration to add type text column to Postgres database files table
  • CUMULUS-2634
    • Added new functions for upserting data to Elasticsearch:
      • @cumulus/es-client/indexer.upsertExecution to upsert an execution
      • @cumulus/es-client/indexer.upsertPdr to upsert a PDR
      • @cumulus/es-client/indexer.upsertGranule to upsert a granule
  • CUMULUS-2510
    • Added execution_sns_topic_arn environment variable to
      sf_event_sqs_to_db_records lambda TF definition.
    • Added to sf_event_sqs_to_db_records_lambda IAM policy to include
      permissions for SNS publish for report_executions_topic
    • Added collection_sns_topic_arn environment variable to
      PrivateApiLambda and ApiEndpoints lambdas.
    • Added updateCollection to @cumulus/api-client.
    • Added to ecs_cluster IAM policy to include permissions for SNS publish
      for report_executions_sns_topic_arn, report_pdrs_sns_topic_arn,
      report_granules_sns_topic_arn
    • Added variables for report topic ARNs to process_dead_letter_archive.tf
    • Added variable for granule report topic ARN to bulk_operation.tf
    • Added pdr_sns_topic_arn environment variable to
      sf_event_sqs_to_db_records lambda TF definition.
    • Added the new function publishSnsMessageByDataType in @cumulus/api to
      publish SNS messages to the report topics to PDRs, Collections, and
      Executions.
    • Added the following functions in publishSnsMessageUtils to handle
      publishing SNS messages for specific data and event types:
      • publishCollectionUpdateSnsMessage
      • publishCollectionCreateSnsMessage
      • publishCollectionDeleteSnsMessage
      • publishGranuleUpdateSnsMessage
      • publishGranuleDeleteSnsMessage
      • publishGranuleCreateSnsMessage
      • publishExecutionSnsMessage
      • publishPdrSnsMessage
      • `pub...

v9.9.4

Release v9.9.4

Changed

• CVE-2022-2477
  • Update node-forge to 1.3.0 in @cumulus/common to address CVE-2022-2477
• CUMULUS-2868
  • Added iam:PassRole permission to step_policy in tf-modules/ingest/iam.tf

v10.1.2

Release 10.1.2

Added

• CUMULUS-2859
  • Update postgres-db-migration lambda timeout to default 900 seconds
  • Add db_migration_lambda_timeout variable to data-persistence module to
    allow this timeout to be user configurable
• CUMULUS-2868
  • Added iam:PassRole permission to step_policy in tf-modules/ingest/iam.tf

v10.1.1

Release v10.1.1

Migration steps

Due to a bug in the PUT /rules/ endpoint, the rule records in PostgreSQL may be out of sync with records in DynamoDB. In order to bring the records into sync, re-run the previously deployed data-migration1 Lambda with a payload of {"forceRulesMigration": true}:

aws lambda invoke --function-name $PREFIX-data-migration1 \
  --payload $(echo '{"forceRulesMigration": true}' | base64) $OUTFILE

Added

• CUMULUS-2846
  • Added @cumulus/db/translate/rule.translateApiRuleToPostgresRuleRaw to translate API rule to PostgreSQL rules and keep undefined fields

Changed

• CUMULUS-NONE
  • Adds logging to ecs/async-operation Docker conatiner that launches async tasks on ECS. Sets default async_operation_image_version to 39.
• CUMULUS-2845
  • Updated rules model to decouple createRuleTrigger from create.
  • Updated rules POST endpoint to call rulesModel.createRuleTrigger directly to create rule trigger.
  • Updated rules PUT endpoints to call rulesModel.createRuleTrigger if update fails and reversion needs to occur.
• CUMULUS-2846
  • Updated version of localstack/localstack used in local unit testing to 0.11.5

Fixed

• Upgraded lodash to version 4.17.21 to fix vulnerability
• CUMULUS-2845
  • Fixed bug in POST /rules endpoint causing rule records to be created inconsistently in DynamoDB and PostgreSQL
• CUMULUS-2846
  • Fixed logic for PUT /rules/ endpoint causing rules to be saved inconsistently between DynamoDB and PostgreSQL
• CUMULUS-2854
  • Fixed queue granules behavior where the task was not accounting for granules that already had createdAt set. Workflows downstream in this scenario should no longer fail to write their granules due to order-of-db-writes constraints in the database update logic.

v9.4.2

Release v9.4.2

CUMULUS-2725

• Updated providers endpoint to return encrypted password
• Updated providers model to try decrypting credentials before encryption to
  allow for better handling of updating providers

CUMULUS-2841

• Add integration test to validate PDR node provider that requires password
  credentials succeeds on ingest

v10.1.0

Release v10.1.0

Added

• CUMULUS-2775
  • Added a configurable parameter group for the RDS serverless database cluster deployed by tf-modules/rds-cluster-tf. The allowed parameters for the parameter group can be found in the AWS documentation of allowed parameters for an Aurora PostgreSQL cluster. By default, the following parameters are specified:
    • shared_preload_libraries: pg_stat_statements,auto_explain
    • log_min_duration_statement: 250
    • auto_explain.log_min_duration: 250
• CUMULUS-2781
  • Add api_config secret to hold API/Private API lambda configuration values
• CUMULUS-2840
  • Added an index on granule_cumulus_id to the RDS files table.

Changed

• CUMULUS-2492
  • Modify collectionId logic to accomodate trailing underscores in collection short names. e.g. shortName____
• CUMULUS-2847
  • Move DyanmoDb table name into API keystore and initialize only on lambda cold start
• CUMULUS-2833
  • Updates provider model schema titles to display on the dashboard.
• CUMULUS-2837
  • Update process-s3-dead-letter-archive to unpack SQS events in addition to
    Cumulus Messages
  • Update process-s3-dead-letter-archive to look up execution status using
    getCumulusMessageFromExecutionEvent (common method with sfEventSqsToDbRecords)
  • Move methods in api/lib/cwSfExecutionEventUtils to
    @cumulus/message/StepFunctions
• CUMULUS-2775
  • Changed the timeout_action to ForceApplyCapacityChange by default for the RDS serverless database cluster tf-modules/rds-cluster-tf
• CUMULUS-2781
  • Update API lambda to utilize api_config secret for initial environment variables

Fixed

• Fixed IAM permissions issue with <prefix>-postgres-migration-async-operation Lambda
  which prevented it from running a Fargate task for data migration.
• CUMULUS-2853
  • Move OAUTH_PROVIDER to lambda env variables to address regression in CUMULUS-2781
  • Add logging output to api app router
• Added Cloudwatch permissions to <prefix>-steprole in tf-modules/ingest/iam.tf to address the
  Error: error creating Step Function State Machine (xxx): AccessDeniedException: 'arn:aws:iam::XXX:role/xxx-steprole' is not authorized to create managed-rule
  error in non-NGAP accounts:
  • events:PutTargets
  • events:PutRule
  • events:DescribeRule

v9.9.3

Release v9.9.3

• CUMULUS-2853
  • Move OAUTH_PROVIDER to lambda env variables to address regression in 9.9.2/CUMULUS-2275
  • Add logging output to api app router

v9.9.2

Release v9.9.2

Added

• CUMULUS-2775
  • Added a configurable parameter group variable, db_parameters, for the RDS serverless database cluster deployed by tf-modules/rds-cluster-tf. The allowed parameters for the parameter group can be found in the AWS documentation of allowed parameters for an Aurora PostgreSQL cluster. By default, the following parameters are specified:
    • shared_preload_libraries: pg_stat_statements,auto_explain
• CUMULUS-2840
  • Added an index on granule_cumulus_id to the RDS files table.

Changed

• CUMULUS-2847
  • Move DyanmoDb table name into API keystore and initialize only on lambda cold start
• CUMULUS-2781
  • Add api_config secret to hold API/Private API lambda configuration values
• CUMULUS-2775
  • Changed the timeout_action to ForceApplyCapacityChange by default for the RDS serverless database cluster tf-modules/rds-cluster-tf

v9.4.1

Release v9.4.1

CUMULUS-2847

• Update dynamo configuration to read from S3 instead of System Manager
  Parameter Store
• Move api configuration initialization outside the lambda handler to
  eliminate unneded S3 calls/require config on cold-start only
• Moved ssh2 package from @cumulus/common to @cumulus/sftp-client and
  upgraded package from ^0.8.7 to ^1.0.0 to address security vulnerability
  issue in previous version.
• Fixed hyrax task package.json dev dependency
• Update CNM lambda dependencies for Core tasks
  • cumulus-cnm-response-task: 1.4.4
  • cumulus-cnm-to-granule: 1.5.4
• Add ssh2 to allowed list re: GHSA-652h-xwhf-q4h6

v9.9.1

Release v9.9.1

Fixed

• CUMULUS-2775
  • Updated @cumulus/api-client to not log an error for 201 response from updateGranule
• CUMULUS-2837
  • Update process-s3-dead-letter-archive to unpack SQS events in addition to
    Cumulus Messages
  • Update process-s3-dead-letter-archive to look up execution status using
    getCumulusMessageFromExecutionEvent (common method with sfEventSqsToDbRecords)
  • Move methods in api/lib/cwSfExecutionEventUtils to
    @cumulus/message/StepFunctions

Changed

• Updated version of @cumulus/cumulus-message-adapter-js from 2.0.3 to 2.0.4 for
  all Cumulus workflow tasks
• CUMULUS-2775
  • Changed @cumulus/api-client/invokeApi() to accept a single accepted status code or an array
    of accepted status codes via expectedStatusCodes