v20.0.0

[v20.0.0] 2025-02-04

ElasticSearch Removal Phase 2 Release

Note: The most recent Cumulus Dashboard version v12.2.0 is currently in testing against these changes. If you rely on the Cumulus Dashboard, there may be incompatibilities with Cumulus Core v20.0.0. We are working quickly to complete testing and release any changes on the Cumulus Dashboard.

Breaking Changes

• CUMULUS-3934
  • Removed ecs_cluster_instance_allow_ssh resource.
  • The ecs_cluster_instance_allow_ssh was implemented before SSM hosts were deployed
    to NGAP accounts and allowed for SSHing into an instance from an SSH bastion, which no longer exists.
  • Tunneling into an EC2 via SSM is still supported. Users relying solely on SSH will need to transition to SSM.
• CUMULUS-2564
  • Updated sync-granule task to add useGranIdPath as a configuration flag.
    This modifies the task behavior to stage granules to
    <staging_path>/<collection_id>/<md5_granuleIdHash> to allow for better S3
    partitioning/performance for large collections.
    Because of this benefit
    the default has been set to true, however as sync-granules relies on
    object name collision, this configuration changes the duplicate collision
    behavior of sync-granules to be per-granule-id instead of per-collection
    when active.
    If the prior behavior is desired, please add "useGranIdPath": false to your
    task config in your workflow definitions that use sync-granule.
• CUMULUS-3698
  • GranuleSearch retrieving files/execution is toggled
    by setting "includeFullRecord" field to 'true' in relevant api endpoint params
  • GranuleSearch does not retrieve files/execution by default unless includeFullRecord is set to 'true'
  • @cumulus/db function getExecutionArnByGranuleCumulusId is removed. To replace this function use getExecutionInfoByGranuleCumulusId with parameter executionColumns set to ['arn'] or unset (['arn'] is the default argument)

Migration Notes

CUMULUS-3833 Migration of ReconciliationReports from DynamoDB to Postgres after Cumulus is upgraded.

To invoke the Lambda and start the ReconciliationReport migration, you can use the AWS Console or CLI:

aws lambda invoke --function-name $PREFIX-ReconciliationReportMigration $OUTFILE

• PREFIX is your Cumulus deployment prefix.
• OUTFILE (optional) is the filepath where the Lambda output will be saved.

CUMULUS-3967

External tooling making use of searchContext in the GET /granules/ endpoint will need to update to make use of standard pagination via limit and page scrolling, as searchContext is no longer supported/is an ES specific feature.

Replace ElasticSearch Phase 2 Epic

• CUMULUS-3967
  • Remove searchContext from API granules GET /granules endpoint.
  • Update relevant tests to validate expected behavior utilizing postgres pagination
• CUMULUS-3229
  • Remove ElasticSearch queries from Rule LIST endpoint
• CUMULUS-3230
  • Remove ElasticSearch dependency from Rule Endpoints
• CUMULUS-3231
  • Updated API pdrs LIST endpoint to query postgres
• CUMULUS-3232
  • Update API PDR endpoints DEL and GET to not update Elasticsearch
• CUMULUS-3233
  • Updated providers list api endpoint and added ProviderSearch class to query postgres
  • Removed Elasticsearch dependency from providers endpoints
• CUMULUS-3235
  • Updated asyncOperations api endpoint to query postgres
• CUMULUS-3236
  • Update API AsyncOperation endpoints POST and DEL to not update
    Elasticsearch
  • Update @cumlus/api/ecs/async-operation to not update Elasticsearch index when
    reporting status of async operation
• CUMULUS-3698
  • GranuleSearch now can retrieve associated files for granules
  • GranuleSearch now can retrieve latest associated execution for granules
• CUMULUS-3806
  • Update @cumulus/db/search to allow for ordered collation as a
    dbQueryParameter
  • Update @cumulus/db/search to allow dbQueryParameters.limit to be set to
    null to allow for optional unlimited page sizes in search results
  • Update/add type annotations/logic fixes to @cumulus/api reconciliation report code
  • Annotation/typing fixes to @cumulus/cmr-client
  • Typing fixes to @cumulus/db
  • Re-enable Reconciliation Report integration tests
  • Update @cumulus/client/CMR.getToken to throw if a non-launchpad token is requested without a username
  • Update Inventory and Granule Not Found reports to query postgreSQL
    database instead of elasticsearch
  • Update @cumulus/db/lib/granule.getGranulesByApiPropertiesQuery to
    allow order by collation to be optionally specified
  • Update @cumulus/db/lib/granule.getGranulesByApiPropertiesQuery to
    be parameterized and include a modifier on temporalBoundByCreatedAt
  • Remove endpoint call to and all tests for Internal Reconciliation Reports
    and updated API to throw an error if report is requested
  • Update Orca reconciliation reports to pull granules for comparison from
    postgres via getGranulesByApiPropertiesQuery
• CUMULUS-3837
  • Added reconciliation_reports table in RDS, including indexes
  • Created pg model, types, and translation for reconciliationReports in @cumulus/db
• CUMULUS-3833
  • Created api types for reconciliation_reports in @cumulus/types/api
  • Updated reconciliation reports lambda to write to new RDS table instead of Dynamo
  • Updated @cumulus/api/endpoints/reconciliation-reports getReport and deleteReport to work with the new RDS table instead of Dynamo
• CUMULUS-3718
  • Updated reconciliation_reports list api endpoint and added ReconciliationReportSearch class to query postgres
  • Added reconciliationReports type to stats endpoint, so aggregate query will work for reconciliation reports
• CUMULUS-3859
  • Updated @cumulus/api/bin/serveUtils to no longer add records to ElasticSearch
  • Removed ElasticSearch from local API server code
  • Updated CollectionSearch to filter granule fields in addition to time frame for active collections
• CUMULUS-3847
  • remove remaining ES indexing in code and tests
  • for asyncOperations test data, change any ES related values to other options
  • remove code from @cumulus/api/lambdas/cleanExecutions leaving a dummy handler, as the code worked with ES. lambda will be rewritten with CUMULUS-3982
  • remove @cumulus/api/endpoints/elasticsearch, @cumulus/api/lambdas/bootstrap, and @cumulus/api/lambdas/index-from-database
• CUMULUS-3983
  • Removed elasticsearch references used in in cumulus tf-modules

Added

• CUMULUS-3757
  • Added a /granules endpoint PATCH/bulkPatchGranuleCollection which updates a batch of granule records collectionId to a new collectionId. This endpoint takes a list of granules and a collectionId, updating the granules' to the collectionId passed with the payload in postgres.
  • Added a /granules endpoint PATCH/bulkPatch which applies PATCH to a list of granules. For its payload, this endpoint takes a list of granules (the updates to be made to the granule, similar to the pre-existing PATCH), a dbConcurrency and dbMaxPool variables for configuring concurrency and database thoroughput for postgres to tailor to performance and database needs.
• CUMULUS-3919
  • Added terraform variables disableSSL and rejectUnauthorized to tf-modules/cumulus-rds-tf module.
• CUMULUS-3959
  • Added documentation to help DAACs troubleshoot database migration issues.
• CUMULUS-3978
  • Added iops and throughput options to elasticsearch_config variable
    in tf-modules/data-persistence; These two options are necessary for gp3 EBS volume type.

Changed

• CUMULUS-3947
  • Bump @cumulus/cumulus-message-adapter-js to version 2.3.0. This will explicitly put the Python cumulus-message-adapter spawn into UTF-8 mode. See https://github.com/nasa/cumulus-message-adapter-js/releases/tag/v2.3.0
• CUMULUS-3967
  • Pinned @aws-sdk/client-s3 in @cumulus/aws-client to 3.726.0 to address breaking changes/incompatibility in releases > 3.726.0
  • Pinned @aws-sdk/client-s3 in @cumulus/lib-storage to 3.726.0 to address breaking changes/incompatibility in releases > 3.726.0
• CUMULUS-3940
  • Added 'dead_letter_recovery_cpu' and 'dead_letter_recovery_memory' to cumulus and archive module configuration to allow configuration of the dead_letter_recovery_operation task definition to better allow configuration of the tool's operating environment.
  • Updated the dead letter recovery tool to utilize it's own log group "${var.prefix}-DeadLetterRecoveryEcsLogs"
  • Added batchSize, concurrency and dbMaxPool options to /endpoints/recoverCumulusMessage (note these values are correct at time of this release only):
    • batchSize - specifies how many DLA objects to read from S3 and hold in memory. Defaults to 1000.
    • concurrency - specifies how many messages to process at the same time. Defaults to 30.
    • dbMaxPool - specifies how many database connections to allow the process to utilize. Defaults to 30. Process should at minimum the value set for concurrency.
  • Add API memory-constrained performance test to test minimum functionality under default+ configuration
  • Updated @cumulus/async-operations.startAsyncOperation to take containerName` as a parameter name, allowing it to specify a container other than the default 'AsyncOperations' container
• CUMULUS-3759
  • Migrated tf-modules/cumulus/ecs_cluster ECS Autoscaling group from launch configurations to launch templates
• **...

v18.5.3

Release v18.5.3

Please note changes in v18.5.3 may not yet be released in future versions, as this
is a backport/patch release on the v18.5.x series of releases. Updates that are
included in the future will have a corresponding CHANGELOG entry in future releases.

Added

• CUMULUS-3757
  • Added a /granules endpoint PATCH/bulkPatchGranuleCollection which updates a batch of granule records collectionId to a new collectionId. This endpoint takes a list of granules, a collectionId, and an esConcurrency variable, updating the granules' to the collectionId passed with the payload in both postgres and elasticsearch, while providing concurrency for updating elasticsearch to tailor for performance and database needs.
  • Added a /granules endpoint PATCH/bulkPatch which applies PATCH to a list of granules. For its payload, this endpoint takes a list of granules (the updates to be made to the granule, similar to the pre-existing PATCH), a dbConcurrency and dbMaxPool variables for configuring concurrency and database thoroughput for postgres to tailor to performance and database needs.
• CUMULUS-3978
  • Added iops and throughput options to elasticsearch_config variable
    in tf-modules/data-persistence; These two options are necessary for gp3 EBS volume type.

Changed

• CUMULUS-3967

  • Pinned @aws-sdk/client-s3 in @cumulus/aws-client to 3.726.0 to address breaking changes/incompatibility in releases > 3.726.0
  • Pinned @aws-sdk/client-s3 in @cumulus/lib-storage to 3.726.0 to address breaking changes/incompatibility in releases > 3.726.0

• CUMULUS-3940

  • Added 'dead_letter_recovery_cpu' and 'dead_letter_recovery_memory' to cumulus and archive module configuration to allow configuration of the dead_letter_recovery_operation task definition to better allow configuration of the tool's operating environment.
  • Updated the dead letter recovery tool to utilize it's own log group "${var.prefix}-DeadLetterRecoveryEcsLogs"
  • Added batchSize, concurrency and dbMaxPool options to /endpoints/recoverCumulusMessage (note these values are correct at time of this release only):
    • batchSize - specifies how many DLA objects to read from S3 and hold in memory. Defaults to 1000.
    • concurrency - specifies how many messages to process at the same time. Defaults to 30.
    • dbMaxPool - specifies how many database connections to allow the process to utilize. Defaults to 30. Process should at minimum the value set for concurrency.
  • Add API memory-constrained performance test to test minimum functionality under default+ configuration
  • Updated @cumulus/async-operations.startAsyncOperation to take containerName` as a parameter name, allowing it to specify a container other than the default 'AsyncOperations' container

Fixed

• CUMULUS-3940
  • Updated process-s3-dead-letter-archive and downstream calls to pass in a esClient to writeRecordsFunction and update downstream calls to utilize the client.
• CUMULUS-3981
  • Added required $metadata field when creating new instance of ServiceException.

v18.5.2

[v18.5.2] 2024-12-12

Breaking Changes

• CUMULUS-3934
  • Removed ecs_cluster_instance_allow_ssh resource.
  • The ecs_cluster_instance_allow_ssh was implemented before SSM hosts were deployed
    to NGAP accounts and allowed for SSHing into an instance from an SSH bastion, which no longer exists.
  • Tunneling into an EC2 via SSM is still supported. Users relying solely on SSH will need to transition to SSM.

Changed

• CUMULUS-3936,CUMULUS-3948
  • Updated tf-modules/cumulus/ecs_cluster_instance_autoscaling_cf_template.yml.tmpl
    user-data for compatibility with Amazon Linux 2023 AMI
  • Fixed tf-modules/cumulus scripts to use Instance Metadata Service V2
  • Updated fake-provider-cf.yml to work for Amazon Linux 2023 AMI
• CUMULUS-3941
  • Updated SendPan task to generate short pan with FAILED disposition.
• CUMULUS-3955
  • Removed VACUUM statements from db migrations. In cases where the PG database is very large, these queries
    can take a long time and exceed the Lambda timeout, causing failures on deployment.

Fixed

• Security Vulnerabilities
  • Updated @octokit/graphql from 2.1.1 to ^2.3.0 to address [CVE-2024-21538]
    (GHSA-3xgq-45jj-v275)

v19.2.0-alpha.1

Release v19.2.0-alpha.1

From feature/es-phase-2 deafd8b

v19.2.0-alpha.0

Release v19.2.0-alpha.0

#3858
The release branch was checked out from feature/es-phase-2 34a9205

v19.1.0

[v19.1.0] 2024-10-07

Migration Notes

This release contains changes listed here as well as changes listed in v19.0.0,
despite v19.0.0 being deprecated. Please review Changelog entries and Migration Notes for
each Cumulus version between your current version and v19.1.0 as normal.

Added

• CUMULUS-3020

  • Updated sfEventSqsToDbRecords to allow override of the default value
    (var.rds_connection_timing_configuration.acquireTimeoutMillis / 1000) + 60)
    via a key 'sfEventSqsToDbRecords' on var.lambda_timeouts on the main cumulus module/archive module

  Please note - updating this configuration is for adavanced users only. Value changes will modify the visibility
  timeout on sfEventSqsToDbRecordsDeadLetterQueue and sfEventSqsToDbRecordsInputQueue and may lead to system
  instability.

• CUMULUS-3756

  • Added excludeFileRegex configuration to UpdateGranulesCmrMetadataFileLinks
  • This is to allow files matching specified regex to be excluded when updating the Related URLs list
  • Defaults to the current behavior of excluding no files.

• CUMULUS-3773

  • Added sftpFastDownload configuration to SyncGranule task.
  • Updated @cumulus/sftp-client and @cumulus/ingest/SftpProviderClient to support both regular and fastDownload.
  • Added sftp support to FakeProvider
  • Added sftp integration test

Changed

• CUMULUS-3928
  • updated publish scripting to use [email protected] for user email
  • updated publish scripting to use esm over common import of latest-version
  • updated bigint testing to remove intermitted failure source.
  • updated postgres dependency version
• CUMULUS-3838
  • Updated python dependencies to latest:
    • cumulus-process-py 1.4.0
    • cumulus-message-adapter-python 2.3.0
• CUMULUS-3906
  • Bumps example ORCA deployment to version v10.0.1.

Fixed

• CUMULUS-3902
  • Update error handling to use AWS SDK V3 error classes instead of properties on js objects

v18.5.1

Release v18.5.1

Please note This is a backport release - changes in v18.5.1 may not yet be released in future versions, as this
is a backport/patch release on the v18.5.x series of releases. Updates that are
included in the future will have a corresponding CHANGELOG entry in future releases.

Added

• CUMULUS-3773
  • Added sftpFastDownload configuration to SyncGranule task.
  • Updated @cumulus/sftp-client and @cumulus/ingest/SftpProviderClient to support both regular and fastDownload.
  • Added sftp support to FakeProvider
  • Added sftp integration test
• CUMULUS-3756
  • Added excludeFileRegex configuration to UpdateGranulesCmrMetadataFileLinks
  • This is to allow files matching specified regex to be excluded when updating the Related URLs list
  • Defaults to the current behavior of excluding no files.
• CUMULUS-3919
  • Added terraform variables disableSSL and rejectUnauthorized to tf-modules/cumulus-rds-tf module.

Changed

• CUMULUS-3928
  • updated publish scripting to use [email protected] for user email
  • updated publish scripting to use esm over common import of latest-version
  • updated bigint testing to remove intermitted failure source.
  • updated postgres dependency version
• CUMULUS-3838
  • Updated python dependencies to latest:
    • cumulus-process-py 1.4.0
    • cumulus-message-adapter-python 2.3.0

Fixed

• CUMULUS-3902
  • Update error handling to use AWS SDK V3 error classes instead of properties on js objects

v18.5.0

[v18.5.0] 2024-10-03

Migration Notes

CUMULUS-3536 Upgrading from Aurora Serverless V1 to V2

• The updates in CUMULUS-3536 require an upgrade of the postgres database.
  Please follow [Upgrading from Aurora Serverless V1 to V2]
  (https://nasa.github.io/cumulus/docs/next/upgrade-notes/serverless-v2-upgrade)

Added

• CUMULUS-3536
  • Added rejectUnauthorized = false to db-provision-user-database as the Lambda
    does not have the Serverless v2 SSL certifications installed.

Changed

• CUMULUS-3725
  • Updated the default parameter group for cumulus-rds-tf to set force_ssl
    to 0. This setting for the Aurora Serverless v2 database allows non-SSL
    connections to the database, and is intended to be a temporary solution
    until Cumulus has been updated to import the RDS rds-ca-rsa2048-g1 CA bundles in Lambda environments.
    See CUMULUS-3724.

Fixed

• CUMULUS-3901
  • Fix error checking in @cumulus/errors to use Error.name in addition to Error.code
• CUMULUS-3824
  • Added the missing double quote in ecs_cluster autoscaling cf template
• CUMULUS-3846
  • improve reliability of unit tests
    • tests for granules api get requests separated out to new file
    • cleanup of granule database resources to ensure no overlap
    • ensure uniqueness of execution names from getWorkflowNameIntersectFromGranuleIds
    • increase timeout in aws-client tests
• Snyk
  • Upgraded moment from 2.29.4 to 2.30.1
  • Upgraded pg from ~8.10 to ~8.12

v18.3.5

[v18.3.5] 2024-09-19

• CUMULUS-3901
  • Fix error checking in @cumulus/errors to use Error.name in addition to Error.code
  • Fix async Operation migration to include Migration Count Report

v19.0.0

Deprecated

This release has been deprecated in favor of the 18.5->19.1 release series.

[v19.0.0] 2024-08-28

Dashboard Requirement

• Cumulus Dashboard v12.2.0

Breaking Changes

• This release includes Replace ElasicSearch Phase 1 updates, we no longer save collection/granule/execution records to
  ElasticSearch, the collections/granules/executions API endpoints are updated to perform operations on the postgres database.

Migration Notes

CUMULUS-3792 Add database indexes. Please follow the instructions before upgrading Cumulus

• The updates in CUMULUS-3792 require a manual update to the postgres database in the production environment.
  Please follow [Update Table Indexes for CUMULUS-3792]
  (https://nasa.github.io/cumulus/docs/next/upgrade-notes/update_table_indexes_CUMULUS_3792)

Replace ElasticSearch Phase 1

• CUMULUS-3238
  • Removed elasticsearch dependency from collections endpoint
• CUMULUS-3239
  • Updated executions list api endpoint and added ExecutionSearch class to query postgres
• CUMULUS-3240
  • Removed Elasticsearch dependency from executions endpoints
• CUMULUS-3639
  • Updated /collections/active endpoint to query postgres
• CUMULUS-3640
  • Removed elasticsearch dependency from granules endpoint
• CUMULUS-3641
  • Updated collections api endpoint to query postgres instead of elasticsearch except if includeStats is in the query parameters
• CUMULUS-3642
  • Adjusted queries to improve performance:
    • Used count(*) over count(id) to count rows
    • Estimated row count for large tables (granules and executions) by default for basic query
  • Updated stats summary to default to the last day
  • Updated ExecutionSearch to not include asyncOperationId by default
• CUMULUS-3688
  • Updated stats api endpoint to query postgres instead of elasticsearch
• CUMULUS-3689
  • Updated stats/aggregate api endpoint to query postgres instead of elasticsearch
  • Created a new StatsSearch class for querying postgres with the stats endpoint
• CUMULUS-3692
  • Added @cumulus/db/src/search BaseSearch and GranuleSearch classes to
    support basic queries for granules
  • Updated granules List endpoint to query postgres for basic queries
• CUMULUS-3693
  • Added functionality to @cumulus/db/src/search to support range queries
• CUMULUS-3694
  • Added functionality to @cumulus/db/src/search to support term queries
  • Updated BaseSearch and GranuleSearch classes to support term queries for granules
  • Updated granules List endpoint to search postgres
• CUMULUS-3695
  • Updated granule list api endpoint and BaseSearch class to handle sort fields
• CUMULUS-3696
  • Added functionality to @cumulus/db/src/search to support terms, not and exists queries
• CUMULUS-3699
  • Updated collections api endpoint to be able to support includeStats query string parameter
• CUMULUS-3792
  • Added database indexes to improve search performance