v8.0.0

Migration Steps

None.

Breaking Changes

• CUMULUS-2428
  • Changed /granules/bulk to use queueUrl property instead of a queueName property for setting the queue to use for scheduling bulk granule workflows

Notable changes

• Bulk granule operations endpoint now supports setting a custom queue for scheduling workflows via the queueUrl property in the request body. If provided, this value should be the full URL for an SQS queue.

All Changes

Added

• CUMULUS-2374
  • Add cookbook entry for queueing PostToCmr step
  • Add example workflow to go with cookbook
• CUMULUS-2421
  • Added experimental ecs_include_docker_cleanup_cronjob boolean variable to the Cumulus module to enable cron job to clean up docker root storage blocks in ECS cluster template for non-device-mapper storage drivers. Default value is false. This fulfills a specific user support request. This feature is otherwise untested and will remain so until we can iterate with a better, more general-purpose solution. Use of this feature is NOT recommended unless you are certain you need it.

Fixed

• CUMULUS-2281
  • Changed discover-granules task to write discovered granules directly to logger, instead of via environment variable. This fixes a problem where a large number of found granules prevents this lambda from running as an activity with an E2BIG error.

v6.0.1

Migration Steps

Cumulus 6.0.1 is a backport release to address urgent, specific issues faced by a DAAC with the Cumulus ECS Cluster.
This fix will not be carried forward in the same form, instead a similar but more useful and configurable change will be worked on the forward release builds. This is only intended as an update to 6.0.0 users, not for anyone on Cumulus v7+.

To enable the new ECS Cluster cleanup job, pass true to the ecs_include_docker_cleanup_cronjob variable in the Cumulus module.

All Changes

Added

• CUMULUS-2421
  • Added cron job to clean up docker storage blocks to ECS cluster template.
  • Added ecs_include_docker_cleanup_cronjob boolean variable to the Cumulus module to enable this behavior. Default value is false.

v7.2.0

Migration Steps

None

All Changes

Added

• CUMULUS-2346
  • Added orca API endpoint to @cumulus/api to get recovery status
  • Add CopyToGlacier step to example IngestAndPublishGranuleWithOrca workflow

v7.1.0

Migration Steps

None.

Notable changes

• sync-granule task will now properly handle syncing 0 byte files to S3

All Changes

Added

• tf-modules/cumulus module now supports a cmr_custom_host variable that can
  be used to set to an arbitray host for making CMR requests (e.g.
  https://custom-cmr-host.com).

• Added buckets variable to tf-modules/archive

• CUMULUS-2345

  • Deploy ORCA with Cumulus, see example/cumulus-tf/orca.tf and example/cumulus-tf/terraform.tfvars.example
  • Add CopyToGlacier step to example IngestAndPublishGranule workflow

• CUMULUS-2424

  • Added childWorkflowMeta to queue-pdrs config. An object passed to this config value will be merged into a child workflow message's meta object. For an example of how this can be used, see example/cumulus-tf/discover_and_queue_pdrs_with_child_workflow_meta_workflow.asl.json.

• CUMULUS-2427

  • Added support for using a custom queue with SQS and Kinesis rules. Whatever queue URL is set on the rule.queueUrl property will be used to schedule workflows for that rule. This change allows SQS/Kinesis rules to use any throttled queues defined for a deployment.

Fixed

• CUMULUS-2394
  • Updated PDR and Granule writes to check the step function workflow_start_time against
    the createdAt field for each record to ensure old records do not
    overwrite newer ones

Changed

• <prefix>-lambda-api-gateway IAM role used by API Gateway Lambda now
  supports accessing all buckets defined in your buckets variable except
  "internal" buckets

• Updated the default scroll duration used in ESScrollSearch and part of the
  reconcilation report functions as a result of testing and seeing timeouts
  at its current value of 2min.

• CUMULUS-2355

  • Added logic to disable /s3Credentials endpoint based upon value for
    environment variable DISABLE_S3_CREDENTIALS. If set to "true", the
    endpoint will not dispense S3 credentials and instead return a message
    indicating that the endpoint has been disabled.

• CUMULUS-2397

  • Updated /elasticsearch endpoint's reindex function to prevent
    reindexing when source and destination indices are the same.

• CUMULUS-2420

  • Updated test function waitForAsyncOperationStatus to take a retryObject
    and use exponential backoff. Increased the total test duration for both
    AsycOperation specs and the ReconciliationReports tests.

• CUMULUS-2427

  • Removed queueUrl from the parameters object for @cumulus/message/Build.buildQueueMessageFromTemplate
  • Removed queueUrl from the parameters object for @cumulus/message/Build.buildCumulusMeta

Fixed

• Fixed issue in @cumulus/ingest/S3ProviderClient.sync() preventing 0 byte files from being synced to S3.

Removed

• Removed variables from tf-modules/archive:
  • private_buckets
  • protected_buckets
  • public_buckets

v7.0.0

Migration Steps

None.

Breaking Changes

Operator API endpoints for the logs (/logs) will now throw an error unless you have integrated with Cloud Metrics

All Changes

Added

• CUMULUS-2345

  • Deploy ORCA with Cumulus, see example/cumulus-tf/orca.tf and example/cumulus-tf/terraform.tfvars.example
  • Add CopyToGlacier step to example IngestAndPublishGranule workflow

• CUMULUS-2376

  • Added cmrRevisionId as an optional parameter to post-to-cmr that will be used when publishing metadata to CMR.

Changed

• CUMULUS-2362

  • Logs endpoints only work with Metrics set up

• CUMULUS-2376

  • Can pass in an optional CMR revision id to the Post to CMR task to throw a version conflict error when multiple workflows are updating CMR metdata
  • Updated publishUMMGJSON2CMR to take in an optional revisionId parameter.
  • Updated publishUMMGJSON2CMR to throw an error if optional revisionId does not match resulting revision ID.
  • Updated publishECHO10XML2CMR to take in an optional revisionId parameter.
  • Updated publishECHO10XML2CMR to throw an error if optional revisionId does not match resulting revision ID.
  • Updated publish2CMR to take in optional cmrRevisionId.
  • Updated getWriteHeaders to take in an optional CMR Revision ID.
  • Updated ingestGranule to take in an optional CMR Revision ID to pass to getWriteHeaders.
  • Updated ingestUMMGranule to take in an optional CMR Revision ID to pass to getWriteHeaders.

• CUMULUS-2350

  • Updates the examples on the /s3credentialsREADME, to include Python and JavaScript code demonstrating how to refresh the s3credential for programatic access.

v6.0.0

Migration steps

• CUMULUS-2255 - Cumulus has upgraded its supported version of Terraform from 0.12.12 to 0.13.6. Please follow these instructions to upgrade your deployments before upgrading to this version.

• CUMULUS-2350

  • If the /s3credentialsREADME, does not appear to be working after deployment, manual redeployment of the API-gateway stage may be necessary to finish the deployment.

Breaking Changes

• CUMULUS-2255 - Cumulus has upgraded its supported version of Terraform from 0.12.12 to 0.13.6.

Notable changes

• CUMULUS-2368
  • Add QueueWorkflow task

All changes

Added

• CUMULUS-2291
  • Add provider filter to Granule Inventory Report
• CUMULUS-2300
  • Added childWorkflowMeta to queue-granules config. Object passed to this value will be merged into a child workflow message's meta object. For an example of how this can be used, see example/cumulus-tf/discover_granules_workflow.asl.json.
• CUMULUS-2350
  • Adds an unprotected endpoinpoint, /s3credentialsREADME, to the s3-credentials-endpoint that displays information on how to use the /s3credentials endpoint
• CUMULUS-2368
  • Add QueueWorkflow task
• CUMULUS-2391
  • Add reportToEms to collections.files file schema
• CUMULUS-2395
  • Add Core module parameter ecs_custom_sg_ids to Cumulus module to allow for
    custom security group mappings
• CUMULUS-2402
  • Officially expose sftp() for use in @cumulus/sftp-client

Changed

• CUMULUS-2255
  • Updated Terraform deployment code syntax for compatibility with version 0.13.6
• CUMULUS-2321
  • Updated API endpoint GET /reconciliationReports/{name} to return the pre-signed s3 URL in addition to report data

Fixed

• Updated hyrax-metadata-updates task so the opendap url has Type 'USE SERVICE API'

• CUMULUS-2310

  • Use valid filename for reconciliation report

• CUMULUS-2351

  • Inventory report no longer includes the File/Granule relation object in the okCountByGranules key of a report. The information is only included when a 'Granule Not Found' report is run.

v5.0.1

Migration Steps

None.

Breaking Changes

None.

All Changes

Changed

• CUMULUS-2344
  • Elasticsearch API now allows you to reindex to an index that already exists
  • If using the Change Index operation and the new index doesn't exist, it will be created
  • Regarding instructions for CUMULUS-2020 (in release 5.0.0), you can now do a change index operation before a reindex operation. This will
    ensure that new data will end up in the new index while Elasticsearch is reindexing.
  • Added a reindexing guide to the Cumulus documentation

v5.0.0

Migration Steps

• Elasticsearch data mappings have been updated to improve search. For example, case insensitive searching will now work (e.g. 'MOD' and 'mod' will return the same granule results). To use the improved Elasticsearch queries:

  • reindex to create a new index with the correct types.
  • Perform a change index operation to use the new index.
  • Note: You may want to upgrade to 5.0.1 as this allows for an alternative method of reindexing. Please see the new Elasticsearch reindexing guide.

• Because the egress_lambda_log_group and egress_lambda_log_subscription_filter resource were removed from the cumulus module, new definitions for these resources must be added to cumulus-tf/main.tf. For reference on how to define these resources, see example/cumulus-tf/thin_egress_app.tf.

  • The tea_stack_name variable being passed into the cumulus module should be removed

Breaking Changes

• CUMULUS-2020
  • Elasticsearch data mappings have been updated to improve search and the API has been updated to reflect those changes. See Migration steps on how to update the Elasticsearch mappings.

Notable Changes

• Added lzards-backup Core task
• Changed the formatting of granule CMR links: instead of a link to the /search/granules.json endpoint, now it is a direct link to /search/concepts/conceptid.format
• Improved PDR spec compliance of parse-pdr by updating @cumulus/pvl to parse fields in a manner more consistent with the PDR ICD, with respect to numbers and dates. Anything not matching the ICD expectations, or incompatible with Javascript parsing, will be parsed as a string instead.

All Changes

Added

• HYRAX-320

  • @cumulus/hyrax-metadata-updatesAdd component URI encoding for entry title id and granule ur to allow for values with special characters in them. For example, EntryTitleId 'Sentinel-6A MF/Jason-CS L2 Advanced Microwave Radiometer (AMR-C) NRT Geophysical Parameters' Now, URLs generated from such values will be encoded correctly and parsable by HyraxInTheCloud

• CUMULUS-1370

  • Add documentation for Getting Started section including FAQs

• CUMULUS-2092

  • Add documentation for Granule Not Found Reports

• CUMULUS-2219

  • Added lzards-backup Core task to facilitate making LZARDS backup requests in Cumulus ingest workflows

• CUMULUS-2280

  • In local api, retry to create tables if they fail to ensure localstack has had time to start fully.

• CUMULUS-2290

  • Add queryFields to granule schema, and this allows workflow tasks to add queryable data to granule record. For reference on how to add data to queryFields field, see example/cumulus-tf/kinesis_trigger_test_workflow.tf.

• CUMULUS-2318

  • Addedasync_operation_image as cumulus module variable to allow for override of the async_operation container image. Users can optionally specify a non-default docker image for use with Core async operations.

Changed

• CUMULUS-2020

  • Updated Elasticsearch mappings to support case-insensitive search

• CUMULUS-2124

  • cumulus-rds-tf terraform module now takes engine_version as an input variable.

• CUMULUS-2279

  • Changed the formatting of granule CMR links: instead of a link to the /search/granules.json endpoint, now it is a direct link to /search/concepts/conceptid.format

• CUMULUS-2296

  • Improved PDR spec compliance of parse-pdr by updating @cumulus/pvl to parse fields in a manner more consistent with the PDR ICD, with respect to numbers and dates. Anything not matching the ICD expectations, or incompatible with Javascript parsing, will be parsed as a string instead.

Removed

• CUMULUS-2258
  • Removed tea_stack_name variable from tf-modules/distribution/variables.tf and tf-modules/cumulus/variables.tf
  • Removed egress_lambda_log_group and egress_lambda_log_subscription_filter resources from tf-modules/distribution/main.tf

v4.0.0

Migration Steps

• In your cumulus-tf deployment, update the name of your cumulus_message_adapter_lambda_layer_arn variable for the cumulus module to cumulus_message_adapter_lambda_layer_version_arn. The value of the variable should remain the same (a layer version ARN of a Lambda layer for the cumulus-message-adapter.
• CUMULUS-2138 - Update all workflows using the MoveGranules step to add UpdateGranulesCmrMetadataFileLinksStepthat runs after it. See the example IngestAndPublishWorkflow for reference.
• CUMULUS-2251
  • Because it has been removed from the cumulus module, a new resource definition for egress_api_gateway_log_subscription_filter must be added to cumulus-tf/main.tf. For reference on how to define this resource, see example/cumulus-tf/main.tf.
• Upgrade your Cumulus dashboard to 3.0.0

Breaking Changes

• CUMULUS-2138 - CMR metadata update behavior has been removed from the move-granules task into a
  new update-granules-cmr-metadata-file-links task.
• CUMULUS-2216
  • /collection and /collection/active endpoints now return collections without granule aggregate statistics by default. The original behavior is preserved and can be found by including a query param of includeStats=true on the request to the endpoint. This is likely to affect the dashboard only but included here for the change of behavior.
• 1956
  • Update the name of the cumulus_message_adapter_lambda_layer_arn output from the cumulus-message-adapter module to cumulus_message_adapter_lambda_layer_version_arn. The output value has changed from being the ARN of the Lambda layer without a version to the ARN of the Lambda layer with a version.
  • Update the variable name in the cumulus and ingest modules from cumulus_message_adapter_lambda_layer_arn to cumulus_message_adapter_lambda_layer_version_arn

Notable Changes

• Add bulk granule reingest functionality
• Fix a deployment error caused by depending on the thin_egress_app module output for a resource count.
• Add filtering for granule inventory reports

All Changes

Added

• CUMULUS-2239
  • Add resource declaration to create a VPC endpoint in tea-map-cache module if deploy_to_ngap is false.
• CUMULUS-2063
  • Adds a new, optional query parameter to the /collections[&getMMT=true] and /collections/active[&getMMT=true] endpoints. When a user provides a value of true for getMMT in the query parameters, the endpoint will search CMR and update each collection's results with new key MMTLink containing a link to the MMT (Metadata Management Tool) if a CMR collection id is found.
• CUMULUS-2170
  • Adds ability to filter granule inventory reports
• CUMULUS-2211
  • Adds granules/bulkReingest endpoint to @cumulus/api
• CUMULUS-2251
  • Adds log_api_gateway_to_cloudwatch variable to example/cumulus-tf/variables.tf.
  • Adds log_api_gateway_to_cloudwatch variable to thin_egress_app module definition.

Changed

• CUMULUS-2248
  • Updates Integration Tests README to point to new fake provider template.
• CUMULUS-2216
  • /collection and /collection/active endpoints now return collections without granule aggregate statistics by default. The original behavior is preserved and can be found by including a query param of includeStats=true on the request to the endpoint.
  • The es/collections Collection class takes a new parameter includeStats. It no longer appends granule aggregate statistics to the returned results by default. One must set the new parameter to any non-false value.
• CUMULUS-2201
  • Update dbIndexer lambda to process requests in serial
  • Fixes ingestPdrWithNodeNameSpec parsePdr provider error
• CUMULUS-2251
  • Moves Egress Api Gateway Log Group Filter from tf-modules/distribution/main.tf to example/cumulus-tf/main.tf

Fixed

• CUMULUS-2251
  • This fixes a deployment error caused by depending on the thin_egress_app module output for a resource count.
• CUMULUS-2233
  • Fixes /s3credentials bug where the expiration time on the cookie was set to a time that is always expired, so authentication was never being recognized as complete by the API. Consequently, the user would end up in a redirect loop and requests to /s3credentials would never complete successfully. The bug was caused by the fact that the code setting the expiration time for the cookie was expecting a time value in milliseconds, but was receiving the expirationTime from the EarthdataLoginClient in seconds. This bug has been fixed by converting seconds into milliseconds. Unit tests were added to test that the expiration time has been converted to milliseconds and checking that the cookie's expiration time is greater than the current time.

Removed

• CUMULUS-2251
  • Removes tea_api_egress_log_group variable from tf-modules/distribution/variables.tf and tf-modules/cumulus/variables.tf.

v3.0.1

CUMULUS-2203

Update Core tasks to use cumulus-message-adapter-js v1.3.2 to resolve memory leak/lambda ENOMEM constant failure issue. This issue caused lambdas to slowly use all memory in the run environment and prevented AWS from halting/restarting warmed instances when task code was throwing consistent errors under load.

CUMULUS-2232

Updated versions for ajv, lodash, googleapis, archiver, and @cumulus/aws-client to remediate vulnerabilities found in SNYK scan.