Skip to content

v2.0.0
Compare
Choose a tag to compare
@laurenfrederick laurenfrederick released this 23 Jul 23:59
· 3363 commits to master since this release
v2.0.0
404fd95
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Migration Steps

  • Upgrade your Cumulus dashboard to version 1.10.0

  • Due to an issue with the AWS API Gateway and how the Thin Egress App Cloudformation template applies updates, you may need to redeploy your thin-egress-app-EgressGateway manually as a one time migration step. If your deployment fails with an error similar to:

    Error: Lambda function (<stack>-tf-TeaCache) returned error: ({"errorType":"HTTPError","errorMessage":"Response code 404 (Not Found)"})

    Then follow the AWS instructions to Redeploy a REST API to a stage for your egress API and re-run terraform apply.

  • Update rules to specify the provider_path and workflows to get the provider_path from config.meta.provider_path. Collections no longer support the provider_path property.

  • Cumulus tasks using the cumuluss/cumulus-ecs-task Docker image must be updated to cumuluss/cumulus-ecs-task:1.7.0 to accommodate an upgrade to Node 12.18.0.

Breaking Changes

  • The minimum supported version of all published Cumulus packages is now Node
    12.18.0

    • Tasks using the cumuluss/cumulus-ecs-task Docker image must be updated to
      cumuluss/cumulus-ecs-task:1.7.0. This can be done by updating the image
      property of any tasks defined using the cumulus_ecs_service Terraform
      module.

  • CUMULUS-1969

    • The DiscoverPdrs task now expects provider_path to be provided at
      event.config.provider_path, not event.config.collection.provider_path
    • event.config.provider_path is now a required parameter of the
      DiscoverPdrs task
    • event.config.collection is no longer a parameter to the DiscoverPdrs
      task
    • Collections no longer support the provider_path property. The tasks that
      relied on that property are now referencing config.meta.provider_path.
      Workflows should be updated accordingly.

  • CUMULUS-1977

    • Moved bulk granule deletion endpoint from /bulkDelete to
      /granules/bulkDelete

Breaking Code Changes

  • Changes to the @cumulus/cumulus-api package

    • The CumulusApiClientError class must now be imported using
      const { CumulusApiClientError } = require('@cumulus/cumulus-api/CumulusApiClientError')

  • The @cumulus/sftp-client/SftpClient class must now be imported using
    const { SftpClient } = require('@cumulus/sftp-client');

  • Instances of @cumulus/ingest/SftpProviderClient no longer implicitly connect
    when download, list, or sync are called. You must call connect on the
    provider client before issuing one of those calls. Failure to do so will
    result in a "Client not connected" exception being thrown.

  • Instances of @cumulus/ingest/SftpProviderClient no longer implicitly
    disconnect from the SFTP server when list is called.

  • Instances of @cumulus/sftp-client/SftpClient must now be explicitly closed
    by calling .end()

  • Instances of @cumulus/sftp-client/SftpClient no longer implicitly connect to
    the server when download, unlink, syncToS3, syncFromS3, and list are
    called. You must explicitly call connect before calling one of those
    methods.

  • Changes to the @cumulus/common package

    • cloudwatch-event.getSfEventMessageObject() now returns undefined if the
      message could not be found or could not be parsed. It previously returned
      null.
    • S3KeyPairProvider.decrypt() now throws an exception if the bucket
      containing the key cannot be determined.
    • S3KeyPairProvider.decrypt() now throws an exception if the stack cannot be
      determined.
    • S3KeyPairProvider.encrypt() now throws an exception if the bucket
      containing the key cannot be determined.
    • S3KeyPairProvider.encrypt() now throws an exception if the stack cannot be
      determined.
    • sns-event.getSnsEventMessageObject() now returns undefined if it could
      not be parsed. It previously returned null.
    • The aws module has been removed.
    • The BucketsConfig.buckets property is now read-only and private
    • The test-utils.validateConfig() function now resolves to undefined
      rather than true.
    • The test-utils.validateInput() function now resolves to undefined rather
      than true.
    • The test-utils.validateOutput() function now resolves to undefined
      rather than true.
    • The static S3KeyPairProvider.retrieveKey() function has been removed.

  • Changes to the @cumulus/cmrjs package

    • @cumulus/cmrjs.constructOnlineAccessUrl() and
      @cumulus/cmrjs/cmr-utils.constructOnlineAccessUrl() previously took a
      buckets parameter, which was an instance of
      @cumulus/common/BucketsConfig. They now take a bucketTypes parameter,
      which is a simple object mapping bucket names to bucket types. Example:
      { 'private-1': 'private', 'public-1': 'public' }
    • @cumulus/cmrjs.reconcileCMRMetadata() and
      @cumulus/cmrjs/cmr-utils.reconcileCMRMetadata() now take a required
      bucketTypes parameter, which is a simple object mapping bucket names to
      bucket types. Example: { 'private-1': 'private', 'public-1': 'public' }
    • @cumulus/cmrjs.updateCMRMetadata() and
      @cumulus/cmrjs/cmr-utils.updateCMRMetadata() previously took an optional
      inBuckets parameter, which was an instance of
      @cumulus/common/BucketsConfig. They now take a required bucketTypes
      parameter, which is a simple object mapping bucket names to bucket types.
      Example: { 'private-1': 'private', 'public-1': 'public' }

  • Changes to @cumulus/aws-client/S3

    • The signature of the getObjectSize function has changed. It now takes a
      params object with three properties:
      • s3: an instance of an AWS.S3 object
      • bucket
      • key
    • The getObjectSize function will no longer retry if the object does not
      exist

  • CUMULUS-1861

    • @cumulus/message/Collections.getCollectionIdFromMessage now throws a
      CumulusMessageError if collectionName and collectionVersion are missing
      from meta.collection. Previously this method would return
      'undefined___undefined' instead
    • @cumulus/integration-tests/addCollections now returns an array of collections that
      were added rather than the count of added collections

  • CUMULUS-1930

    • The @cumulus/common/util.uuid() function has been removed

  • CUMULUS-1955

    • @cumulus/aws-client/S3.multipartCopyObject now returns an object with the
      AWS etag of the destination object
    • @cumulus/ingest/S3ProviderClient.list now sets a file object's path
      property to undefined instead of null when the file is at the top level
      of its bucket
    • The sync methods of the following classes in the @cumulus/ingest package
      now return an object with the AWS s3uri and etag of the destination file
      (they previously returned only a string representing the S3 URI)
      • FtpProviderClient
      • HttpProviderClient
      • S3ProviderClient
      • SftpProviderClient

  • CUMULUS-1958

    • The following methods exported from @cumulus/cmr-js/cmr-utils were made
      async, and added distributionBucketMap as a parameter:
      • constructOnlineAccessUrl
      • generateFileUrl
      • reconcileCMRMetadata
      • updateCMRMetadata

Notable Changes

  • CUMULUS-1991

    • Updated CMR metadata generation to use "Download file.hdf" (where file.hdf is the filename of the given resource) as the resource description instead of "File to download"
    • CMR metadata updates now respect changes to resource descriptions (previously only changes to resource URLs were respected)

  • CUMULUS-1902

  • CUMULUS-1417

    • Added a checksumFor property to collection files config. Set this
      property on a checksum file's definition matching the regex of the target
      file. More details in the 'Data Cookbooks
      Setup'
      documentation.
    • Added checksumFor validation to collections model.

  • CUMULUS-1956

    • The /s3credentials endpoint that is deployed as part of distribution now
      supports authentication using tokens created by a different application. If
      a request contains the EDL-ClientId and EDL-Token headers,
      authentication will be handled using that token rather than attempting to
      use OAuth.
    • If the s3Credentials endpoint is invoked with an EDL token and an
      X-Request-Id header, that X-Request-Id header will be forwarded to
      Earthata Login.

  • CUMULUS-1958

    • Add the ability for users to specify a bucket_map_key to the cumulus
      terraform module as an override for the default .yaml values that are passed
      to TEA by Core. Using this option requires that each configured
      Cumulus 'distribution' bucket (e.g. public/protected buckets) have a single
      TEA mapping. Multiple maps per bucket are not supported.
    • Updated Generating a distribution URL, the MoveGranules task and all CMR
      reconciliation functionality to utilize the TEA bucket map override.
    • Updated deploy process to utilize a bootstrap 'tea-map-cache' lambda that
      will, after deployment of Cumulus Core's TEA instance, query TEA for all
      protected/public buckets and generate a mapping configuration used
      internally by Core. This object is also exposed as an output of the Cumulus
      module as distribution_bucket_map.
    • docs

  • CUMULUS-1982

    • The globalConnectionLimit property of providers is now optional and
      defaults to "unlimited"

  • CUMULUS-2016

    • Upgrade TEA to version 79

All Changes

Added

  • CUMULUS-1902

  • CUMULUS-2058

    • Added lambda_processing_role_name as an output from the cumulus module
      to provide the processing role name

  • CUMULUS-1417

    • Added a checksumFor property to collection files config. Set this
      property on a checksum file's definition matching the regex of the target
      file. More details in the 'Data Cookbooks
      Setup'
      documentation.
    • Added checksumFor validation to collections model.

  • CUMULUS-1956

    • Added @cumulus/earthata-login-client package
    • The /s3credentials endpoint that is deployed as part of distribution now
      supports authentication using tokens created by a different application. If
      a request contains the EDL-ClientId and EDL-Token headers,
      authentication will be handled using that token rather than attempting to
      use OAuth.
    • @cumulus/earthata-login-client.getTokenUsername() now accepts an
      xRequestId argument, which will be included as the X-Request-Id header
      when calling Earthdata Login.
    • If the s3Credentials endpoint is invoked with an EDL token and an
      X-Request-Id header, that X-Request-Id header will be forwarded to
      Earthata Login.

  • CUMULUS-1957

    • If EDL token authentication is being used, and the EDL-Client-Name header
      is set, @the-client-name will be appended to the end of the Earthdata
      Login username that is used as the RoleSessionName of the temporary IAM
      credentials. This value will show up in the AWS S3 server access logs.

  • CUMULUS-1958

    • Add the ability for users to specify a bucket_map_key to the cumulus
      terraform module as an override for the default .yaml values that are passed
      to TEA by Core. Using this option requires that each configured
      Cumulus 'distribution' bucket (e.g. public/protected buckets) have a single
      TEA mapping. Multiple maps per bucket are not supported.
    • Updated Generating a distribution URL, the MoveGranules task and all CMR
      reconciliation functionality to utilize the TEA bucket map override.
    • Updated deploy process to utilize a bootstrap 'tea-map-cache' lambda that
      will, after deployment of Cumulus Core's TEA instance, query TEA for all
      protected/public buckets and generate a mapping configuration used
      internally by Core. This object is also exposed as an output of the Cumulus
      module as distribution_bucket_map.

  • CUMULUS-1961

    • Replaces DynamoDB for Elasticsearch for reconciliationReportForCumulusCMR
      comparisons between Cumulus and CMR.

  • CUMULUS-1970

    • Created the add-missing-file-checksums workflow task
    • Added @cumulus/aws-client/S3.calculateObjectHash() function
    • Added @cumulus/aws-client/S3.getObjectReadStream() function

  • CUMULUS-1887

    • Add additional fields to the granule CSV download file

  • CUMULUS-2019

    • Add infix search to es query builder @cumulus/api/es/es/queries to
      support partial matching of the keywords

Changed

  • CUMULUS-2032

    • Updated @cumulus/ingest/HttpProviderClient to utilize a configuration key
      httpListTimeout to set the default timeout for discovery HTTP/HTTPS
      requests, and updates the default for the provider to 5 minutes (300 seconds).
    • Updated the DiscoverGranules and DiscoverPDRs tasks to utilize the updated
      configuration value if set via workflow config, and updates the default for
      these tasks to 5 minutes (300 seconds).

  • CUMULUS-176

    • The API will now respond with a 400 status code when a request body contains
      invalid JSON. It had previously returned a 500 status code.

  • CUMULUS-1861

    • Updates Rule objects to no longer require a collection.
    • Changes the DLQ behavior for sfEventSqsToDbRecords and
      sfEventSqsToDbRecordsInputQueue. Previously failure to write a database
      record would result in lambda success, and an error log in the CloudWatch
      logs. The lambda has been updated to manually add a record to
      the sfEventSqsToDbRecordsDeadLetterQueue if the granule, execution, or
      pdr record fails to write, in addition to the previous error logging.

  • CUMULUS-1956

    • The /s3credentials endpoint that is deployed as part of distribution now
      supports authentication using tokens created by a different application. If
      a request contains the EDL-ClientId and EDL-Token headers,
      authentication will be handled using that token rather than attempting to
      use OAuth.

  • CUMULUS-1977

    • API endpoint POST /granules/bulk now returns a 202 status on a successful
      response instead of a 200 response
    • API endpoint DELETE /granules/<granule-id> now returns a 404 status if the
      granule record was already deleted
    • @cumulus/api/models/Granule.update() now returns the updated granule
      record
    • Implemented POST /granules/bulkDelete API endpoint to support deleting
      granules specified by ID or returned by the provided query in the request
      body. If the request is successful, the endpoint returns the async operation
      ID that has been started to remove the granules.
    • Added @cumulus/api/models/Granule.getRecord() method to return raw record
      from DynamoDB
    • Added @cumulus/api/models/Granule.delete() method which handles deleting
      the granule record from DynamoDB and the granule files from S3

  • CUMULUS-1982

    • The globalConnectionLimit property of providers is now optional and
      defaults to "unlimited"

  • CUMULUS-1997

    • Added optional launchpad configuration to @cumulus/hyrax-metadata-updates task config schema.

  • CUMULUS-1991

    • @cumulus/cmrjs/src/cmr-utils/constructOnlineAccessUrls() now throws an error if cmrGranuleUrlType = "distribution" and no distribution endpoint argument is provided

  • CUMULUS-2011

    • Reconciliation reports are now generated within an AsyncOperation

  • CUMULUS-2016

    • Upgrade TEA to version 79

Fixed

  • CUMULUS-1991

    • Added missing DISTRIBUTION_ENDPOINT environment variable for API lambdas. This environment variable is required for API requests to move granules.

  • CUMULUS-1961

    • Fixed granules and executions query params not getting sent to API in granule list operation in @cumulus/api-client

Deprecated

  • @cumulus/aws-client/S3.calculateS3ObjectChecksum()

  • @cumulus/aws-client/S3.getS3ObjectReadStream()

  • @cumulus/common/log.convertLogLevel()

  • @cumulus/collection-config-store

  • @cumulus/common/util.sleep()

  • CUMULUS-1930

    • @cumulus/common/log.convertLogLevel()
    • @cumulus/common/util.isNull()
    • @cumulus/common/util.isUndefined()
    • @cumulus/common/util.negate()
    • @cumulus/common/util.noop()
    • @cumulus/common/util.isNil()
    • @cumulus/common/util.renameProperty()
    • @cumulus/common/util.lookupMimeType()
    • @cumulus/common/util.thread()
    • @cumulus/common/util.mkdtempSync()

Removed

  • The deprecated @cumulus/common.bucketsConfigJsonObject function has been
    removed
  • The deprecated @cumulus/common.CollectionConfigStore class has been removed
  • The deprecated @cumulus/common.concurrency module has been removed
  • The deprecated @cumulus/common.constructCollectionId function has been
    removed
  • The deprecated @cumulus/common.launchpad module has been removed
  • The deprecated @cumulus/common.LaunchpadToken class has been removed
  • The deprecated @cumulus/common.Semaphore class has been removed
  • The deprecated @cumulus/common.stringUtils module has been removed
  • The deprecated @cumulus/common/aws.cloudwatchlogs function has been removed
  • The deprecated @cumulus/common/aws.deleteS3Files function has been removed
  • The deprecated @cumulus/common/aws.deleteS3Object function has been removed
  • The deprecated @cumulus/common/aws.dynamodb function has been removed
  • The deprecated @cumulus/common/aws.dynamodbDocClient function has been
    removed
  • The deprecated @cumulus/common/aws.getExecutionArn function has been removed
  • The deprecated @cumulus/common/aws.headObject function has been removed
  • The deprecated @cumulus/common/aws.listS3ObjectsV2 function has been removed
  • The deprecated @cumulus/common/aws.parseS3Uri function has been removed
  • The deprecated @cumulus/common/aws.promiseS3Upload function has been removed
  • The deprecated @cumulus/common/aws.recursivelyDeleteS3Bucket function has
    been removed
  • The deprecated @cumulus/common/aws.s3CopyObject function has been removed
  • The deprecated @cumulus/common/aws.s3ObjectExists function has been removed
  • The deprecated @cumulus/common/aws.s3PutObject function has been removed
  • The deprecated @cumulus/common/bucketsConfigJsonObject function has been
    removed
  • The deprecated @cumulus/common/CloudWatchLogger class has been removed
  • The deprecated @cumulus/common/collection-config-store.CollectionConfigStore
    class has been removed
  • The deprecated @cumulus/common/collection-config-store.constructCollectionId
    function has been removed
  • The deprecated @cumulus/common/concurrency.limit function has been removed
  • The deprecated @cumulus/common/concurrency.mapTolerant function has been
    removed
  • The deprecated @cumulus/common/concurrency.promiseUrl function has been
    removed
  • The deprecated @cumulus/common/concurrency.toPromise function has been
    removed
  • The deprecated @cumulus/common/concurrency.unless function has been removed
  • The deprecated @cumulus/common/config.parseConfig function has been removed
  • The deprecated @cumulus/common/config.resolveResource function has been
    removed
  • The deprecated @cumulus/common/DynamoDb.get function has been removed
  • The deprecated @cumulus/common/DynamoDb.scan function has been removed
  • The deprecated @cumulus/common/FieldPattern class has been removed
  • The deprecated @cumulus/common/launchpad.getLaunchpadToken function has been
    removed
  • The deprecated @cumulus/common/launchpad.validateLaunchpadToken function has
    been removed
  • The deprecated @cumulus/common/LaunchpadToken class has been removed
  • The deprecated @cumulus/common/message.buildCumulusMeta function has been
    removed
  • The deprecated @cumulus/common/message.buildQueueMessageFromTemplate
    function has been removed
  • The deprecated @cumulus/common/message.getCollectionIdFromMessage function
    has been removed
  • The deprecated @cumulus/common/message.getMaximumExecutions function has
    been removed
  • The deprecated @cumulus/common/message.getMessageExecutionArn function has
    been removed
  • The deprecated @cumulus/common/message.getMessageExecutionName function has
    been removed
  • The deprecated @cumulus/common/message.getMessageFromTemplate function has
    been removed
  • The deprecated @cumulus/common/message.getMessageGranules function has been
    removed
  • The deprecated @cumulus/common/message.getMessageStateMachineArn function
    has been removed
  • The deprecated @cumulus/common/message.getQueueName function has been
    removed
  • The deprecated @cumulus/common/message.getQueueNameByUrl function has been
    removed
  • The deprecated @cumulus/common/message.hasQueueAndExecutionLimit function
    has been removed
  • The deprecated @cumulus/common/Semaphore class has been removed
  • The deprecated @cumulus/common/string.globalReplace functon has been removed
  • The deprecated @cumulus/common/string.isNonEmptyString functon has been
    removed
  • The deprecated @cumulus/common/string.isValidHostname functon has been
    removed
  • The deprecated @cumulus/common/string.match functon has been removed
  • The deprecated @cumulus/common/string.matches functon has been removed
  • The deprecated @cumulus/common/string.replace functon has been removed
  • The deprecated @cumulus/common/string.toLower functon has been removed
  • The deprecated @cumulus/common/string.toUpper functon has been removed
  • The deprecated @cumulus/common/testUtils.getLocalstackEndpoint function has been removed
  • The deprecated @cumulus/common/util.setErrorStack function has been removed
  • The @cumulus/common/util.uuid function has been removed
  • The deprecated @cumulus/common/workflows.getWorkflowArn function has been
    removed
  • The deprecated @cumulus/common/workflows.getWorkflowFile function has been
    removed
  • The deprecated @cumulus/common/workflows.getWorkflowList function has been
    removed
  • The deprecated @cumulus/common/workflows.getWorkflowTemplate function has
    been removed
  • @cumulus/aws-client/StepFunctions.toSfnExecutionName()
  • @cumulus/aws-client/StepFunctions.fromSfnExecutionName()
  • @cumulus/aws-client/StepFunctions.getExecutionArn()
  • @cumulus/aws-client/StepFunctions.getExecutionUrl()
  • @cumulus/aws-client/StepFunctions.getStateMachineArn()
  • @cumulus/aws-client/StepFunctions.pullStepFunctionEvent()
  • @cumulus/common/test-utils/throttleOnce()
  • @cumulus/integration-tests/api/distribution.invokeApiDistributionLambda()
  • @cumulus/integration-tests/api/distribution.getDistributionApiRedirect()
  • @cumulus/integration-tests/api/distribution.getDistributionApiFileStream()
Assets 7
Loading