v9.7.0

Notable Changes

• CUMULUS-2583

  • The queue-granules task now updates granule status to queued when a granule is queued. In order to prevent issues with the private API endpoint and Lambda API request and concurrency limits, this functionality runs with limited concurrency, which may increase the task's overall runtime when large numbers of granules are being queued. If you are facing Lambda timeout errors with this task, we recommend converting your queue-granules task to an ECS activity. This concurrency is configurable via the task config's concurrency value.

• CUMULUS-2676

  • The discover-granules task has been updated to limit concurrency on checks to identify and skip already ingested granules in order to prevent issues with the private API endpoint and Lambda API request and concurrency limits. This may increase the task's overall runtime when large numbers of granules are discovered. If you are facing Lambda timeout errors with this task, we recommend converting your discover-granules task to an ECS activity. This concurrency is configurable via the task config's concurrency value.
  • Updated memory of <prefix>-sfEventSqsToDbRecords Lambda to 1024MB

• CUMULUS-2000

  • The Queue Granules task can be configured to batch multiple granules into workflow payloads

All Changes

Added

• CUMULUS-2000

  • Updated @cumulus/queue-granules to respect a new config parameter: preferredQueueBatchSize. Queue-granules will respect this batch size as best as it can to batch granules into workflow payloads. As workflows generally rely on information such as collection and provider expected to be shared across all granules in a workflow, queue-granules will break batches up by collection, as well as provider if there is a provider field on the granule. This may result in batches that are smaller than the preferred size, but never larger ones. The default value is 1, which preserves the current behavior of queueing 1 granule per workflow.

• CUMULUS-2630

  • Adds a new workflow DiscoverGranulesToThrottledQueue that discovers and writes
    granules to a throttled background queue. This allows discovery and ingest
    of larger numbers of granules without running into limits with lambda
    concurrency.

Changed

• CUMULUS-2695
  • Updates the example/cumulus-tf deployment to change
    archive_api_reserved_concurrency from 8 to 5 to use fewer reserved lambda
    functions. If you see throttling errors on the <stack>-apiEndpoints you
    should increase this value.
  • Updates cumulus-tf/cumulus/variables.tf to change
    archive_api_reserved_concurrency from 8 to 15 to prevent throttling on
    the dashboard for default deployments.