HARMONY-2008: Fix query-cmr granule size parsing logic #699
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…eAndDistributionInformation size fields to get file size information for a granule.
indiejames
reviewed
Feb 14, 2025
| granuleSizeInBytes += info.Size * 1024 * 1024 * 1024 * 1024 * 1024; | ||
| break; | ||
| default: | ||
| logger.warn(`Unknown SizeUnit: ${info.SizeUnit}`); |
There was a problem hiding this comment.
Shouldn't this be an error?
There was a problem hiding this comment.
We treat 0 as being unable to retrieve the size and handle it elsewhere in the work item updater code. I was just going to leave the behavior the same since we don't want to fail things when we cannot figure out the size.
ygliuvt
approved these changes
Feb 14, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Jira Issue ID
HARMONY-2008
Description
Some providers use Size and SizeUnits in their CMR UMM-G metadata and others user SizeInBytes to specify the size of the granule files. This changes it to be able to correctly pull the size from either field. The UMM schema provides guidance that all providers should use SizeInBytes going forward, so if both are provided we use the SizeInBytes field.
Local Test Steps
npm run buildinservices/query-cmrbin/restart-servicesI performed performance testing in my sandbox - a 110K granule request took 10 minutes to complete all of the query-cmr tasks and populate all of the work items.
I then redeployed my sandbox pointing to production metadata and issued the original request for this ticket. Each query-cmr work item took between 11 and 28 seconds to populate 2000 work items. Previously each query-cmr work item was taking at least 3 minutes and as long as 30 minutes.
PR Acceptance Checklist