Improve Harmony Job Status Check Logic

[frankinspace]

During ASF testing of bignbit with OPERA RTC products, some harmony requests were taking a very long time causing a large number of lambdas to run every 20 seconds to query harmony. A few suggestions for improvements were:

• Make delay between status checks configurable.
  • This could be set as a parameter to the terraform module (https://github.com/podaac/bignbit/blob/develop/terraform/variables.tf)
  • Then substituted into the step function definition template https://github.com/podaac/bignbit/blob/develop/terraform/outputs.tf#L82-L101 and https://github.com/podaac/bignbit/blob/develop/terraform/state_machine_definition.tpl#L276-L280
• Introduce a max # of retries configuration option as well
  • Could be implemented via refactoring get_harmony_job_status to return an error when the job is incomplete, then implementing a step function retry strategy including max attempts and a retry backoff

[asjohnston-asf]

Including a backoff factor to the wait time would be lovely as well.

[frankinspace]

@asjohnston-asf I like your idea of refactoring get_harmony_job_status. The retry policy for that step in the workflow could be tuned by setting them via tf module parameter and would cover everything we want I think https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html