Nextflow to gracefully handle Google gRPC API call failures

[robnewman]

New feature

Can Nextflow support retrying of gRPC API failures, specifically the following different failures:

• UNAVAILABLE
• DEADLINE_EXCEEDED
• RESOURCE_EXHARSTED
• UNKNOWN

Usage scenario

Running Nextflow pipelines on Google Batch, when there are job failures due to the resource not available (e.g. the requested VM was not be able to start due to exceeding the Google project quota), Nextflow was not able to receive any signal from these jobs and this causes the Nextflow job to not stop, consuming resources.

Suggest implementation

Could you make Nextflow support retrying of gRPC API failures based on failure response codes?

[pditommaso]

Duplicate of #4537

[pditommaso]

I believe this is already implemented

nextflow/plugins/nf-google/src/main/nextflow/cloud/google/batch/client/BatchClient.groovy

Lines 181 to 182 in 00376aa

	if( t instanceof UnavailableException )
	return true

[robnewman]

Duplicate of #4537

Thanks @pditommaso. Per @jorgee's comment, it didn't look like his PR resolved all the call failure modes:
#5690 (comment)

[bentsherman]

Closing until we get a more specific (reproducible) bug report.