Add Google Batch NOT_FOUND error management

[jorgee]

close #5422
close #5727

This PR includes a possible fix for the NotFoundException returned by the Google Batch API when getting some tasks.

When the client.getTaskStatus throws a NotFoundException it is caught and managed in the following way:

• If the retrieved from client.listTasks has several tasks. It tries to find the task and check the status
• otherwise or if the task is not found in the list it retrieves the job status.
• If it is not able to neither get the task or job status it returns pending

Including a unit test producing the NotFoundException to validate the logic.

Two corner cases could not be correctly managed.

1. When neither task status nor a job status is found, it is returning PENDING. I think that it could happen in the initial stage when Google Batch is creating the job and tasks. I am assuming that a task or job status will be received at some point in the execution. It is the same management that we did when no tasks were found in the job.
2. When a task is not found in Google Batch, it belongs to a task array job and the job status is RUNNING or FAILED, we could get an incorrect task status. It is not very important in the case of RUNNING, but with FAILED we could get a task failed but, its state is unknown. According to the API documentation, the RUNNING or FAILED job states mean that at least one of the tasks is in this state. Job could be FAILED when there has been a job failure (such as the invalid type one we saw in Google Batch run hangs when a job fail to start #5550). So, I am assuming that when the task status is not found, it is not in the list and the Job is FAILED, It is due to a job failure and the task is also FAILED. Other alternatives that I have considered are: set the task to PENDING, but I think we could get a deadlock if it is job failure; or throw an exception to abort the execution. Other suggestions are welcome.

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	6eff10a
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67ac78b8e2347d000837e74a

[pditommaso]

Should @ejseqera run stress test on this?

[ejseqera]

I'm on it

[pditommaso]

@ejseqera any update on this? let us know if you need assistance for this branch

[ejseqera]

I've attempted a large nf-core/sarek stress test run (~1390 jobs) and ran into several igenomes download issues and more importantly, several java.lang.NullPointerException: Cannot invoke "com.google.common.hash.HashCode.asBytes()" because "hash" is null in the TaskProcessor suggesting there might be an underlying issue with task submission/retry logic. See attached log.

Will update once I have results from the rerun with local reference data but this seems unrelated to the reference data staging errors.

[jorgee]

@ejseqera It as an unrelated issue, the stage of the file is failing and in the retry the hash is null. I am looking why the hash is null in this case.

[bentsherman]

@jorgee the old implementation of checking job status: #3892

[jorgee]

I have updated the branch with the following changes
Add a flag to indicate if a task belongs to an array, when a task belongs to an array I use the current way to check the status but with a fallback to the job status if it fails. For single tasks, it uses the old job status check.
In PR #5723, I have included an alternative for managing the status of tasks in arrays that reduces the API calls and does not require the specific management of the NotFoundException.

[ejseqera]

@jorgee Should I go ahead and run another test with this updated branch now? Is it worth also testing #5723?

[jorgee]

@jorgee Should I go ahead and run another test with this updated branch now? Is it worth also testing #5723?

Yes, please check with this updated branch. I have also included a change to fix the hash null due to the read timeout. #5723 only affects when using task arrays.

[jorgee]

This might be addressable using existing Nextflow configuration settings google.httpReadTimeout and google.httpConnectTimeout so I can retry but let me know if you have any further insights. Log is attached.
KwJ0dUQ6dCkN7.log

@ejseqera It was a temporal domain name resolution problem "UnknownHostException: storage.googleapis.com". The read timeout or connect timeout are not going to fix it. Maybe increase the retry config values google.storage.maxDelay and google.storage.maxAttempts.

[ejseqera]

I ran another large-scale test (57gB7BLSjrLuXj, ~1200 successful tasks) and it looks like the original NotFoundException issue appears to be resolved with this implementation.

During testing, I did encounter some other exceptions, but these are separate from the NotFoundException issue this PR addresses:

1. DeadlineExceededException from the Batch API after ~1200 successful tasks

• The error occurred when the gRPC call to batch.googleapis.com exceeded its 60-second deadline. This happened despite having configured a retry policy:

      retryPolicy {
         maxDelay = '60s'
         maxAttempts = 15
         jitter = 0.5
         delay = '5s'
      }

• This can be mitigated with alternative retry policy configuration
• Not related to the original task status resolution problem

3. RejectedExecutionException during pipeline shutdown
   • Related to thread pool management during termination
   • Again, separate from the task status handling

These new findings could be addressed separately through configuration or future PRs if needed, but I don't think they impact the effectiveness of this PR's solution for the NotFoundException issue. What do you think? @pditommaso @jorgee

Logs for two separate runs in different regions attached.
57gB7BLSjrLuXj.log
20EJcGulEDB2GF.log

[pditommaso]

@ejseqera it looks like the RejectedExecutionException error is always a side effect DeadlineExceededException. I'll open a separate issue for the first