High failed rate when running GCP gemini batch inference jobs

[westonli-thu]

Hi, we are using batch inference with gemini-1.5-pro, while we found:

1. when we submit > 1 tasks, at most 1 task can succeed, the other will failed (is this because the quota?)

2. when we run batch inference job (even though we only submit 1 task), it sometimes occurs the following issue & the whole task will fail:

RESOURCE_EXHAUSTED error occurred: {"error": {"code": 429, "message": "Online prediction request quota exceeded for gemini-1.5-pro. Please try again later with backoff.", "status": "RESOURCE_EXHAUSTED"}}

Can we have any methods that:

if the quota exceed, the job can automatically pending & retry

if some of the samples fails to be inferred, keep the results of other samples, do not let the complete job failed.