CI run is slowed down by failing to download fetches

[gregtatum]

I'm working on some performance optimization to bring down our CI times, and I found some tasks are taking 12 minutes to resolve due to issues downloading fetches and artifacts. This compounds when tasks depend upon each other, so a few tasks failing in this way can increase CI runs by 20-30 minutes. After some fixes that I'm working towards merging in now, this will be the slowest part of the CI pipeline.

• translate-mono-src-ru-en-1/2 12m 34s (failing downloads)
• translate-mono-src-ru-en-2/2 2m 22s

And here is a profile of the task:

• Task group profile
• translate-mono-src-ru-en-1/2 profile
• translate-mono-src-ru-en-1/2 live log

Here you can see that it spend 12 minutes attempting to download fetches and artifacts.

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/SVZMV_7KS3mGqDMrMgj9_A/artifacts/public/build/marian.tar.zst
Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>
sleeping for 91.00s (attempt 2/5)
Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>
sleeping for 90.00s (attempt 2/5)
Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>
sleeping for 89.00s (attempt 2/5)
Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>
sleeping for 90.00s (attempt 2/5)
attempt 3/5

[bhearsum]

#549 is related, possibly the same?