Ensure token is refreshed on Unauthenticated

[EngHabu]

Overview

When investigating an intermittent auth issue, @katrogan and myself made a few observations that lead us down to find a couple of issues in how propeller (through flyteidl/client) refreshes tokens.

1. AuthInterceptor attempts to forward the request as is (whether it has a token or not), if it fails with Unauthenticated, it attempts to renew it,
2. The way it renews it is by initializing a new TokenSource and attaching that to the CredentialsFuture
3. When that is called, it checks if the InMemoryTokenCache (a singleton) has a valid token and use it, if the cache says it's invalid, only then does it actually attempt to renew the token.

The problems are:

1. If the TokenCache says the token is valid, we don't renew the token even if the control plane responded with Unauthenticated [Wrong]
2. The way we access/write tokens to TokenCache is not thread safe (no atomic or locks around it)

The fixes:

1. Make TokenCache use atomic.Value to ensure the token is propagated across threads
2. Add a Mutex around the code that updates the token in the cache
3. Nil the token in the cache when the server responds with Unauthenticated

We should follow up with metrics to ensure the correct behavior from then on.

Logs from Admin indicating a spike in Token is expired errors: https://unionai.grafana.net/goto/qCJRpJfSg?orgId=1
Logs from nginx showing propeller attempting to get new tokens: https://unionai.grafana.net/goto/KqVkt1BSg?orgId=1

Test Plan

• Test against regressions

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 62.42%. Comparing base (80a8765) to head (99436ee).
Report is 569 commits behind head on master.

❗ Current head 99436ee differs from pull request most recent head c357fc2. Consider uploading reports for the commit c357fc2 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5293      +/-   ##
==========================================
+ Coverage   58.91%   62.42%   +3.51%     
==========================================
  Files         645      263     -382     
  Lines       55351    15853   -39498     
==========================================
- Hits        32609     9897   -22712     
+ Misses      20159     5090   -15069     
+ Partials     2583      866    -1717     

Flag	Coverage Δ
unittests	?
unittests-datacatalog	69.31% <ø> (?)
unittests-flytecopilot	17.79% <ø> (?)
unittests-flyteplugins	61.94% <ø> (?)
unittests-flytestdlib	65.75% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.