Rotate auth token in backoff_handler

[laurentS]

This PR fixes #129 to handle the edge case described in it.

The limit to this is that the token will be rotated on every RetriableAPIError with this code, which may not be desirable.
I don't see any obvious way to prevent this, short of throwing a different exception upon rate limit errors, which would require modifying request_decorator. Happy to hear any other ideas :)

[ericboucher]

I am SHOCKED that the backoff library doesn't return the response or even just its status code...

Anyways, investigating this I found out that we were missing a key component, actually updating to the SDK that has this new featured 😉

[ericboucher]

@laurentS looks like upgrading the SDK broke tests around collaborators, we probably need to change our implementation of alternative_sync_chidren

[ericboucher]

I opened - litl/backoff#158

Noting that the SDK should upgrade to backoff 2.0.1 to get better typing @edgarrmondragon ;)

[edgarrmondragon]

@ericboucher

Noting that the SDK should upgrade to backoff 2.0.1 to get better typing @edgarrmondragon ;)

Yup, we're blocked by transferwise/pipelinewise-singer-python#61

I opened - litl/backoff#158

What you want is already possible in backoff 2.0 with the (undocumented) backoff.runtime: https://github.com/litl/backoff/blob/92e7b6e910accee30118a6f3d2cef4c62caa38f5/tests/test_integration.py#L32-L37.

We ended up porting it to the SDK (because of the blocker above) as RESTStream.backoff_runtime and there's an example here: https://sdk.meltano.com/en/latest/code_samples.html#custom-backoff that accesses the response object via the RetriableAPIError instance 🙂

Wdyt?

[laurentS]

I found a way to rotate the token only on relevant errors. It's probably not the most reliable solution, but should be fine as a temp fix until the PRs mentioned above are merged.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
0.0% Duplication