SNOW-802952: Handle 504 Gateway Timeout Error by setting socket_timeout

[paultouma24]

By definition, a 504 Gateway Error is a server error response code indicates that the server(Snowflake in this case), while acting as a gateway or proxy, did not get a response in time from the upstream server(still Snowflake) that it needed in order to complete the request.

What is the current behavior?

I am using Snowflake Connector v3.0.0 to make requests to Snowflake.

The Gateway 504 errors that I am getting are occurring due the usage of the connector exactly here. The request has a socket_timeout of 60 seconds, and when the return status code for this is a “retryable” one(aka 504 Gateway), a RetryRequest error is raised, as demonstrated in the logs.

This is wrapped from a call from here that handles RetryRequest ‘s right below it here. In the handling of RetryRequest, you can see that other errors can be logged (& I haven’t seen those in the logs), so this means that anytime I've seen a RetryRequest error, the return value has been strictly None to force a retry ( & not {} because I didn't see any errors that a return value of {} would be accompanied with).

With a guaranteed return value of None, this While True would just keep retrying until a valid object is returned. Therefore, since I don’t see any other errors, I can assume that the subsequent retry’s that occurred, actually ended up succeeding. Otherwise, the retries would continue until there was another error (which I haven’t seen).

What is the desired behavior?

The setting for socket_timeout is sticky and hard to increase past the default value. Ideally, setting this for instances with large requests could help reduce number of 504's.

How would this improve snowflake-connector-python?

This would allow the connector to be more configurable based on the end user's preferences for handling timeout errors when processing requests to Snowflake.

[sfc-gh-aling]

hey @paultouma24 thanks for the feedbacks.

Just want to make sure my understanding is correct, are you looking for timeout being configurable in a clear/obvious way for early failure?

also I think you can try passing socket_timeout to the connector public function/method you call -- it will be passed as part of kwargs and finally reach to the _request_exec call

[paultouma24]

hi @sfc-gh-aling ,

thanks for taking a look.

Yes, we looked into setting socket_timeout, but it looks like we cannot actually set it greater than the default value: https://github.com/snowflakedb/snowflake-connector-python/blob/v3.0.0/src/snowflake/connector/network.py#L1011-L1015

Is there any other way to set this value, that would allow it to be greater than DEFAULT_SOCKET_CONNECT_TIMEOUT?

Thanks

[rvalek]

Thanks @paultouma24 for the detailed analysis! I'm experiencing a similar issue and I second the question.

Thanks!

[NNP337]

I am facing similar issue, Did you find any fix for this ?
I found this on Snowflake documentation, https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-connect#managing-connection-timeouts where they maintains socket timeout to 300s. have you tried this ?

[paultouma24]

I am facing similar issue, Did you find any fix for this ? I found this on Snowflake documentation, https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-connect#managing-connection-timeouts where they maintains socket timeout to 300s. have you tried this ?

It looks like this was addressed in a later version: #1759

if you update to the latest version, passing the socket_timeout to the connector kwargs should now be respected