When polly is used at rest client level and an (locally running) API is called which throws an error, the API is not retried

[nrpatil1993]

Description
I've created an API running locally which throws/returns a NotFound exception. A NotFound exception code is also added in the policy when creating the Policy Handler. The API is expected to be retried, but the retry never happens.

To Reproduce

   var testClient = GetRestClient();

    try

    {

        await testClient.GetTestData("test");

    }

    catch(ApiException ex)

    {

        Console.WriteLine(ex.StatusCode);

    }
//////////////////////////// RestClient/////////////////////////
private ITeamplayEntitlementApi GetRestClient()
{

    var policyHandler = new PolicyHttpMessageHandler(RetryPolicyOnException());

    DelegatingHandler delegatingHandler = policyHandler;

    if (delegatingHandler != null && delegatingHandler.InnerHandler == null)
    {
        delegatingHandler.InnerHandler = new HttpClientHandler();
    }

    HttpClient httpclient = new HttpClient(delegatingHandler)
    {
        BaseAddress = new Uri("https://localhost:5001/weatherforecast")
    };
    return RestClient.For<ITestApi>(httpclient);
}

private IAsyncPolicy<HttpResponseMessage> RetryPolicyOnException()
{
    TimeSpan timeOut = TimeSpan.FromMilliseconds(50000);
    void OnRetry(Exception ex, TimeSpan t, int retryAttempt, Context context)
    {
        if (retryAttempt <= 2)
        {
            Console.WriteLine($"Call failed with exception : {ex.Message}. The call will be retried.");
        }

        Console.WriteLine($"Call failed. Retrying the call. Retry-Attempt:{retryAttempt}.");
    };
    return new DefaultPolicyFactory()
                .BuildPolicyBuilder(new List<int>())
                .WaitAndRetryAsync(new List<TimeSpan> { TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(1) , TimeSpan.FromSeconds(1) }, OnRetry)
                .AsAsyncPolicy<HttpResponseMessage>();
}

public class DefaultPolicyFactory : IPolicyBuilderFactory

{

    public PolicyBuilder BuildPolicyBuilder(List<int> errorCodesToExclude)
    {

        return Policy.Handle<ApiException>(ex => ex.StatusCode == HttpStatusCode.NotFound && !IsExcluded(errorCodesToExclude, HttpStatusCode.NotFound))

                    .Or<ApiException>(ex => ex.StatusCode == HttpStatusCode.RequestTimeout && !IsExcluded(errorCodesToExclude, HttpStatusCode.RequestTimeout))

                    .Or<ApiException>(ex => (int) ex.StatusCode == 423 && !IsExcluded(errorCodesToExclude, 423))

                    .Or<ApiException>(ex => (int) ex.StatusCode == 425 && !IsExcluded(errorCodesToExclude, 425))

                    .Or<ApiException>(ex => (int) ex.StatusCode == 429 && !IsExcluded(errorCodesToExclude, 429))

                    .Or<ApiException>(ex => ex.StatusCode == HttpStatusCode.InternalServerError && !IsExcluded(errorCodesToExclude, HttpStatusCode.InternalServerError))

                    .Or<ApiException>(ex => ex.StatusCode == HttpStatusCode.BadGateway)

                    .Or<ApiException>(ex => ex.StatusCode == HttpStatusCode.ServiceUnavailable)

                    .Or<HttpRequestException>()

                    .Or<ApiException>(ex => ex.StatusCode == HttpStatusCode.GatewayTimeout)

                    .Or<ApiException>(ex => ex.StatusCode == HttpStatusCode.TooManyRequests);

    }

    private static bool IsExcluded(List<int> errorCodesToExclude, HttpStatusCode statusCode)
    {
        return errorCodesToExclude.Contains((int) statusCode);
    }

    private static bool IsExcluded(List<int> errorCodesToExclude, int statusCode)
    {
        return errorCodesToExclude.Contains(statusCode);
    }
}

///////////////////////// API ///////////////////////////////////
  [HttpGet("test")]
    public IActionResult Get()
    {
        Console.WriteLine("API Called"); // Gets called only once
        return NotFound();
    }

Version Info

• RestEase version: 1.5.0
• Target framework version: net6.0

Additional Info
One surprising thing is when I don't start the API, and just run the code to call API with retry, the call fails with error "No connection could be made because the target machine actively refused it" and it gets retried.

[canton7]

The layers here are RestEase --> Polly --> HttpClientHandler --> Internet. RestEase is the thing which throws an ApiException, but Polly never sees that this happens after Polly has returned a response to RestEase.

Write your policy based on what the HttpClientHandler returns, i.e. based on the HttpResponseMessage.

[nrpatil1993]

Do you mean something like this
Policy.Handle<HttpResponseMessage>(ex=>ex.StatusCode == HttpStatusCode.NotFound) ?
Policy.Handle accepts only an Exception. We cannot add a HttpResponseMessage there.
Or did I misinterpreted you?

[canton7]

I'm not particularly familiar with Polly, but you want to be writing policies which work with HttpClient, ignoring RestEase, for example the retry policy here (msg is a HttpResponseMessage). HttpClient only raises exceptions if a connection can't be made: it will return a HttpResponseMessage with a suitable StatusCode if the remote side returns a response.

RestEase is a wrapping the HttpClient+Polly combination. Write your Polly policies as if RestEase isn't involved (there is plenty of documentation on that, because it's a common case), then use RestEase as a wrapper around that.

[nrpatil1993]

Ok, you're right, understood the issue. Polly doesn't know what APIException is. It works correctly after I used HttpResponseMessage. Thanks for the help 🙂

[canton7]

Awesome, glad it's working!