In this post, we will see how to resolve Polly re-try policy not working in conjunction with circuit breaker with Ocelot
Question:I wanted to use
Pollyre-try and circuit breaker with
Ocelotapi gateway. I am trying to wrap policies with
circuit breakerworks, but
Below code just throw the exception, but NO re-try happening. When I am calling the API 3 times, circuit opens.
Best Answer:Even though the problem has been solved by removing Ocelot, let me share my thoughts about your policies.
The escalation policy
The way how you chain the policies to each other defines an escalation order.
So, first let’s see how does your policy chain looks like.
PollyQoSProviderhelper class defines two policies in the following order:
- A Circuit Breaker which triggers for
- And a Timeout
These policies do not return any value.
You have defined two other policies:
- A Retry which triggers for
HttpRequestExceptionor when the status code is either 404 or 408 or 5XX
- A Circuit Breaker which triggers when the status code is 500
These policies do return with an
You have chained them in the following order (from the most outer to the most inner):
Circuit Breaker which triggers for HttpRequestException, TimeoutRejectedException and TimeoutException Timeout Retry which triggers for HttpRequestException or when the status code is either 404 or 408 or 5XX Circuit Breaker which triggers when the status code is 500
This may or may not be your desired resiliency strategy. I would advice you to reassess whether this is what you really want/need.
Chaining the policies
The policies of the
PollyQoSProviderare defined for async methods (
Task) whereas yours are defined for async functions (
Task<HttpResponseMessage>). The static
WrapAsyncdoes not allow to combine these two types of policies. On the hand the instance level
WrapAsyncdoes. (For more information about this constraints please read this SO topic.)
Because you have used the combination of the two that’s why the chained policy is an
IAsyncPolicy<HttpResponseMessage>. Even though it’s working, I usually suggest to use only the static
WrapAsyncto chain policies due to its compile-time compatibility guarantees.
Multi level circuit breakers
I’m using Polly for awhile and I haven’t encountered any use case where multiple (nested) circuit breakers would be really required. Most of time you can (and should) solve it with a single CB which can trigger for multiple different conditions:
Circuit Breaker which triggers when the status code is 500 or for the following exceptions: HttpRequestException, TimeoutRejectedException and TimeoutException
Circuit breaker and its shared state
The CB was designed in a way that it can be shared between multiple components. If you have already detected that the downstream is temporarily inaccessible then use this information everywhere rather than issue new requests and come to the same conclusion.
So, defining a CB inside a
DelegatingHandleris against this. Each and every
DelegatingHandlerwill have its own CB so, they do not share state via the
ICircuitController. Aim for reusing CB policy.
Timeout can work in optimistic or in pessimistic mode. Even though your code looks like at first glance it uses optimistic, unfortunately it does not.
PollyQoSProvider‘s CB was defined in a way that it can break for optimistic timeout (
TimeoutRejectedException) and pessimistic timeout (
TimeoutException) as well.
If you have better answer, please add a comment about this, thank you!