Introduction

If you find your code having to execute itself every time it encounters errors in calling an operation, then it is time to reconsider limiting your execution to a specific number of times and have a backup plan once all attempts fail or else you could end up with an infinite loop waiting for your call to be resolved.

Introducing the Retry pattern. This design pattern is meant to handle repetitive calls automatically and can be implemented variously. In this article, we will see how the retry pattern works by using the Polly library as per Microsoft's recommendation .

Prerequisite

Intermediate or advanced knowledge in C# and dotnet core
The project created in this article is using dotnet 6
Visual Studio 2022

The Retry Pattern

The retry pattern is a design pattern that handles failures of calls or operations in general by multiple retry strategies. There are many articles out there that describe the implementation in C#, but we don't actually need to write it manually!

Introducing Polly

Polly by App-vNext is a library designed for handling resiliency and is mature enough that Microsoft recommends it as a library to achieve resilient code on its learning page. Polly works by wrapping code in policies and handling it based on the policy assigned to it; Retry, Circuit-Breaker, Cancel, etc..

Solution Outline

Let's create an ASP.NET Core Web API project straight from Visual Studio. This will give the standard WeatherForecast app a ready-built controller and Model.

Modify The Project

Let's begin by adding a service and a repository, then move code around the controller to the repository.
The outline should look something like this

WeatherForecastRepository.cs

namespace WeatherForecast.Repository
{
    public interface IWeatherForecastRepository
    {
        Task<IEnumerable<WeatherForecast>> GetForecastAsync();
    }
    public class WeatherForecastRepository : IWeatherForecastRepository
    {
        private static readonly string[] Summaries = new[]
        {
        "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
    };
        public async Task<IEnumerable<WeatherForecast>> GetForecastAsync()
        {
            return Enumerable.Range(1, 5).Select(index => new WeatherForecast
            {
                Date = DateTime.Now.AddDays(index),
                TemperatureC = Random.Shared.Next(-20, 55),
                Summary = Summaries[Random.Shared.Next(Summaries.Length)]
            })
            .ToArray();
        }
    }
}

WeatherForecastService.cs

using WeatherForecast.Repository;

namespace WeatherForecast.Services
{
    public interface IWeatherForecastService
    {
        Task<IEnumerable<WeatherForecast>> GetForecastAsync();
    }
    public class WeatherForecastService : IWeatherForecastService
    {
        private readonly IWeatherForecastRepository _repository;
        public WeatherForecastService(IWeatherForecastRepository repository) 
        {
            _repository = repository;
        }    

        public async Task<IEnumerable<WeatherForecast>> GetForecastAsync()
        {
            return await _repository.GetForecastAsync();
        }
    }
}

WeatherForecastController.cs

using Microsoft.AspNetCore.Mvc;
using WeatherForecast.Services;

namespace WeatherForecast.Controllers
{
    [ApiController]
    [Route("[controller]")]
    public class WeatherForecastController : ControllerBase
    {
        private readonly ILogger<WeatherForecastController> _logger;
        private readonly IWeatherForecastService _weatherForecastService;

        public WeatherForecastController(ILogger<WeatherForecastController> logger, IWeatherForecastService weatherForecastService)
        {
            _logger = logger;
            _weatherForecastService= weatherForecastService;
        }

        [HttpGet(Name = "GetWeatherForecast")]
        public async Task<IEnumerable<WeatherForecast>> Get()
        {
            return await _weatherForecastService.GetForecastAsync();
        }
    }
}

Program.cs

// Add services to the container.
builder.Services.AddTransient<IWeatherForecastService, WeatherForecastService>();
builder.Services.AddTransient<IWeatherForecastRepository, WeatherForecastRepository>();

In the code snippets above, we just move codes from the controller to the repository and then add the services to the builder to be injectable by the .net core DI

Implement retry in the service

Add Polly Package

Let's start by adding the Polly package from Nuget Package Manager or via the Nuget console.

Add Retry and Fallback Policy

Now we implement the policy in our GetForecastAsync() in WeatherForecastService.cs

        public async Task<IEnumerable<WeatherForecast>> GetForecastAsync()
        {
            var retryPolicy = Policy<IEnumerable<WeatherForecast>>
                .Handle<WebException>()
                .Or<Exception>()
                .RetryAsync<IEnumerable<WeatherForecast>>(5, onRetry: (exception, retryCount, context) =>
                {
                    _logger.LogInformation($"Retry #{retryCount} : Reason: {exception}");
                });

            var fallbackPolicy = Policy<IEnumerable<WeatherForecast>>
                .Handle<WebException>()
                .Or<Exception>()
                .FallbackAsync((action) =>
                {
                    _logger.LogInformation("All Retries Failed");
                    return null;
                });

            return await fallbackPolicy.WrapAsync(retryPolicy)
                .ExecuteAsync(() => _repository.GetForecastAsync());
        }
    }

from the code above we create a retry policy with a retry count of 5 then a fallback policy that returns null if all retry attempt failed before finally wrapping them all together.

Run the API and you should still see the result

But we still don't get to see this pattern in action..

How do we test this?

We have the code in place, great. But how do we see this in action? Let's create a unit test to see the log being printed out and control our success & failure rate.

Setup a xUnit Test Project

Let's create a xUnit test project to see this pattern in action. We will be mocking the GetForecastAsync() method from the repository, so add the Moq NuGet package as well. Add a reference to the main project, the initial configuration should look like this.

using Microsoft.Extensions.DependencyInjection;
using WeatherForecast.Repository;
using WeatherForecast;
using Polly;
using Xunit.Abstractions;
using System.Net;
using Moq;

namespace WeatherForecastTest
{
    public class UnitTest1
    {
        private readonly IServiceProvider _serviceProvider;
        private readonly ITestOutputHelper _output;
        public IServiceProvider ServiceProvider => _serviceProvider;
        public ITestOutputHelper Output => _output;

        public UnitTest1(ITestOutputHelper output)
        {
            var services = new ServiceCollection();
            services.AddLogging();
            services.AddTransient<IWeatherForecastRepository,WeatherForecastRepository>();
            _serviceProvider = services.BuildServiceProvider();
            _output = output;
        }
    }
}

Test #1: 5 attempts, 4 fails, the last one succeeds

Create a Fact or test method to test the above scenario.

[Fact]
        public async void WeatherForecastService_ExecuteFiveRetryLastAttemptSucceed_PolicyResultEqualsExpectedData()
        {
            //arrange
            var serviceMock = new Mock<IWeatherForecastRepository>();

            //mock returning result
            WeatherForecastModel forecast = new()
            {
                TemperatureC = 30,
                Summary = "test",
                Date = DateTime.Now
            };

            var data = Enumerable.Range(1,1).Select(index => new WeatherForecastModel
            {
                Date = DateTime.Now,
                TemperatureC = 30,
                Summary = "Summer"
            }).ToArray();

            //mock repetitive calls with 5 outcomes
            serviceMock.SetupSequence(p => p.GetForecastAsync())
                .Throws(new Exception())
                .Throws(new Exception())
                .Throws(new Exception())
                .Throws(new Exception())
                .ReturnsAsync(data);

            //create retry & fallback policy and wrap the execution
            var retryPolicy = Policy<IEnumerable<WeatherForecastModel>>
                .Handle<WebException>()
                .Or<Exception>()
                .RetryAsync<IEnumerable<WeatherForecastModel>>(5, onRetry: (exception, retryCount, context) =>
                {
                    Output.WriteLine($"Retry #{retryCount} : Reason: {exception}");
                });

            var fallbackPolicy = Policy<IEnumerable<WeatherForecastModel>>
                .Handle<WebException>()
                .Or<Exception>()
                .FallbackAsync((action) =>
                {
                    Output.WriteLine("All Retries Failed");
                    return null;
                });

            //act
            var result = await fallbackPolicy.WrapAsync(retryPolicy)
                .ExecuteAsync(() => serviceMock.Object.GetForecastAsync());

            if(result != null)
            {
                foreach (var item in result)
                {
                    Output.WriteLine($"Result: {item.TemperatureC},{item.TemperatureF}, {item.Summary}");
                }
            }

            //assert
            Assert.Equal(data, result);
        }

In the code above, first, we mock the repository service to return a customized Enumerable<WeatherForecastModel> object. We will mimic the failed calls with SetupSequence and as you can see, it will purposely fail the call 4 times before letting it succeed the 5th time. Run the test result, and we should see the following output

Test #2: 5 attempts, all failed

In this second test all retry attempts failed and it will trigger fallback action in the fallback policy

[Fact]
        public async void WeatherForecastService_ExecuteFiveRetryAllAttemptFails_PolicyResultEqualsNull()
        {
            //arrange
            var serviceMock = new Mock<IWeatherForecastRepository>();

            //mock returning result
            WeatherForecastModel forecast = new()
            {
                TemperatureC = 30,
                Summary = "test",
                Date = DateTime.Now
            };

            var data = Enumerable.Range(1, 1).Select(index => new WeatherForecastModel
            {
                Date = DateTime.Now,
                TemperatureC = 30,
                Summary = "Summer"
            }).ToArray();

            //mock repetitive calls with 5 outcomes
            serviceMock.SetupSequence(p => p.GetForecastAsync())
                .Throws(new Exception())
                .Throws(new Exception())
                .Throws(new Exception())
                .Throws(new Exception())
                .Throws(new Exception());

            //create retry & fallback policy and wrap the execution
            var retryPolicy = Policy<IEnumerable<WeatherForecastModel>>
                .Handle<WebException>()
                .Or<Exception>()
                .RetryAsync<IEnumerable<WeatherForecastModel>>(5, onRetry: (exception, retryCount, context) =>
                {
                    Output.WriteLine($"Retry #{retryCount} : Reason: {exception}");
                });

            var fallbackPolicy = Policy<IEnumerable<WeatherForecastModel>>
                .Handle<WebException>()
                .Or<Exception>()
                .FallbackAsync((action) =>
                {
                    Output.WriteLine("All Retries Failed : Fallback Policy Triggered");
                    return Task.FromResult(null as IEnumerable<WeatherForecastModel>);
                });

            //act
            var result = await fallbackPolicy.WrapAsync(retryPolicy)
                .ExecuteAsync(() => serviceMock.Object.GetForecastAsync());

            if (result != null)
            {
                foreach (var item in result)
                {
                    Output.WriteLine($"Result: {item.TemperatureC},{item.TemperatureF}, {item.Summary}");
                }
            }

            //assert
            Assert.Null(result);
        }

as you can see, after the 5th try, it triggers the output logging in the fallback policy and we instruct the policy to return null which is asserted as true.

Take-Home Message

We have seen the simplest implementation of the retry and fallback pattern. This should open the path to more complicated scenarios with advanced implementation with Polly.

Keep in mind that services especially external APIs will have downtime at some point, and don't repeat the same mistake my team did, keep calling the method once it jumps into the Catch block!

Plan your resiliency, know when to give up, and keep exploring!

Retry and Fallback Pattern in .Net 6 with Polly

Table of contents