Understanding, Creating, Using and Testing the HttpClient

Introduction

The HttpClient class is often used but also often not fully understood. It's behaviour can be influenced with DelegationHandler implementations, instances can be used via dependency injection and how it works can be tested with integration tests. This article describes how these things work.

Background

This article is intended for .NET Core developers who used the HttpClient at least once and want to know more about it.

Using the Code

First, we want to setup the creation and dependency injection of the HttpClient. In an ASP.NET Core application, this is typically done in the ConfigureServices method. An HttpClient instance has to be injected into a SearchEngineService instance via dependency injection. Two handlers manage the behaviour of the HttpClient: LogHandler and RetryHandler . This is how the ConfigureServices implementation looks like:

public void ConfigureServices(IServiceCollection services)
{
   services.AddControllers();
   services.AddTransient<LogHandler>();
   services.AddTransient<RetryHandler>();
   var googleLocation = Configuration["Google"];
   services.AddHttpClient<ISearchEngineService, SearchEngineService>(c =>
   {
       c.BaseAddress = new Uri(googleLocation);
   }).AddHttpMessageHandler<LogHandler>()
       .AddHttpMessageHandler<RetryHandler>();
}

As becomes clear from the code above, the LogHandler is set before the RetryHandler. The LogHandler is the first handler so this handles what needs to happen directly at the moment the HttpClient is called. Here is the implementation of the LogHandler:

public class LogHandler : DelegatingHandler
{
    private readonly ILogger<LogHandler> _logger;

    public LogHandler(ILogger<LogHandler> logger)
    {
        _logger = logger;
    }

    protected override async Task<HttpResponseMessage> 
      SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var response = await base.SendAsync(request, cancellationToken);
        _logger.LogInformation("{response}", response);
        return response;
    }
}

As becomes clear from the code above, this handler implementation just logs responses from a web request after the base method is called. What this base method triggers, is set by the second handler: the RetryHandler. This handler does a retry in case of an accidental server error. If it succeeds directly or gives a server error more than 3 times, the last result counts and is returned.

public class RetryHandler : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> 
      SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        HttpResponseMessage result = null;
        for (int i = 0; i < 3; i++)
        {
            result = await base.SendAsync(request, cancellationToken);
            if (result.StatusCode >= HttpStatusCode.InternalServerError)
            {
                continue;
            }
            return result;
        }
        return result;
    }
}

As described before, the HttpClient that is managed by these handlers, needs to be injected into a SearchEngineService instance. This class has just one method. The method calls the HttpClient instance and returns length of the content as a response.

public class SearchEngineService : ISearchEngineService
{
    private readonly HttpClient _httpClient;

    public SearchEngineService(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    public async Task<int> GetNumberOfCharactersFromSearchQuery(string toSearchFor)
    {
        var result = await _httpClient.GetAsync($"/search?q={toSearchFor}");
        var content = await result.Content.ReadAsStringAsync();
        return content.Length;
    }
}

The SearchEngineService is a dependency of the controller class, This controller class has one get method that returns the result of a method call as an ActionResult. Here is the controller class.

[Route("api/[controller]")]
[ApiController]
public class SearchEngineController : ControllerBase
{
    private readonly ISearchEngineService _searchEngineService;

    public SearchEngineController(ISearchEngineService searchEngineService)
    {
        _searchEngineService = searchEngineService;
    }

    [HttpGet("{queryEntry}", Name = "GetNumberOfCharacters")]
    public async Task<ActionResult<int>> GetNumberOfCharacters(string queryEntry)
    {
        var numberOfCharacters = 
            await _searchEngineService.GetNumberOfCharactersFromSearchQuery(queryEntry);
        return Ok(numberOfCharacters);
    }
}

To write an integration test for this controller, we use IntegrationFixture (NuGet package here, documentation here, article with some similar code here). The external dependency is being replaced by a mock server that returns an internal server error after the first request and succeeds after the second request. A call to our controller method is done. This triggers a call to the SearchEngineService which calls the HttpClient. As explained before, such a call triggers a call to the LogHandler which afterwards triggers a call to the RetryHandler. Since the first call gives a server error, a retry is done. The RetryHandler does not trigger the LogHandler (it is the other way around). Therefore, our application just logs one response while there are actually two responses (one failing and one succeeding). Here is the code of our integration tests:

[Fact]
public async Task TestDelegate()
{
    // arrange
    await using (var fixture = new Fixture<Startup>())
    {
        using (var searchEngineServer = fixture.FreezeServer("Google"))
        {
            SetupUnStableServer(searchEngineServer, "Response");
            var controller = fixture.Create<SearchEngineController>();

            // act
            var response = await controller.GetNumberOfCharacters("Hoi");

            // assert, external
            var externalResponseMessages = 
            searchEngineServer.LogEntries.Select(l => l.ResponseMessage).ToList();
            Assert.Equal(2, externalResponseMessages.Count);
            Assert.Equal((int)HttpStatusCode.InternalServerError, 
                        externalResponseMessages.First().StatusCode);
            Assert.Equal((int)HttpStatusCode.OK, externalResponseMessages.Last().StatusCode);

            // assert, internal
            var loggedResponse = 
               fixture.LogSource.GetLoggedObjects<HttpResponseMessage>().ToList();
            Assert.Single(loggedResponse);
            var externalResponseContent = 
               await loggedResponse.Single().Value.Content.ReadAsStringAsync();
            Assert.Equal("Response", externalResponseContent);
            Assert.Equal(HttpStatusCode.OK, loggedResponse.Single().Value.StatusCode);
            Assert.Equal(8, ((OkObjectResult)response.Result).Value);
        }
    }
}

private void SetupUnStableServer(FluentMockServer fluentMockServer, string response)
{
    fluentMockServer.Given(Request.Create().UsingGet())
        .InScenario("UnstableServer")
        .WillSetStateTo("FIRSTCALLDONE")
        .RespondWith(Response.Create().WithBody(response, encoding: Encoding.UTF8)
            .WithStatusCode(HttpStatusCode.InternalServerError));

    fluentMockServer.Given(Request.Create().UsingGet())
        .InScenario("UnstableServer")
        .WhenStateIs("FIRSTCALLDONE")
        .RespondWith(Response.Create().WithBody(response, encoding: Encoding.UTF8)
            .WithStatusCode(HttpStatusCode.OK));
}

If you look at the code shown above, you see two assert sections. In the first assert section, we verify the logs of our external (mocked) server. Since the first web request was failing. we expect a second web request (with a second response) to be executed so there should be two responses, which is exactly what we verify.

In the second assert section, we verify the logs of our application itself. As explained, only a single response is expected to be logged so that is what we verify in this second assert section.

If you want to familiarize furthermore, I recommend downloading the source code on GitHub shown in this article. You can, for example, change the order of the handlers or add a new handler and see what happens. By testing with IntegrationFixture, you can easily verify the logs of both our own application and the external (mocked) server.

Points of Interest

While writing this article and the example code, I gained a better understanding of how the HttpClient really works. By using handlers, you will be enabled to do so much more than just a web request. You can build logging, a retry mechanism or anything in addition that you want when doing a web request.

History

• 31^st May, 2020: Initial version