In this post, I'll be sharing my journey during the process of an HTTP Server development from scratch. The purpose of this post will be refactor the code, add HTTP Compression to the server. It only supports the gzip compression scheme. Also, I'll add support for persistent HTTP connections and explicit connection closure using the Connection: close header.

Refactoring the code

I replaced the dictionary-based approach for handling endpoints and responses with dynamic response construction in code. Additionally, I used a HashSet to store and validate the supported endpoints more efficiently.

Compression headers

In this stage, I'll add support for the Accept-Encoding and Content-Encoding headers. An HTTP client uses the Accept-Encoding header to specify the compression schemes. The server then chooses one of the compression schemes listed in Accept-Encoding and compresses the response body with it. Then, the server sends a response with the compressed body and a Content-Encoding header. Content-Encoding specifies the compression scheme that was used. If the server doesn't support any of the compression schemes specified by the client, then it will not compress the response body.

Here's a breakdown of the request that specified gzip compression:

GET /echo/foo HTTP/1.1\r\n

// Headers
Host: localhost:4221\r\n
User-Agent: curl/7.64.1\r\n
Accept: */*\r\n
Accept-Encoding: gzip\r\n  // Client specifies it supports the gzip compression scheme.

Here's the response body is compressed with gzip:

// Status line
HTTP/1.1 200 OK\r\n

// Headers (empty)
Content-Encoding: gzip\r\n // Server specifies that the response body is compressed with gzip.
Content-Type: text/plain\r\n  // Original media type of the body.
Content-Length: 23\r\n        // Size of the compressed body.
\r\n      // CRLF that marks the end of the headers

// Compressed body.

To implement this feature on the server, the first step is to look for the Accept-Encoding header in the request. If this is present and includes a valid encoding, the server response must include Content-Encoding header. Otherwise, It should send a standard response and omit the Content-Encoding header. For this project, I'll assume the server only supports the gzip compression scheme.

I started by initialising the encoding header as an empty string. Then, I searched for the Accept-Encoding header in the request. If it isn't found, the encoding header remains an empty string. If the header is present, I validated that the encoding data isn't a null or empty value and contains the : separator. Next, I extracted the value, I checked it in the HashSet of supported encodings (In this case only gzip), and generated the Content-Encoding header when applicable.

// ...
HashSet<string> validEndpoints = new HashSet<string> { "/", "/echo", "/user-agent", "/files" };
HashSet<string> validCompressions = new HashSet<string> { "gzip" };

// ...

else if (urlPath == "/echo"){
      string data = requestTarget.Replace("/echo/", "");

      string encodingHeader = ""; // default: no encoding header
      string encodingData = httpRequestParts.FirstOrDefault(item => item.StartsWith("Accept-Encoding", StringComparison.OrdinalIgnoreCase)) ?? "";

      if (!string.IsNullOrEmpty(encodingData) && encodingData.Contains(':')){
        string encodingValue = encodingData.Split(":", 2)[1].Trim().ToLower();

        if (validCompressions.Contains(encodingValue)){
          encodingHeader = $"Content-Encoding: {encodingValue}\r\n";
        }
      }

      string response = $"{httpVersion} 200 OK\r\n" +
            "Content-Type: text/plain\r\n" +
            encodingHeader +// Replacing content-encoding (when requested)
            $"Content-Length: {data.Length}\r\n\r\n" + // Replacing content-length
            data;                                      // Replacing response body
      socket.Send(Encoding.UTF8.GetBytes(response));
    }
// ...

Multiple compression schemes

In this stage, I'll add support for Accept-Encoding headers that contain multiple compression schemes. An HTTP client can specify that it supports multiple compression schemes by setting Accept-Encoding to a comma-separated list:

Accept-Encoding: encoding-1, encoding-2, encoding-3

To manage multiple compression schemes, I first extract the list of encoding types and convert them to lowercase. Then, I use a foreach loop to iterate through each encoding type. Next, I check if the encoding type exists in the HashSet of supported encodings. If gzip is detected, I update the encoding header.

// ...
    if (!string.IsNullOrEmpty(encodingData) && encodingData.Contains(':'))
      {
        string encodingValues = encodingData.Split(":", 2)[1].Trim().ToLower();

        foreach (var compressionType in encodingValues.Split(",").Select(values => values.Trim()))
        {
          if (validCompressions.Contains(compressionType))
          {
            if (compressionType == "gzip")
            {
              encodingHeader = $"Content-Encoding: {compressionType}\r\n";
              break;
            }
          }
        }
      }
// ...

Gzip compression

For this stage, I'll add support for gzip compression to the HTTP server. GZip is a format designed to compress HTTP content before it is delivered to a client. It utilizes the gzip algorithm to reduce file size and optimize transmission. By reducing file size, Gzip enhances data transmission speeds, which is why web servers often compress data before sending it to the client’s browser.

In order to achieve this task on the server. If gzip is detected, I update the encoding header and apply gzip compression to the response data. I created a function called handleGzipCompression that takes the data to be compressed. GZipStream takes an stream and a compression mode as parameters - Compress in this case. I am using a memory stream as the output stream. When data is written into the GZipStream, it goes into the output stream as compressed data. Then, I convert the result into a byte array that will be returned by the function.

Additionally, I modified how the response is constructed to ensure the compressed data is sent correctly to the client.

// ...
      string data = requestTarget.Replace("/echo/", "");
      byte[] responseBody = Encoding.UTF8.GetBytes(data); // Encode the response body using UTF-8 by default
// ...
        if (compressionType == "gzip") // If gzip is supported and requested, compress the body and update headers
            {
              encodingHeader = $"Content-Encoding: {compressionType}\r\n";
              responseBody = handleGzipCompression(data); // Apply gzip compression
              break;
            }
        }
// ...
      string responseHeaders = $"{httpVersion} 200 OK\r\n" +
                        "Content-Type: text/plain\r\n" +
                        encodingHeader + // content-encoding (when requested)
                        $"Content-Length: {responseBody.Length}\r\n\r\n";  // Replacing content-length

      socket.Send([.. Encoding.UTF8.GetBytes(responseHeaders), .. responseBody]);

// ...

byte[] handleGzipCompression(string originalData)
{
  byte[] dataToCompress = Encoding.UTF8.GetBytes(originalData);
  using (var outputStream = new MemoryStream())
  {
    using (var compressor = new GZipStream(outputStream, CompressionMode.Compress))
    {
      compressor.Write(dataToCompress, 0, dataToCompress.Length);
    }

    return outputStream.ToArray();
  }
}

Adding persistent connections

For this stage, I'll add support for persistent HTTP connections. By default, HTTP/1.1 connections are persistent, meaning the same TCP connection can be reused for multiple requests. This connection is persistent unless the client sends a Connection: close header. The server should not close the connection after each request.

To accomplish this feature, I updated the handleRequest function by adding a while loop to validate if the client´s connection remains open. Next, I check if the Connection header appear in the request. If its value is close, the shouldClose flag is set to true, otherwise it remains false. Finally, at the end of each request processed, I check the value of shouldClose, if it is true, the client connection is closed.

// ...
void handleRequest(Socket socket)
{
  Console.WriteLine($"Connection from {socket.RemoteEndPoint} has been established");

  while (socket.Connected)
  {
    // ...
    string connectionData = httpRequestParts.FirstOrDefault(item => item.StartsWith("Connection", StringComparison.OrdinalIgnoreCase)) ?? "";
    bool shouldClose = false;

    if (connectionData.Contains(':')) // Check for `Connection: close` header
    {
      string connectionValue = connectionData.Split(":", 2)[1].Trim().ToLower();
      if (connectionValue == "close")
      {
        shouldClose = true;
      }
    }
    // ...

    //Closes the socket to free up system resources
    if (shouldClose)
    {
      Console.WriteLine($"Connection from {socket.RemoteEndPoint} has been closed by the client.");
      socket.Close();
      break;
    }
  }
}

Adding connection closure

In this stage, I added support for explicit connection closure using the Connection: close header.

To implement this, I added a connectionHeader value with an empty string by default. When the Connection header appear in the request, then I update the value with the proper Connection: close header in the response. Also, I update all the responses of the endpoints in order to use this variable.

    string connectionHeader = "";
// ...
    if (connectionData.Contains(':')) // Check for `Connection: close` header
    {
      string connectionValue = connectionData.Split(":", 2)[1].Trim().ToLower();
      if (connectionValue == "close")
      {
        connectionHeader = $"Connection: close\r\n";
        shouldClose = true;
      }
    }
// ...

        string response = $"{httpVersion} 200 OK\r\n" +
                          $"{connectionHeader}\r\n";

        socket.Send(Encoding.UTF8.GetBytes(response));

Refactoring the code

I added a new function called generateHttpResponse, which is responsible for generating the proper HTTP response. It takes the HTTP version, status code, status message, a dictionary of the response headers, and the body data in bytes as parameters. The function returns a byte array representing the HTTP response, ready to be sent to the client.

// ...
    //Sends a response string to the connected client.
      if (urlPath == "/")
      {
        socket.Send(generateHttpResponse(httpVersion, 200, "OK", headers, body));
      }
// ...
byte[] generateHttpResponse(string httpVersion, int statusCode, string statusMessage, Dictionary<string, string> headers, byte[] body)
{
  string response = $"{httpVersion} {statusCode} {statusMessage}\r\n";

  foreach ((string headerName, string value) in headers)
  {
    response += $"{headerName}: {value}\r\n";
  }

  response += "\r\n";

  byte[] responseBytes = Encoding.UTF8.GetBytes(response);

  return [.. responseBytes, .. body];
}

I also created a function called extractHTTPRequest, which is responsible for extracting the HTTP Request data. This function takes HTTP Request in bytes as parameter and it returns a tuple containing three items.

A dictionary with the elements of the request line.
A dictionary with all the headers in the request.
A string that contains the body content.

// ...
    Dictionary<string, string> requestLine;
    Dictionary<string, string> requestHeaders;
    string requestBody;

    (requestLine, requestHeaders, requestBody) = extractHTTPRequest(httpRequest);
// ...
(Dictionary<string, string> requestLine, Dictionary<string, string> headers, string body) extractHTTPRequest(byte[] httpRequest)
{
  string[] requestParts = Encoding.UTF8.GetString(httpRequest).Split("\r\n");
  Dictionary<string, string> requestLineDict = new();
  Dictionary<string, string> headers = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);

  string requestLine = requestParts[0];

  int indexEmptyLine = Array.IndexOf(requestParts, String.Empty); // First empty line (that separates headers from the body).
  string[] headersArray = requestParts[1..indexEmptyLine];

  // Parse request line
  string[] requestLineParts = requestLine.Split(" ");
  if (requestLineParts.Length == 3)
  {
    (requestLineDict["Method"], requestLineDict["Target"], requestLineDict["Version"]) = (requestLineParts[0], requestLineParts[1], requestLineParts[2]);
  }

  // Extract headers
  foreach (string header in headersArray)
  {
    string[] data = header.Split(":", 2);
    (string headerName, string value) = (data[0].Trim(), data[1].Trim());
    headers[headerName] = value;
  }

  // Extract body
  string body = string.Join(Environment.NewLine, requestParts[(indexEmptyLine + 1)..]);

  return (requestLineDict, headers, body);
}

If you're curious to see how everything came together, you can check out the full project on GitHub: HTTP Server (C#). The code is organized to follow each stage of the process, and I’ve tried to keep it clean and easy to follow. Feel free to explore, fork the repo, or open an issue if you have ideas or suggestions—I'd love to hear your thoughts!

Conclusion

This experience allowed me to learn and improve my back-end development skills. Build this project gave me a solid foundation for understanding how the web server works.

Resources

Here are some of the materials and references that helped me throughout this project:

Compressing Strings Using GZip in C# - A post that explains the use of GZipStream class from .NET Framework.
Build your own HTTP server - Challenge page with the stages to build this project from scratch.
HTTP/1.1: Protocol Overview (MDN) - A great resource for understanding HTTP basics.

Step-by-Step Guide to Creating an HTTP Server in C# - Part 2

Refactoring the code

Compression headers

Multiple compression schemes

Gzip compression

Adding persistent connections

Adding connection closure

Refactoring the code

Conclusion

Resources

Subscribe to my newsletter

Saul Hernandez

Saul Hernandez