Nginx Upgrade

A small and simple plan for doing an nginx upgrade

This is in the context of using nginx inside Kubernetes as a container. So, you will see references to recommend using ingress and ingress controller, or Gateways, instead of standalone nginx instances.

Google Docs copy of this post

Scope

• Only standalone nginx instances are part of this upgrade

• Upgrade with no changes in features or functionality from the perspective of the clients of the nginx. We don’t promise that any of the new features (in the new version) of the nginx will be used unless it comes with no changes with respect to changes from the perspective of the clients of the nginx and also assuming it is not too much of a change to make in the existing configuration (Infrastructure as Code, nginx configuration etc) and also assuming it’s easy to test. Or else we will do just a lift and shift of sorts as much as possible - where we just put the configuration file with no changes or minimal changes in the new version and deploy it. Anything else will be done separately and not as part of this upgrade

Success Criteria

• The new version of nginx is rolled out without any service disruption - to the relevant services

  • Check Metrics, Events, Logs to look for any service disruption

    • Look at Resource Usage: CPU, RAM, Network (traffic - both inbound and outbound), Disk

    • Look at HTTP errors - 4xx and 5xx. Especially 5xx

Plan

• Decide on the version of nginx to use. Preferably the latest version - which is 1.29 as of this document writing (July 24th 2025) and it was released on 25th June 2025 9:39 AM IST. References: [1], [2], [3]

• Try out a blind upgrade - without checking differences between the old version and the new version

  • Simply upgrade the nginx version to the new version - in the Dockerfile and build a new image using CI and publish it (push it to container image registry) and use it in the dev environment and test it out

  • Configure the new nginx version with the same configuration that’s used in the old version. So that it will point to the correct set of applications and have correct and expected configuration (unless there are breaking changes in the new version)

  • Look for errors in the logs, metrics, events

  • Look for errors at the platform level in Kubernetes - Kubernetes related events etc

  • Check resource usage: CPU, RAM, Network, Disk. Use metrics and dashboards for this. Setup monitoring and observability beforehand, setup MELTS (Metrics, Events, Logs, Traces, Spans) beforehand and setup alerts beforehand - all of these - before doing the upgrade

  • Try running application tests against the new nginx

  • Run load tests also, to understand performance. Do benchmarking also. All of this is to compare the performance between the old version and the new version. So, run the load tests and benchmarking tests for old version too, preferably before doing the upgrade and before running it for the new version

• Understand what features of the nginx are being used by the clients (human users, product engineering teams, services etc). Confirm the understanding with the clients (teams etc)

  • For example, nginx has features to run plugins to extend nginx, using lua programming language source code etc. Check if any nginx plugins are being used

  • nginx has features to extend nginx using nginx modules, written in C programming language. Check if any nginx modules are being used

  • Understand any customisations done to nginx, any custom features used in nginx etc

  • Understand the compatibility between

    • Nginx and any nginx plugins (written in Lua etc)

    • Nginx and any nginx modules (written in C etc)

• Check compatibility between the versions of the different softwares involved at play over here

  • Nginx version

    • Versions of the Nginx plugins

    • Versions of the Nginx Modules

  • Compatibility between nginx and the base image operating system version

• Check the changelog of all the versions from the old version to the new version

  • Look for breaking changes

    • Removal of features

    • Removal of configuration directives in the nginx configuration

    • Rename of configuration directives in the nginx configuration

    • Change in behaviour in the nginx in general

    • Change in behaviour for any of the nginx configuration directives

  • Understand how the breaking changes affect us - based on our usage of the nginx, assuming we are not using all the features and configuration directives of nginx

• Ensure there’s a plan for

  • Ability to be able to downgrade, smoothly, without any issues

Future

• Recommend product teams to use ingress and ingress controller. Or Gateway API and Gateways, which is the successor of ingress and ingress controller

References:

[1]: https://github.com/nginx/nginx

[2]: https://github.com/nginx/nginx/releases

[3]: https://github.com/nginx/nginx/releases/tag/release-1.29.0