Handling Large File Uploads in Kubernetes: Resolving 413 Payload Too Large Error with Ingress-NGINX Controller

Application

The backend application, developed using Java and Spring Boot, is hosted on Google Kubernetes Engine (GKE). An API is provided for customers and internal applications to upload files and process them.

Problem

Customers were unable to upload files with large datasets. Additionally, the requests were being rejected and did not reach the downstream applications.

Root Cause and Solution

Error: Unexpected HTTP status: 413 Payload Too Large. This error was logged in both the front-end and back-end application logs. Initially, the root cause of the issue was unclear, so we needed to investigate where the request was being rejected. After analyzing the request flow across the network, we discovered that the ingress-nginx controller was responsible for blocking the request.

ingress-nginx is an Ingress controller for Kubernetes, using NGINX as a reverse proxy and load balancer.

By default, the Kubernetes NGINX ingress controller allows a maximum request size of 1 MB. To increase the allowed maximum request size, we needed to configure the proxy-body-size setting in the NGINX ConfigMap.

nginx.ingress.kubernetes.io/proxy-body-size: <filesize in MB>

Example:

nginx.ingress.kubernetes.io/proxy-body-size: 50m

Learning

The API was tested in the QA environment with files smaller than 1 MB and with valid data. Here are the key takeaways:

• Always gather information regarding the dataset size.

• Understand how many customers are uploading large datasets.

• Carefully review all configurations; in this case, ingress was often overlooked from a functional testing perspective.

• Thoroughly test the flow of API requests across the network and familiarise yourself with the default policies for each component.

References

• https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/

• https://www.f5.com/products/nginx/nginx-ingress-controller

• https://docs.nginx.com/nginx-management-suite/acm/how-to/policies/request-body-size-limit/

• https://docs.nginx.com/nginx-ingress-controller/overview/design/