Ultimate guide: How to ship logs from EKS to CloudWatch using FluentBit

Shipping logs from your Amazon EKS (Elastic Kubernetes Service) cluster to Amazon CloudWatch is a crucial part of maintaining visibility and observability in your Kubernetes environment. Fluent Bit, a lightweight and efficient log processor, is commonly used to collect, process, and forward logs to various destinations, including CloudWatch. This article will guide you through setting up Fluent Bit on your EKS cluster to send logs to CloudWatch.

Step 1: Attach IAM policy to the EKS worker node role:

For FluenBit pods to ship logs from EKS nodes to CloudWatch, the nodes should have necessary permissions to perform CloudWatch operations.

Attach the “CloudWatchAgentServerPolicy” aws managed policy to the EKS worker node role

Step 2: Create amazon-cloudwatch namespace where fluent bit will be located.

kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml

Step 3: Create ConfigMap for FluentBit using the below command:

ClusterName=cluster-name
RegionName=cluster-region
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
kubectl create configmap fluent-bit-cluster-info \
--from-literal=cluster.name=${ClusterName} \
--from-literal=http.server=${FluentBitHttpServer} \
--from-literal=http.port=${FluentBitHttpPort} \
--from-literal=read.head=${FluentBitReadFromHead} \
--from-literal=read.tail=${FluentBitReadFromTail} \
--from-literal=logs.region=${RegionName} -n amazon-cloudwatch

Step 4: Deploy the FluentBit as a daemonset:

In case the worker nodes are of linux based instances, use the below command to deploy the FluentBit daemonset:

kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml

If the worker nodes are of windows based instances:

kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit-windows.yaml

That’s it, now verify the FluentBit pods status:

kubectl get pods -n amazon-cloudwatch

Thank you for taking the time to read this article! Keep up the great work, and happy deploying! 🚀 😊