Implementation of Fluent Bit with AWS CloudWatch within the Logging in Kubernetes - Best Practice and Benefits

Introduction

Logging is important when monitoring and debugging applications in a fast-paced container orchestration world. With Kubernetes being the most popular container orchestration platform, a reliable logging solution is also needed to facilitate perfect observability. Fluent Bit as a lightweight log processor, together with AWS CloudWatch, goes a long way in helping make an effective, highly scalable logging solution for Kubernetes environments. It provides best practices, sample codes, and pros and cons of Fluent Bit and AWS CloudWatch for implementing Kubernetes logging.

Benefits of logging

• Enhanced Observability: Improved visibility into the application’s behavior and performance through centralized logging.

• Faster Debugging: Easier to identify and troubleshoot issues by aggregating logs from multiple sources.

• Compliance and Security: Ensures that log data is securely stored and accessible for compliance and auditing purposes.

Pros and Cons of Fluent Bit

Pros

• Centralized logging: Aggregated logs from all pods of Kubernetes at one point, which are easy to handle and analyze.

• Scalability: The combination of Fluent Bit with CloudWatch gives room for scalability purposes to enable high-volume logs to be smoothly filed.

• Cost efficiency: Fluent Bit is lightweight and consumes a low amount of resources compared to other log processors.

• It has a rich configuration, so Fluent Bit provides flexibility in log filtering, parsing, and routing.

Cons

• Initial Setup Complexity: The initial setup of Fluent Bit with CloudWatch can be complex and requires careful configuration.

• Cost: Depending on the log volume, the cost of storing logs in CloudWatch can be significant.

• Maintenance: Ongoing maintenance is required to ensure the logging setup remains efficient and cost-effective.

Set up Fluent Bit on EKS / AWS using Terraform

Below is the Terraform configuration for deploying the Fluent Bit Helm chart onto an EKS cluster. It uses the helm_release resource, which manages Helm charts using Terraform, and deploys the Fluent Bit logging solution from AWS's Helm repository.

resource "helm_release" "aws-fluentbit" {

    name = "aws-fluentbit"

    repository = "https://aws.github.io/eks-charts"

    chart = "aws-for-fluent-bit"

    namespace = "logging"

    create_namespace = true

    timeout = 3600

    replace = true

    values = [file("fluentbit.yaml")]

}

Below it the fluentbit.yml file

cloudWatchLogs:

    enabled: true

    match: "*"

    region: "eu-north-1"

    logGroupName: "/aws/eks/fluentbit-cloudwatch/logs"

    logGroupTemplate: "/aws/eks/fluentbit-cloudwatch/workload/$kubernetes['namespace_name']"

    logStreamTemplate: "$kubernetes['pod_name'].$kubernetes['container_name']"

    logKey: "log"

    logFormat: "json"

    autoCreateGroup: true

    logRetentionDays: 7

    autoRetryRequests: true

How to export application logs to cloudwatch

Fluent Bit collects logs from /var/log/containers/*.log, which is where Kubernetes stores the container logs (stdout and stderr).

To mount /logs/staging.log from inside the container to /var/log/staging.log on the host or Kubernetes node, you need to make use of Kubernetes hostPath volumes. Here’s an updated version of your deployment YAML to achieve this:

volumes:
    - name: log-volume
      hostPath:
      path: /var/log/containers/applicationname-logfile.log # Host path for app1 logs
      type: "DirectoryOrCreate"

volumeMounts:
    - name: log-volume
      mountPath: /applicationpath/log/staging.log
      subPath: applicationname-staging.log
      readOnly: false

Verify Logs in CloudWatch

• Navigate to the CloudWatch console.

• Check the log group /aws/containerinsights/<application-name>/application.

• Verify the log streams prefixed with <application-name>-<pod-name>.

Conclusion

Fluent Bit and AWS CloudWatch integration offers robustness and scalability for a solution in the domain of Kubernetes logging. Best practices and resolutions of challenges help organizations achieve high-level observability, enhanced debugging, and security standard compliance in the application logs. This might need some initial effort and careful configuration, but it is all worth it, given the benefits that accrue in terms of centralized and efficient log management.

Resources:

Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs https://artifacthub.io/packages/helm/aws/aws-for-fluent-bit

---

Stay Tuned!

Be sure to follow and subscribe for more updates and upcoming blogs.