Yandex Cloud Logging from Ubuntu 22.04

Fluent Bit is a lightweight and versatile log collector and forwarder designed to handle high-volume data streams from various sources. It is an open-source tool that offers a range of features, including filtering, buffering, and routing capabilities. Fluent Bit is widely used in cloud environments, such as Kubernetes and Docker, for collecting and forwarding logs.

In this guide, I will show you how to use Fluent Bit to send Docker logs to Yandex Cloud Logging. Yandex Cloud Logging is a managed logging service that allows you to store, search, and analyze logs from various sources in one centralized location.

By following this guide, you will learn how to set up and configure Fluent Bit to collect and forward Docker logs to Yandex Cloud Logging. We will cover the necessary steps to set up Fluent Bit and Yandex Cloud Logging, configure Fluent Bit to collect Docker logs, and forward them to Yandex Cloud Logging.

Firstly, install Docker

# Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo addgroup dev docker

Then install libssl from archive (cant be installed from apt-get)

# libssl
wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1-1ubuntu2.1~18.04.21_amd64.deb
sudo dpkg -i libssl1.1_1.1.1-1ubuntu2.1~18.04.21_amd64.deb

Install td-agent-bit

wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -
sudo sh -c 'echo "deb https://packages.fluentbit.io/ubuntu/focal focal main" >> /etc/apt/sources.list'
sudo apt-get update
sudo apt-get install td-agent-bit
sudo systemctl start td-agent-bit
sudo systemctl enable td-agent-bit
sudo systemctl status td-agent-bit

Install GOlang

wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz
tar -xzf go1.17.6.linux-amd64.tar.gz
export PATH=$PWD/go/bin:$PATH

Install gcc

# gcc
sudo apt install build-essential

Clone, compile, install fluent-bit-plugin-yandex

# fluent-bit-plugin-yandex
git clone https://github.com/yandex-cloud/fluent-bit-plugin-yandex.git
cd fluent-bit-plugin-yandex/
export fluent_bit_version=1.8.6
export plugin_version=dev
CGO_ENABLED=1 go build     -buildmode=c-shared \
  -o ./yc-logging.so \
  -ldflags "-X main.PluginVersion=${plugin_version}" \
  -ldflags "-X main.FluentBitVersion=${fluent_bit_version}"
sudo cp yc-logging.so /usr/lib/td-agent-bit/yc-logging.so
sudo sh -c'echo "\tPath /usr/lib/td-agent-bit/yc-logging.so" >> /etc/td-agent-bit/plugins.conf'

/etc/td-agent-bit/td-agent-bit.conf:

[SERVICE]
log_level debug

[INPUT]
    Name forward
    Listen 0.0.0.0
    port 24224

[OUTPUT]
    Name            yc-logging
    Match           *
    resource_type   logtest
    folder_id       <FOLDER_ID>
    message_key     log
    level_key       SEVERITY
    default_level   INFO
    authorization   instance-service-account

sudo systemctl restart td-agent-bit

Now you can try to run container: docker run --rm --log-driver=fluentd hello-world