Logging with Logrus: Streamlining MongoDB Integration

Logging plays a crucial role in modern programming by capturing errors, warnings, performance metrics, and debug messages. It serves as a means for our software to communicate with us, providing insights into its behavior.

In this blog, we'll learn how to transfer our logs to MongoDB, enhancing our troubleshooting capabilities. By storing logs in MongoDB, we can easily analyze and debug issues, improving the efficiency of our software development process.

Installing Logrus

Logrus is a structured logger for Go (golang), completely API compatible with the standard library logger. Currently logrus is in maintenance-mode and there will be no new features added to this package, so if have not added logging package to your project then you should checkout Zap, Zerolog. They are popular logging packages in go. And if you still want to go ahead in installing Logrus package then use the following command

go get github.com/sirupsen/logrus

Basics of Logrus

Now that we have installed the logrus package let me show you few basic of how to use different methods for different purposes.

package main

import "github.com/sirupsen/logrus"

func main() {
    logger := logrus.New()
    // SetReportCaller will add the function name and filepath from where log was added
    logger.SetReportCaller(true)

    logger.Info("Log some information")
    logger.Error("Log some error")
    logger.Println("add some print statement")
    logger.Debug("Log something for debugging purpose")
}

go run main.go
time="2024-04-27T10:24:13+05:30" level=info msg="Log some information" func=main.main file="D:/go-app/logger/logrus-blog.go:9"
time="2024-04-27T10:24:13+05:30" level=error msg="Log some error" func=main.main file="D:/go-app/logger/logrus-blog.go:10"
time="2024-04-27T10:24:13+05:30" level=info msg="add some print statement" func=main.main file="D:/go-app/logger/logrus-blog.go:11"

These are few basic logging options that are provided by logrus. You can also set the logging level. This is helpful has you don't want to add debugging logs into your production server. You can set log level using logger.SetLevel(level) function. For additional information and all the functions provided by logrus you can head ahead to their documentation here: https://github.com/sirupsen/logrus?tab=readme-ov-file

Extracting logs into file

Logrus supports different logging options like output logs into console or file. We can set the output by using SetOutput function as follow:

Logging into stdout

package main

import (
    "os"

    "github.com/sirupsen/logrus"
)

func main() {
    logger := logrus.New()
    logger.SetReportCaller(true)
    logger.SetLevel(logrus.InfoLevel)
    logger.SetOutput(os.Stdout)

    logger.Info("Log some information")
    logger.Error("Log some error")
    logger.Println("add some print statement")
    logger.Debug("Log something for debugging purpose")
}

Logging into file

package main

import (
    "os"

    "github.com/sirupsen/logrus"
)

func main() {
    logger := logrus.New()
    logger.SetReportCaller(true)
    logger.SetLevel(logrus.InfoLevel)

    logFile := "logger.log"
    file, err := os.OpenFile(logFile, os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0644)
    if err != nil {
        logger.Error(err)
        panic(err)
    }

    defer file.Close()

    logger.SetOutput(file)

    logger.Info("Log some information")
    logger.Error("Log some error")
    logger.Println("add some print statement")
    logger.Debug("Log something for debugging purpose")
}

This will output the logs into logger.log file in the current directory.

Using hooks to extract logs into MongoDB

In Logrus, hooks are essentially interfaces that can be added to the logger to execute custom logic or side effects when log entries are made. A hook in Logrus only has to implement the Hook interface, which involves implementing two methods: Levels() and Fire().

• Levels(): Defines for which log levels this hook will be triggered.

• Fire(): Contains the logic that gets executed when a log of the specified levels is made.

First lets connect to MongoDB.

package main

import (
    "context"
    "fmt"

    "github.com/sirupsen/logrus"
    "go.mongodb.org/mongo-driver/bson"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

func main() {
    logger := logrus.New()
    logger.SetReportCaller(true)

    mongoClient := connectToMongo()
    defer func() {
        if err := mongoClient.Disconnect(context.TODO()); err != nil {
            panic(err)
        }
    }()
}

func connectToMongo() *mongo.Client {
    serverAPI := options.ServerAPI(options.ServerAPIVersion1)
    opts := options.Client().ApplyURI("mongodb+srv://<username>:<password>@local-logger.vqrzbh1.mongodb.net/?retryWrites=true&w=majority&appName=local-logger").SetServerAPIOptions(serverAPI)

    client, err := mongo.Connect(context.TODO(), opts)
    if err != nil {
        panic(err)
    }

    // Send a ping to confirm a successful connection
    if err := client.Database("admin").RunCommand(context.TODO(), bson.D{{"ping", 1}}).Err(); err != nil {
        panic(err)
    }

    fmt.Println("Pinged your deployment. You successfully connected to MongoDB!")
    return client
}

This is what I am using to connect to mongo. This is very naive implementation you should change the implementation of connecting mongodb.

Setup Hook

package main

import (
    "context"
    "fmt"
    "time"

    "github.com/sirupsen/logrus"
    "go.mongodb.org/mongo-driver/bson"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

func main() {
    logger := logrus.New()
    logger.SetReportCaller(true)

    mongoClient := connectToMongo()
    defer func() {
        if err := mongoClient.Disconnect(context.TODO()); err != nil {
            panic(err)
        }
    }()

    mongoHook := MongoHook{
        Client: mongoClient,
    }
}

type MongoHook struct {
    Client *mongo.Client
}

type Log struct {
    Timestamp time.Time
    Level     string
    Message   string
    File      string
    Function  string
}

func (hook MongoHook) Levels() []logrus.Level {
    return []logrus.Level{logrus.ErrorLevel}
}

func (hook MongoHook) Fire(entry *logrus.Entry) error {
    logObj := Log{
        Timestamp: entry.Time,
        Level:     entry.Level.String(),
        Message:   entry.Message,
        File:      entry.Caller.File,
        Function:  entry.Caller.Function,
    }

    collection := hook.Client.Database("<your-databasename>").Collection("<your-collectionname>")
    collection.InsertOne(context.TODO(), logObj)
    return nil
}

func connectToMongo() *mongo.Client {
    serverAPI := options.ServerAPI(options.ServerAPIVersion1)

    opts := options.Client().ApplyURI("mongodb+srv://<username>:<password>@local-logger.vqrzbh1.mongodb.net/?retryWrites=true&w=majority&appName=local-logger").SetServerAPIOptions(serverAPI)
    client, err := mongo.Connect(context.TODO(), opts)
    if err != nil {
        panic(err)
    }

    // Send a ping to confirm a successful connection
    if err := client.Database("admin").RunCommand(context.TODO(), bson.D{{"ping", 1}}).Err(); err != nil {
        panic(err)
    }

    fmt.Println("Pinged your deployment. You successfully connected to MongoDB!")
    return client
}

There is lot of stuff going on here now, let's cover everything step by step.

MongoHook: This is struct whose methods will implement the hook interface of logrus package. I have added the mongo's client instance to it.

Levels(): In the Levels() method I am returning the level of log which will be return into mongodb, has I don't want everything to go into mongo and troubleshooting becomes difficult.

Fire(): This method is responsible for actually writing the logs into our MongoDB. The first thing I do is create a object which I want to write into database. The entry object provides various field which could be used for troubleshooting and you can choose them based on your choice. I have added few of them for demo purpose. Once the object is created I select the collection into which the logs are to be written. And finally the logs are written into the collection using InsertOne() function.

In this example I have used MongoDB to write my logs but we can use any entity to write our logs. We just need to implement the Hook interface of the logrus package and we are good to go.
Integrating Logrus with external entity in MongoDB for logging in your Golang applications can greatly enhance your troubleshooting and debugging processes. Start implementing these techniques in your projects today to experience the benefits firsthand. Happy logging!