Building A Custom Payroll System: MongoDB, Go, and Yellowcard API Series - Part 1

It took an entire day to decide on a project for an assessment test. I kept going back and forth between my flatmate's room and mine, explaining the benefits of building a currency rate tracker instead of a payroll system to showcase my technical skills.

Well, I finally decided to build a payroll system using MongoDB, which wasn't commonly used by the Go community, so library support was limited compared to SQL 🫠. Another decision I had to make was how to build the application. Having a sketch on paper is enough to set up the application code. Choosing the right routing library is also important. In this case, we will use the Gin library framework.

Let's get started.

Architecture 🏛️

I didn't plan to discuss this application's architecture right away. I wanted to explain it while going through the code, but one of my reviewers requested it, so let's dive in.

Design decisions, thought processes, trade-offs, and pros and cons are crucial when building an application. For the payroll system, MongoDB was chosen to explore the synergy between Golang and MongoDB and avoid migrations like in SQL, especially since all assessments come with deadlines.

Let's take a peek at the application design diagram for a while.

Here is a rough illustration of the application. Initially, I tried supporting database pools, but for simplicity's sake and to avoid premature optimization, I removed it. Instead, I focused on the following assumption.

• Businesses can add, edit, delete employees and make payments.

• All payments are in Naira only

• Refunds and cancellations are not supported

• The account has been funded via the YellowCard dashboard

• The user is the business owner

• The provided information has been validated

• KYC meta information for employees and employers has been collected and verified the above assumptions and requirements help determine the key modules to focus on: the employee, webhook, authentication, and disbursement modules.

That's about it for the assumptions of this design. We will explore more as we move forward.

Configuration 📜

Usually, every application starts with identifying configuration properties. One important thing to note is that we won't be using an env file but a yaml file for all our configurations. However, the code includes support for loading from an env file.

Let's have a closer look at it. Create a file common/config.go and add some structures for storing environmental file paths.

type ConfEnvSetting struct {
    YamlFilePath []string
    EnvFilePath  string
}

This structure is designed to store the paths to all the files we want to load. Next, we add our application configuration details

type Config struct {
    LogLevel   string `config:"logLevel"`
    ServerPort string `config:"serverPort"`

    YellowCardCredentials struct {
        ApiKey    string `config:"apiKey"`
        SecretKey string `config:"secretKey"`
        BaseUrl   string `config:"baseUrl"`
    }

    JWTCredentials struct {
        AccessTokenSecret string `config:"accessTokenSecret"`
        AccessTokenClaim  struct {
            Issuer   string `config:"issuer"`
            Audience string `config:"audience"`
        }
        AccessTokenTTL time.Duration `config:"accessTokenTTL"`
    }

    MongoDB struct {
        DBUri        string `config:"dbUri"`
        DatabaseName string `config:"databaseName"`
    }

    AllowedCorsOrigin []string `config:"allowedCorsOrigin"`
}

Still in common/config.go, let's run go mod init <your-project-name> to set up our go.mod and go.sum files to manage external library dependencies. In this case, we will install the following libraries: github.com/gookit/config/v2, github.com/gookit/config/v2/yaml, and github.com/samber/lo.

Now, let's add the LoadConfiguration function to load the configuration into our main.go.

import (
    "encoding/json"
    "fmt"
    "log"
    "sync"
    "time"

    "github.com/gookit/config/v2"
    "github.com/gookit/config/v2/yaml"
    "github.com/samber/lo"
)

// Insert the previous code hear.

var (
    cfg    = &Config{}
    loaded = false
)
var once sync.Once

func LoadConfiguration(cfgEnvSetting ConfEnvSetting) (*Config, error) {

    if loaded {
        return cfg, nil
    }

    config.WithOptions(func(opt *config.Options) {
        opt.DecoderConfig.TagName = "config"
    })

    config.WithOptions(config.ParseEnv)

    if len(cfgEnvSetting.YamlFilePath) != 0 {
        config.AddDriver(yaml.Driver)
    }
    files := []string{}
    if !lo.IsEmpty(cfgEnvSetting.EnvFilePath) {
        files = append(files, cfgEnvSetting.EnvFilePath)
    }
    files = append(files, cfgEnvSetting.YamlFilePath...)
    err := config.LoadFiles(files...)
    if err != nil {
        return nil, err
    }

    once.Do(func() {
        err := config.Decode(cfg)
        if err != nil {
            log.Panic(err)
        }
        loaded = true
    })

    if cfg.LogLevel == "debug" {
        cfgJson, err := json.MarshalIndent(cfg, "", "    ")
        if err == nil {
            fmt.Printf("Cudium-Backend API Config: %s\n", cfgJson)
        }
    }

    return cfg, nil
}

func GetConfig() *Config {
    return cfg
}

With this done, let's set up our logger.

Logging 🪵

Logging should be a priority for any application being built as it helps identify where issues might occur, especially when dealing with bugs in production 😞.

From my personal experience, using logged stack traces is very important to easily identify where crashes or bugs might lie. This helps pinpoint the line that caused the error.

Let's see how we can implement a basic logging package for the application.

Create an internals/logger.go file and put the code below in it. I will explain what the code does in a bit.

package internals

import (
    "os"

    "github.com/op/go-logging"
)

// Logger is a server logger
type Logger interface {
    Fatal(args ...interface{})
    Fatalf(format string, args ...interface{})
    Panic(args ...interface{})
    Panicf(format string, args ...interface{})
    Critical(args ...interface{})
    Criticalf(format string, args ...interface{})
    Error(args ...interface{})
    Errorf(format string, args ...interface{})
    Warning(args ...interface{})
    Warningf(format string, args ...interface{})
    Notice(args ...interface{})
    Noticef(format string, args ...interface{})
    Info(args ...interface{})
    Infof(format string, args ...interface{})
    Debug(args ...interface{})
    Debugf(format string, args ...interface{})
}

// LoggerID ia a logger ID
const LoggerID = "yc-backend"

var (
    // Logger settings
    logger           = logging.MustGetLogger(LoggerID)
    logConsoleFormat = logging.MustStringFormatter(
        `%{color}%{time:2006/01/02 15:04:05} [YC-Backend] >> %{message} %{color:reset}`,
    )
)

func init() {
    // Prepare logger
    logConsoleBackend := logging.NewLogBackend(os.Stderr, "", 0)
    logConsolePrettyBackend := logging.NewBackendFormatter(logConsoleBackend, logConsoleFormat)
    lvl := logging.AddModuleLevel(logConsolePrettyBackend)
    logging.SetBackend(lvl)
    //logging.SetLevel(logging.INFO, LoggerID)
    // Set log level based on env
    switch os.Getenv("APP_LOG_LEVEL") {
    case "debug":
        logging.SetLevel(logging.DEBUG, LoggerID)
    case "info":
        logging.SetLevel(logging.INFO, LoggerID)
    case "warn":
        logging.SetLevel(logging.WARNING, LoggerID)
    case "err":
        logging.SetLevel(logging.ERROR, LoggerID)
    default:
        logging.SetLevel(logging.INFO, LoggerID) // log everything by default
    }
}

// GetLogger returns the logger
func GetLogger() Logger {
    return logger
}

In this code snippet, we created a logger interface to help us print application runtime information to the console.

Next, let's build our application engine.

Engine 🚂

You might be wondering, "Why do we need an engine?" We need a configurable engine to bring together all the necessary parts to make the application work. For example, if our application requires a database, this is the perfect place to integrate it, along with any other components we need to add.

Let's create a folder engine/app.go and add the following line of code to the file. Also, import the following libraries github.com/gin-contrib/cors ,github.com/gin-contrib/gzip,github.com/gin-gonic/gin,github.com/samber/lo and go.mongodb.org/mongo-driver/mongo.

import (
    "context"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "sync"
    "syscall"
    "time"
    "yc-backend/internals" // import the logger

    "github.com/gin-contrib/cors"
    "github.com/gin-contrib/gzip"
    "github.com/gin-gonic/gin"
    "github.com/samber/lo"
    "go.mongodb.org/mongo-driver/mongo"
)

type Application struct {
    Config  *common.Config
    DB      *mongo.Client
    Logger  internals.Logger
    Context context.Context
    server  *http.Server

    mux  *gin.Engine
    wg   sync.WaitGroup
    quit chan os.Signal
}

func (srv *Application) Setup() *Application {
    r := gin.New()

    // we will add middleware integration ...
    r.Use(gzip.Gzip(gzip.DefaultCompression))
    r.Use(gin.Logger())
    r.Use(gin.Recovery())
    r.Use(func(ctx *gin.Context) {
        cfg := common.ConfigFromCtx(ctx)
        if cfg == nil {
            panic(errors.New("allowed cross-origin not set"))
        }
        cors.New(cors.Config{
            AllowMethods:     []string{"GET", "POST", "PUT", "DELETE", "PATCH", "OPTIONS"},
            AllowHeaders:     []string{"Accept", "Authorization", "Content-Type", "X-CSRF-Token"},
            AllowCredentials: true,
            ExposeHeaders:    []string{"Content-Length"},
            AllowOriginFunc: func(origin string) bool {
                return lo.Contains(cfg.AllowedCorsOrigin, origin)
            },
            MaxAge: 30 * time.Second,
        })(ctx)
        ctx.Next()
    })

    r.GET("/status-check", func(ctx *gin.Context) {
        ctx.JSON(http.StatusOK, utils.SuccessResponse("alive🫵", gin.H{"status": "ok"}))
    })

    srv.mux = r
    srv.wg = sync.WaitGroup{}
    srv.server = &http.Server{
        Addr:              srv.Config.ServerPort,
        Handler:           srv.mux,
        ReadTimeout:       15 * time.Millisecond,
        WriteTimeout:      30 * time.Millisecond,
        ReadHeaderTimeout: 15 * time.Millisecond,
    }

    srv.quit = make(chan os.Signal, 1)
    signal.Notify(srv.quit, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
    return srv
}

In the snippet above, we created an application structure, attached an HTTP server, and configured our router. A status check endpoint lets us easily verify if our server is active.

Let's handle graceful shutdown and server listener logic also for our base logic

func (srv *Application) GracefulShutdown() {
    go func(quit chan os.Signal, dbm *mongo.Client) {
        <-quit
        shutdownCtx, shutdownCancelFunc := context.WithTimeout(srv.Context, 5*time.Second)
        go func() {
            defer shutdownCancelFunc()
            <-shutdownCtx.Done()
            if shutdownCtx.Err() == context.DeadlineExceeded {
                log.Fatal("graceful shutdown timed out.. forcing exit.")
            }
        }()

        err := srv.server.Shutdown(shutdownCtx)
        if err != nil {
            log.Fatal(err)
        }
        err = dbm.Disconnect(shutdownCtx)
        if err != nil {
            log.Fatal(err)
        }
        log.Fatal("graceful shutdown... forcing exit.")
    }(srv.quit, srv.DB)
}

func (srv *Application) ListenAndServe() error {
    if err := srv.mux.Run(srv.Config.ServerPort); err != nil {
        return err
    }
    <-srv.Context.Done()
    return nil
}

We expect that when we try to shut down the server, all currently handled requests will be completed successfully. This code should also be included in the engine/app.go file.

Database Setup 🚀

For the final part of this article, which will be part of a possible 3-part series, we will set up main.go in our root folder where our go.mod the file is located. Then, add the following code to connect our MongoDB database to our application. In the next parts of the series, we will learn how to use the repository model to work with the Mongo client.

import (
    "context"
    "log"
    "time"

    "github.com/gin-gonic/gin"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

func main() {
    // code goes here...
}

func setupDatabase(clientOpts *options.ClientOptions) *mongo.Client {
    ctx, cancelFunc := context.WithTimeout(context.Background(), 15*time.Second)
    defer cancelFunc()

    client, err := mongo.Connect(ctx, clientOpts)
    if err != nil {
        log.Fatal(err)
    }

    ctx, cancelFunc = context.WithTimeout(context.Background(), 1*time.Second)
    defer cancelFunc()
    err = client.Ping(ctx, nil)
    if err != nil {
        log.Fatal(err)
    }
    log.Println(">>> Mongodb client connected")
    return client
}

The setupDatabase function allows Mongo db integration with our application and should an error occur when we call this function we exit the program. Prevention is better than cure I guess 🫠.

Now we have our basic application set up with the essential components. We can now complete and add the necessary business logic to the application. In the next part of this series, we will add models, repositories, and more.

I am Caleb and you can reach me on Linkedin or follow me on Twitter. @Soundboax