Building A Custom Payroll System: MongoDB, Go, and Yellowcard API Series - Part 2

Prerequisite - check out the first article here

In the last article, we explored the architecture, engine, logging, database, and other basic setups for our proposed payroll system. These are just the initial steps to get things started. In this part of the series, we will look at models, repositories, and middleware to ensure that our router can access the necessary dependencies.

Let's take a look at what the model relationship looks like before introducing the code representation.

I don't usually draw an ER (entity relationship) diagram until I have identified all the attributes and figured out the relationships between the data models. By that time, I am also writing the code for the model. Then, I can go to the drawing board. This is my typical approach.

Let's talk about the model above alongside its code equivalent mapping 😉.

Model & ER Diagram 💃

To transfer funds to employees, we need some verified information. In our previous article, we made some assumptions to keep the application simple. The employer has verified the information through the KYC (Know Your Customer) process, ensuring that all information provided to the payment provider is valid.
Looking at the user table definition, let's create a user.go file in the model folder. So, we have model/user.go. Let's add this code to it.

package models

import (
    "encoding/json"
    "reflect"
    "time"

    "github.com/samber/lo"
    "go.mongodb.org/mongo-driver/bson/primitive"
)

type User struct {
    ID                 primitive.ObjectID `validate:"mongodb" json:"id,omitempty" bson:"_id,omitempty"`
    FirstName          string             `json:"firstName,omitempty" bson:"firstName,omitempty"`
    LastName           string             `json:"lastName,omitempty" bson:"lastName,omitempty"`
    Email              string             `validate:"required,email" json:"email,omitempty" bson:"email,omitempty"`
    Password           string             `validate:"required" json:"password,omitempty" bson:"password,omitempty"`
    CreatedAt          *time.Time         `json:"-" bson:"createdAt,omitempty" validate:"required"`
    UpdatedAt          *time.Time         `json:"-" bson:"updatedAt,omitempty" validate:"required"`
    MiddleName         string             `json:"middleName,omitempty" bson:"middleName,omitempty"`
    BVN                string             `json:"bvn,omitempty" bson:"bvn,omitempty"`
    DOB                string             `json:"dob,omitempty" bson:"dob,omitempty"`
    Address            string             `json:"address,omitempty" bson:"address,omitempty"`
    Phone              string             `json:"phone,omitempty" bson:"phone,omitempty"`
    Country            string             `json:"country,omitempty" bson:"country,omitempty"`
    IdNumber           string             `json:"idNumber,omitempty" bson:"idNumber,omitempty"`
    IdType             string             `json:"idType,omitempty" bson:"idType,omitempty"`
    AdditionalIdType   string             `json:"additionalIdType,omitempty" bson:"additionalIdType,omitempty"`
    AdditionalIdNumber string             `json:"additionalIdNumber,omitempty" bson:"additionalIdNumber,omitempty"`
}

Here, we define a struct for our User class with primitive objectId support for MongoDB's primary key. This helps us establish relationships moving forward. Later on, we will add some functionality to help us mask away some fields.

Then, we have a similar scenario with the employee regarding the KYC information. Create a models/employee.go file and add this code

import (
    "encoding/json"
    "reflect"
    "time"

    "github.com/samber/lo"
    "go.mongodb.org/mongo-driver/bson/primitive"
)

type Employee struct {
    ID               primitive.ObjectID `bson:"_id,omitempty" json:"id,omitempty" validate:"required"`
    FirstName        string             `bson:"firstName,omitempty" json:"firstName,omitempty" validate:"required"`
    LastName         string             `bson:"lastName,omitempty" json:"lastName,omitempty" validate:"required"`
    MiddleName       string             `bson:"middleName,omitempty" json:"middleName,omitempty"`
    Email            string             `bson:"email,omitempty" json:"email,omitempty" validate:"required,email"`
    CreatedAt        *time.Time         `bson:"createdAt,omitempty" json:"-" validate:"required"`
    UpdatedAt        *time.Time         `bson:"updatedAt,omitempty" json:"-" validate:"required"`
    BVN              string             `bson:"bvn,omitempty" json:"bvn,omitempty"`
    DOB              string             `bson:"dob,omitempty" json:"dob,omitempty"`
    Address          string             `bson:"address,omitempty" json:"address,omitempty"`
    Phone            string             `bson:"phone,omitempty" json:"phone,omitempty"`
    Country          string             `bson:"country,omitempty" json:"country,omitempty"`
    IDNumber         string             `bson:"idNumber,omitempty" json:"idNumber,omitempty"`
    IDType           string             `bson:"idType,omitempty" json:"idType,omitempty"`
    AdditionalIDType string             `bson:"additionalIdType,omitempty" json:"additionalIdType,omitempty"`
    Salary           float64            `bson:"salary,omitempty" json:"salary,omitempty" validate:"required"`
    UserID           primitive.ObjectID `bson:"user_id,omitempty" json:"user_id,omitempty" validate:"required"`
    AccountName      string             `bson:"account_name,omitempty" json:"account_name,omitempty" validate:"required"`
    AccountType      string             `bson:"account_type,omitempty" json:"account_type,omitempty" validate:"required"`
    BankName         string             `bson:"bank_name,omitempty" json:"bank_name,omitempty" validate:"required"`
}

Here we have two fields with primitive.ObjectID We will use this to relate the other model to the employee model.

For storing payments made to employees in the system, we add models/disbursment.go file to store it.

package models

import (
    "time"

    "go.mongodb.org/mongo-driver/bson/primitive"
)

type Disbursement struct {
    ID           primitive.ObjectID `bson:"_id,omitempty" json:"id,omitempty" validate:"required"`
    ReceiverID   primitive.ObjectID `bson:"receiver_id,omitempty" json:"receiver_id,omitempty" validate:"required"`
    CreatedAt    *time.Time         `bson:"createdAt,omitempty" json:"-" validate:"required"`
    UpdatedAt    *time.Time         `bson:"updatedAt,omitempty" json:"-" validate:"required"`
    SalaryAmount float64            `bson:"salary_amount,omitempty" json:"salary_amount,omitempty" validate:"required"`
    SenderID     primitive.ObjectID `bson:"sender_id,omitempty" json:"sender_id,omitempty" validate:"required"`
    Status       string             `bson:"status,omitempty" json:"status,omitempty" validate:"required"`
    Payment      Payment            `bson:"payment,omitempty" json:"payment,omitempty" validate:"required"`
}

As mentioned earlier, it's important to mask sensitive information such as passwords, createdAt, and updatedAt to ensure this data is not returned when queried. To achieve this, we add code that checks the runtime value of a model, allowing us to access the information and decide which parts of the data to exclude.

So, we add the following snippet to models/user.go.

var (
    userOmitList = []string{
        "Password",
        "CreatedAt",
        "CpdatedAt",
    }
)

func (u *User) Omit() (User, error) {
    copiedUser := *u

    userType := reflect.TypeOf(copiedUser)
    fieldValues := make(map[string]interface{})

    for i := 0; i < userType.NumField(); i++ {
        field := userType.Field(i)
        if !lo.Contains(userOmitList, field.Name) {
            value := reflect.ValueOf(*u).
                FieldByName(field.Name).
                Interface()
            fieldValues[field.Name] = value
        }
    }

    filteredJSON, err := json.Marshal(fieldValues)
    if err != nil {
        return User{}, err
    }

    var filteredUser User
    err = json.Unmarshal(filteredJSON, &filteredUser)
    if err != nil {
        return User{}, err
    }

    filteredUser.ID = u.ID
    return filteredUser, nil
}

And in models/employee.go add

var (
    employeeOmitList = []string{
        "CreatedAt",
        "UpdatedAt",
    }
)

func (e *Employee) Omit() (Employee, error) {
    copiedEmployee := *e

    employeeType := reflect.TypeOf(copiedEmployee)
    fieldValues := make(map[string]interface{})

    for i := 0; i < employeeType.NumField(); i++ {
        field := employeeType.Field(i)
        if !lo.Contains(employeeOmitList, field.Name) {
            value := reflect.ValueOf(*e).
                FieldByName(field.Name).
                Interface()
            fieldValues[field.Name] = value
        }
    }

    filteredJSON, err := json.Marshal(fieldValues)
    if err != nil {
        return Employee{}, err
    }

    var filteredEmployee Employee
    err = json.Unmarshal(filteredJSON, &filteredEmployee)
    if err != nil {
        return Employee{}, err
    }

    filteredEmployee.ID = e.ID
    return filteredEmployee, nil
}

Having reached this point, we can design a repository to help us manage these models.

Repository 🗃️

Using our model is very important. To avoid repeating our code in many places, a repository is needed to wrap around the model and give access to the database collection, in this case, MongoDB.

Let's create a new folder repository/repository.go and in this case, to keep the article short and precise you can check out the actual implementation of the repository code.

I generated this with GPT-4 but had to make a lot of changes to get it working . You can check out here.

package repository

import (
    "context"
    "yc-backend/models"

    "go.mongodb.org/mongo-driver/bson"
    "go.mongodb.org/mongo-driver/bson/primitive"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

type Repositories struct {
    User         Repository[models.User]
    Employee     Repository[models.Employee]
    Disbursement Repository[models.Disbursement]
}

func InitRepositories(db *mongo.Database) *Repositories {
    // register all collection here so we can provide via gin.context
    userRepo := NewRepository[models.User](db.Collection("users"))
    employeeRepo := NewRepository[models.Employee](db.Collection("employees"))
    disbursementRepo := NewRepository[models.Disbursement](db.Collection("disbursement"))
    return &Repositories{
        User:         userRepo,
        Employee:     employeeRepo,
        Disbursement: disbursementRepo,
    }
}

// IRepository defines the methods that a repository must implement.
type IRepository[T comparable] interface {
    Create(ctx context.Context, document T) (any, error)
    FindOneById(ctx context.Context, id primitive.ObjectID) (*T, error)
    FindOne(ctx context.Context, filter bson.D) (*T, error)
    FindMany(ctx context.Context, filter bson.D) ([]T, error)
    UpdateOneById(ctx context.Context, id primitive.ObjectID, document T) error
    UpdateMany(ctx context.Context, filter bson.D, document T) error
    DeleteById(ctx context.Context, id primitive.ObjectID) error
    DeleteMany(ctx context.Context, filter bson.D) error
    Count(ctx context.Context, filter bson.D) (int64, error)
    CreateIndex(ctx context.Context, keys bson.D, opt *options.IndexOptions) (string, error)
    EstimatedDocumentCount(ctx context.Context) (int64, error)
    Aggregate(ctx context.Context, pipeline mongo.Pipeline, opts ...*options.AggregateOptions) ([]*T, error)
}

// Repository is a MongoDB repository implementation.
type Repository[T comparable] struct {
    collection *mongo.Collection // MongoDB collection
}

// implementation details

Here we declare a few things our repository will support writing and reading from the database.

People consider the above pattern an anti-pattern in Golang, but I don't mind. I also like breaking rules sometimes to see if they will work.🫠.

Middleware Via Injection 💉

Middleware always seems like a made-up term. The funny thing is, it's simply a way to process incoming requests or outgoing responses before or after the main handler does its job.

For our application, we need to inject some components into the Gin context so we can have access to it in our route handlers. This is done so we can avoid passing them via a higher-order function.

Create a file in common/middlware.go the same place where we have ours config.go and add the code. Later on, we will add an authorization middleware there when we start dealing with logging-in users.

package common

import (
    "errors"
    "net/http"
    "strings"
    "<project-name>/internals"
    "<project-name>/repository"
    "<project-name>/utils"

    "github.com/gin-gonic/gin"
    "github.com/google/uuid"
    "go.mongodb.org/mongo-driver/bson/primitive"
    "go.mongodb.org/mongo-driver/mongo"
)

const (
    configContextKey     = "_yc_config"
    requestIdContextKey  = "_yc_request_id"
    dbContextKey         = "__yc_db"
    repositoryContextKey = "__yc_repo"
    poolContextKey       = "__yc_pool"
    loggerContextKey     = "__yc_logger"
    UserKey              = "__user"
)

func RequestIdFromCtx(ctx *gin.Context) string {
    return ctx.MustGet(requestIdContextKey).(string)
}

func ConfigFromCtx(ctx *gin.Context) *Config {
    return ctx.MustGet(configContextKey).(*Config)
}

func DbFromCtx(ctx *gin.Context) *mongo.Client {
    return ctx.MustGet(dbContextKey).(*mongo.Client)
}

func ReposFromCtx(ctx *gin.Context) *repository.Repositories {
    return ctx.MustGet(repositoryContextKey).(*repository.Repositories)
}

func LoggerFromCtx(ctx *gin.Context) internals.Logger {
    return ctx.MustGet(loggerContextKey).(internals.Logger)
}

func AddConfigMiddleware(cfg *Config) gin.HandlerFunc {
    return func(ctx *gin.Context) {
        ctx.Set(configContextKey, cfg)
        ctx.Next()
    }
}

func AddLoggerMiddleware(logger internals.Logger) gin.HandlerFunc {
    return func(ctx *gin.Context) {
        ctx.Set(loggerContextKey, logger)
        ctx.Next()
    }
}

func AddReposToMiddleware(db *mongo.Client) gin.HandlerFunc {
    return func(ctx *gin.Context) {
        databaseName := ctx.MustGet(configContextKey).(*Config).MongoDB.DatabaseName
        collection := db.Database(databaseName)
        repos := repository.InitRepositories(collection)
        ctx.Set(repositoryContextKey, repos)
        ctx.Set(dbContextKey, db)
        ctx.Next()
    }
}

func AddRequestIDMiddleware() gin.HandlerFunc {
    return func(ctx *gin.Context) {
        id := uuid.New().String()
        ctx.Set(requestIdContextKey, id)
        ctx.Next()
    }
}

Here's a summary: the constants ending with ContextKey allow us to set values on the context struct, which are passed down to our route handlers. We currently place repository, configuration and logger components in our context and we will retrieve them with the defined keys.

An article should be short right? There is quite a lot of information to digest.

In the next part, we will be connecting the dots with helpers, routes and a payment provider.

You can check the repo for this tutorial here.

I am Caleb and you can reach me on Linkedin or follow me on Twitter. @Soundboax