Go Queue - Flexible Background Job Processing for Go Applications

Introduction

In modern application development, background job processing is essential for handling time-consuming tasks without blocking user interactions. Whether you're sending emails, processing images, or generating reports, a reliable queue system is crucial.

Today, I'm excited to introduce Go Queue, a flexible job queue library for Go that supports multiple database backends while maintaining a consistent API. Go Queue is designed to be simple to integrate yet powerful enough for production workloads.

The Problem Go Queue Solves

Most queue libraries in the Go ecosystem are tightly coupled to specific storage backends. This creates challenges when:

• Your team has expertise with a particular database

• You need to migrate between storage solutions

• You're building applications for different environments

GoQueue addresses these issues by providing a unified interface across Redis, PostgreSQL, MySQL, AWS SQS, and in-memory storage.

Getting Started with GoQueue

Getting started with GoQueue is straightforward. First, install the package

go get github.com/saravanasai/goqueue

Step 1: Define Your Job

Create a job type that implements the goqueue.Job interface

package jobs

import (
    "context"
    "fmt"

    "github.com/saravanasai/goqueue"
)

// EmailJob implements the job.Job interface
type EmailJob struct {
    To      string `json:"to"`
    Subject string `json:"subject"`
}

// Process implements the job.Job interface
func (e *EmailJob) Process(ctx context.Context) error {
    fmt.Printf("Sending email to %s: %s\n", e.To, e.Subject)
    return nil
}

// Register job type for serialization
func init() {
    goqueue.RegisterJob("EmailJob", func() goqueue.Job {
        return &EmailJob{}
    })
}

Step 2: Configure and Initialize Your Queue

Choose a database backend and configure your queue

import (
    "github.com/saravanasai/goqueue"
    "github.com/saravanasai/goqueue/config"
    "github.com/saravanasai/goqueue/middleware"
)

// Configure with MySQL
cfg := config.NewMySQLConfig("root:root@tcp(localhost:3306)/mydb?parseTime=true")

// Add retry configuration
cfg = cfg.WithMaxRetryAttempts(5)
      .WithExponentialBackoff(true)
      .WithRetryDelay(1 * time.Minute)

// Add logging middleware
loggingMiddleware := middleware.LoggingMiddleware(logger.NewZapLogger())
cfg = cfg.WithMiddleware(loggingMiddleware)

// Create the queue
q, err := goqueue.NewQueueWithDefaults("emails", cfg)
if err != nil {
    log.Fatalf("Failed to create queue: %v", err)
}

Step 3: Dispatch Jobs

Send jobs to the queue for processing

// Create an email job
emailJob := &jobs.EmailJob{
    To:      "user@example.com",
    Subject: "Welcome to GoQueue!",
}

// Dispatch immediately
if err := q.Dispatch(emailJob); err != nil {
    log.Fatalf("Failed to dispatch job: %v", err)
}

// Or schedule for later
if err := q.DispatchWithDelay(emailJob, 5*time.Minute); err != nil {
    log.Fatalf("Failed to schedule job: %v", err)
}

Step 4: Process Jobs with Workers

Start worker processes to handle the queued jobs

// Start 2 concurrent workers
ctx := context.Background()
err = q.StartWorkers(ctx, 2)
if err != nil {
    log.Fatalf("Failed to start workers: %v", err)
}

// Keep the application running
var wg sync.WaitGroup
wg.Add(1)
wg.Wait()

Key Features

Go Queue isn't just about database flexibility. It includes features essential for production systems

• Automatic retries with configurable backoff

• Dead letter queues for failed jobs

• Middleware support for custom processing logic

• Metrics collection for monitoring

• Non-blocking retry mechanism

The library handles common edge cases like job serialization, connection pooling, and transaction management, so you can focus on your application logic.

When to Use Go Queue

Go Queue is particularly valuable when:

1. You need a queue system that works with your existing infrastructure

2. You want the flexibility to switch databases as your application evolves

3. You're building for multiple environments (development, staging, production)

4. You need comprehensive retry logic and error handling

Conclusion

Building reliable background processing systems shouldn't require locking yourself into specific technologies. Go Queue provides the flexibility modern Go applications need while maintaining the reliability production systems demand.

The project is actively maintained and open to contributions. Whether you're handling thousands of jobs per second or just need occasional background processing, Go Queue scales to meet your needs.

Try it out today at github.com/saravanasai/goqueue and let me know what you think!