Building a file management system

This blog post is part of the HNG Internship Stage Zero Task - Backend Track. It is meant to address the task of writing a technical blog article describing a recent, difficult backend problem I had solved, and exactly I solved it. I will also share my journey and aspirations with the HNG Internship.

Introduction:

You might be wondering what the HNG internship is. Let me explain: HNG is considered the most prestigious internship in Nigeria and Africa. This internship attracts thousands of aspiring software developers, product managers, data analysts, product marketers, and QA testers from around Africa, and the program attracts employers who are looking to hire top talent. The HNG Internship is a perfect opportunity for growth, learning, and career development.

The Challenge: Efficient Background Processing.

I recently tasked myself with building out a personal project to improve my Node.js skills. I finally settled on creating a file management system. The features included user authentication via a token, listing all files, uploading new files, changing file permissions, viewing files, generating thumbnails for images, handling large file uploads, and sending welcome emails to new users. This needed to be done without compromising the performance of the main application. The problem was to implement a background job processing system that could handle these tasks efficiently.

To solve this, I used the Bull Queue package. Using Bull for managing background jobs would ensure that file processing and email sending did not block the main thread, keeping the application responsive. I created two Bull queues, one for processing files and another for sending welcome emails.

import Queue from 'bull';
import imageThumbnail from 'image-thumbnail';
import fs from 'fs';
import path from 'path';
import dbClient from './utils/db';

const fileQueue = new Queue('fileQueue');
const FOLDER_PATH = process.env.FOLDER_PATH || '/tmp/files_manager';

fileQueue.process(async (job, done) => {
  const { fileId, userId } = job.data;

  if (!fileId || !userId) {
    throw new Error('Missing fileId or userId');
  }

  const file = await dbClient.filesCollection.findOne({ _id: new ObjectId(fileId), userId: new ObjectId(userId) });

  if (!file) {
    throw new Error('File not found');
  }

  const filePath = path.join(FOLDER_PATH, file.localPath);

  if (!fs.existsSync(filePath)) {
    throw new Error('File not found');
  }

  try {
    const sizes = [500, 250, 100];
    for (const size of sizes) {
      const options = { width: size };
      const thumbnail = await imageThumbnail(filePath, options);
      const thumbnailPath = `${filePath}_${size}`;
      fs.writeFileSync(thumbnailPath, thumbnail);
    }
    done();
  } catch (error) {
    done(error);
  }
});

What did I learn?

building the file management system taught me the Importance of job processing in building a scalable system, By offloading time-consuming tasks to the background queues. This way the main application remains responsive and efficient.

Conclusion:

my experience with building this project not only improved my Node.js and express skills but has reinforced the importance of continuously learning and adapting to new technologies. The HNG Internship offers me a platform to further this growth, allowing collaboration with like-minded peers.