Complete Multer File Upload Tutorial: Build Adaptive Memory & Disk Storage System in Node.js (2025)

File uploads are a common requirement in many web applications. Multer is a Node.js middleware specifically designed to simplify the handling of multipart/form-data, primarily used for uploading files. This guide will walk you through setting up multer to manage file uploads, demonstrating how to use both memory storage (for speed) and disk storage (for stability and larger files), and automatically switching between them based on system resources.

Prerequisites

Before you begin, ensure you have the following installed:

• Node.js: Download and Install Node.js

• npm (Node Package Manager): Usually comes bundled with Node.js.

• Express.js: A fast, unopinionated, minimalist web framework for Node.js.

• Multer: A Node.js middleware for handling multipart/form-data.

• Dotenv: A zero-dependency module that loads environment variables from a .env file.

• GoogleGenAI: The official Node.js client library for the Google Gemini API.

• CORS: Node.js CORS middleware.

Project Setup

Let's start by setting up a new Express.js project.

1. Create a Project Directory:

    mkdir multer-upload-guide
    cd multer-upload-guide
   

2. Initialize Node.js Project:

    npm init -y
   

3. Install Dependencies:

    npm install express multer cookie-parser morgan dotenv @google/genai cors debug
   

4. Create Essential Files:

   Create the following files in your project's root directory:

   • app.js (or index.js): This will contain your main Express application logic.

   • .env: To store your environment variables (e.g., Google API Key).

   • public directory: For static assets (optional, but good practice).

   • uploads directory: This directory will be created programmatically to store uploaded files.

Understanding Multer Storage Options

Multer provides two primary ways to store uploaded files:

1. Disk Storage

Saves files directly to the server's file system.

• Pros: Memory efficient, persistent, handles large files

• Cons: Slower I/O operations, requires disk space management

2. Memory Storage

Stores files in server RAM as Buffer objects.

• Pros: Faster processing, direct Buffer access

• Cons: Memory consumption, not persistent, Out of Memory risk with large files

3. Adaptive Storage (Recommended)

Combines both approaches by choosing storage type based on:

• File size

• Current memory usage

• System resources

This prevents memory exhaustion while maintaining speed for small files.

In summary, the adaptive storage strategy allows your application to be performant for common use cases while maintaining high availability and resilience when faced with resource-intensive operations.

Step-by-Step Implementation

Now, let's dive into the code for app.js.

1. Import Required Modules

At the top of your app.js file, import all the necessary modules:

const express = require('express');
const path = require('path');
const cookieParser = require('cookie-parser');
const logger = require('morgan');
const multer = require('multer');
const fs = require('fs');
const cors = require('cors');
const dotenv = require('dotenv');
const { GoogleGenAI } = require('@google/genai');
const os = require('os');

2. Configure Environment Variables

Load environment variables from your .env file. Create a .env file in your project root and add your Google API key:

GOOGLE_API_KEY=YOUR_GOOGLE_GEMINI_API_KEY

Then, in app.js:

dotenv.config();

3. Initialize Express Application

const app = express();

4. Configure CORS for Security

Cross-Origin Resource Sharing (CORS) restricts web pages from making requests to a different domain than the one that served the web page. For security, it's crucial to define allowed origins.

const allowedOrigins = [
  'http://localhost:3000',
  // Add other trusted front-end origins here
];

app.use(cors({
  origin: function (origin, callback) {
    // We restrict origins to prevent unauthorized access to our API.
    // Allowing no origin enables certain testing tools and server-to-server calls.
    if (!origin) return callback(null, true);
    if (allowedOrigins.includes(origin)) {
      return callback(null, true);
    } else {
      return callback(new Error('Not allowed by CORS'));
    }
  },
  methods: ['GET', 'POST', 'OPTIONS'],
  allowedHeaders: ['Content-Type', 'Authorization'],
  credentials: true
}));

5. Set Payload Limits

For handling larger file uploads, it's essential to increase the payload limits for JSON and URL-encoded bodies.

// Payload limits to accommodate larger file uploads without rejection.
app.use(express.json({ limit: '50mb' }));
app.use(express.urlencoded({ extended: true, limit: '50mb' }));

6. Ensure Upload Directory Exists

Before multer can store files, the destination directory must exist. This code checks for the existence of the uploads directory and creates it if it doesn't.

const uploadsDir = path.join(__dirname, 'uploads');
// uploads directory to ensure files have a place to be saved.
if (!fs.existsSync(uploadsDir)) {
  fs.mkdirSync(uploadsDir, { recursive: true });
}

7. Dynamic Storage Selection Logic

This is a core part of optimizing resource usage. We'll define a MEMORY_THRESHOLD and a function shouldUseMemoryStorage that dynamically decides whether to use memory storage or disk storage based on available memory and file size.

// Threshold to decide when to use disk storage to prevent server crashes.
const MEMORY_THRESHOLD = 0.8; // 80% usage
let useMemoryStorage = true; // Tracks the chosen storage for the current request

const shouldUseMemoryStorage = (fileSize) => {
  try {
    const freeMem = os.freemem();
    const totalMem = os.totalmem();
    const memoryUsageRatio = 1 - (freeMem / totalMem);

    // Prefer disk storage for large files (e.g., > 15MB) or when memory is scarce to avoid crashes.
    if (fileSize > 15 * 1024 * 1024 || memoryUsageRatio > MEMORY_THRESHOLD) {
      console.log('Using disk storage due to file size or memory pressure');
      return false;
    }
    return true;
  } catch (error) {
    // Default to disk storage if we can't get memory info, ensuring the app remains stable.
    console.error('Error checking memory:', error);
    return false;
  }
};

• MEMORY_THRESHOLD: If system memory usage exceeds this percentage, disk storage will be preferred.

• shouldUseMemoryStorage(fileSize): This function calculates the current memory usage ratio and checks the incoming fileSize. It returns true for memory storage if the file is small and memory usage is below the threshold, otherwise false for disk storage. This helps prevent "Out Of Memory" (OOM) errors for large files or under heavy load.

8. Define Multer Storage Options

multer provides two main storage engines: diskStorage and memoryStorage.

const storage = multer.diskStorage({
  destination: function (req, file, cb) {
    // Files are saved to disk for persistence and to free up server memory.
    cb(null, uploadsDir);
  },
  filename: function (req, file, cb) {
    // timestamp to filenames to prevent conflicts when multiple files have the same name.
    cb(null, Date.now() + '-' + file.originalname);
  }
});

const memoryStorage = multer.memoryStorage(); // For faster, in-memory processing

9. Implement Adaptive Upload Middleware

This custom middleware, adaptiveUpload, dynamically configures multer to use either memoryStorage or diskStorage based on the shouldUseMemoryStorage logic.

const adaptiveUpload = (req, res, next) => {
  const contentLength = req.headers['content-length'] ?
                        parseInt(req.headers['content-length']) : 0;
   // dynamically choose storage type to optimize between speed (memory) 
   // and stability (disk) based on file size and server resources.
  useMemoryStorage = shouldUseMemoryStorage(contentLength);
  const upload = multer({
    storage: useMemoryStorage ? memoryStorage : storage,
    limits: {
      fileSize: 50 * 1024 * 1024, // Limit file size to 50MB to prevent abuse.
      files: 1 // Allow only one file per upload request.
    }
  }).single('file'); // 'file' matches the name attribute in the HTML form's input.

  upload(req, res, function (err) {
    // Return specific error messages to the client for better debugging and user feedback.
    if (err instanceof multer.MulterError) {
      return res.status(400).json({ error: `Multer error: ${err.message}` });
    } else if (err) {
      return res.status(500).json({ error: `Unknown error: ${err.message}` });
    }
    next();
  });
};

• contentLength: We retrieve the Content-Length header to estimate the file size before multer processes it, allowing us to make an informed decision about storage type.

• .single('file'): This indicates that you are expecting a single file upload with the field name file. If you expect multiple files, you would use .array() or .fields().

• Error Handling: Basic error handling for multer specific errors and other general errors during the upload process.

10. Initialize GoogleGenAI

Load your Google API key and initialize the Google Gemini AI client.

const ai = new GoogleGenAI({ apiKey: process.env.GOOGLE_API_KEY });

11. General Express Middleware

Include standard Express middleware for logging, cookie parsing, and serving static files.

// morgan for development logging to see request details.
app.use(logger('dev'));
// parse cookies to enable session management or other cookie-dependent features.
app.use(cookieParser());
// serve static files from the 'public' directory so the browser can access frontend assets.
app.use(express.static(path.join(__dirname, 'public')));

12. File Upload Endpoint (/upload)

This is the main API endpoint where files will be uploaded. It uses the adaptiveUpload middleware before processing the file.

app.post('/upload', adaptiveUpload, async (req, res) => {
  try {
    // Verify a file was received to prevent errors from empty requests.
    if (!req.file) {
      return res.status(400).json({ error: 'No file uploaded' });
    }

    let tempFilePath;
    const { mimetype, originalname } = req.file;

    if (useMemoryStorage) {
      // memory to disk temporarily because many external APIs require a file path.
      tempFilePath = path.join(uploadsDir, `temp-${Date.now()}-${originalname}`);
      fs.writeFileSync(tempFilePath, req.file.buffer);
    } else {
      // If disk storage was used, the file is already available on disk.
      tempFilePath = req.file.path;
    }

    const uploadPromise = ai.files.upload({
      file: tempFilePath,
      config: { mimeType: mimetype, displayName: originalname },
    });
    const timeoutPromise = new Promise((_, reject) => {
      // set a timeout to prevent the request from hanging if the external API is slow.
      setTimeout(() => reject(new Error('Gemini API upload timeout')), 180000);
    });

    const uploadedFile = await Promise.race([uploadPromise, timeoutPromise]);
    // Delete the temporary file to prevent disk space from filling up.
    fs.unlinkSync(tempFilePath);

    res.json({
      message: 'File uploaded to Gemini Files API',
      metadata: uploadedFile,
    });
  } catch (err) {
    console.error('Upload error:', err);
    res.status(500).json({
      error: 'File upload failed',
      details: err.message,
      storageType: useMemoryStorage ? 'memory' : 'disk'
    });
    // clean up temporary files on disk, even if an error occurs.
    if (req.file && !useMemoryStorage && req.file.path) {
      try { fs.unlinkSync(req.file.path); } catch (e) { console.error('Failed to clean up file:', e); }
    }
  }
});

• File Handling:

  • If memoryStorage was used, the file content is in req.file.buffer. We write this buffer to a temporary file on disk because external APIs (like Gemini Files API) often require a file path.

  • If diskStorage was used, the file is already saved to req.file.path.

• Google Gemini API Upload: The file is then uploaded to the Google Gemini Files API.

• Timeout: A timeout is implemented to prevent the request from hanging indefinitely if the external API is slow or unresponsive.

• Cleanup: Crucially, fs.unlinkSync(tempFilePath) ensures that temporary files created for memory-stored uploads, or the original disk-stored files in case of errors, are always deleted after processing to prevent disk space exhaustion.

13. Export the Express Application

module.exports = app; // Exporting app for use by the server entry point

Running the Application

To run your Express application, you'll typically have a www or server.js file that imports app.js and starts the server.

Example bin/www (or server.js) file:

#!/usr/bin/env node

/**
 * Module dependencies.
 */

var app = require('../app');
var debug = require('debug')('multer-upload-guide:server');
var http = require('http');

/**
 * Get port from environment and store in Express.
 */

var port = normalizePort(process.env.PORT || '3000');
app.set('port', port);

/**
 * Create HTTP server.
 */

var server = http.createServer(app);

/**
 * Listen on provided port, on all network interfaces.
 */

server.listen(port);
server.on('error', onError);
server.on('listening', onListening);

/**
 * Normalize a port into a number, string, or false.
 */

function normalizePort(val) {
  var port = parseInt(val, 10);

  if (isNaN(port)) {
    // named pipe
    return val;
  }

  if (port >= 0) {
    // port number
    return port;
  }

  return false;
}

/**
 * Event listener for HTTP server "error" event.
 */

function onError(error) {
  if (error.syscall !== 'listen') {
    throw error;
  }

  var bind = typeof port === 'string'
    ? 'Pipe ' + port
    : 'Port ' + port;

  // handle specific listen errors with friendly messages
  switch (error.code) {
    case 'EACCES':
      console.error(bind + ' requires elevated privileges');
      process.exit(1);
      break;
    case 'EADDRINUSE':
      console.error(bind + ' is already in use');
      process.exit(1);
      break;
    default:
      throw error;
  }
}

/**
 * Event listener for HTTP server "listening" event.
 */

function onListening() {
  var addr = server.address();
  var bind = typeof addr === 'string'
    ? 'pipe ' + addr
    : 'port ' + addr.port;
  debug('Listening on ' + bind);
}

To run your server, execute:

node bin/www

Or, if you prefer using nodemon for automatic restarts during development:

npm install -g nodemon
nodemon bin/www

Testing the Upload Endpoint

You can test the upload endpoint using tools like Postman, Insomnia, or a simple HTML form.

Example HTML Form for Testing:

Create an index.html file in your public directory:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>File Upload</title>
</head>
<body>
    <h1>Upload a File</h1>
    <form action="http://localhost:3000/upload" method="post" enctype="multipart/form-data">
        <input type="file" name="file" required>
        <button type="submit">Upload</button>
    </form>
    <div id="response"></div>

    <script>
        const form = document.querySelector('form');
        const responseDiv = document.getElementById('response');

        form.addEventListener('submit', async (e) => {
            e.preventDefault();
            const formData = new FormData(form);
            try {
                const res = await fetch(form.action, {
                    method: 'POST',
                    body: formData,
                });
                const data = await res.json();
                responseDiv.textContent = JSON.stringify(data, null, 2);
                if (!res.ok) {
                    responseDiv.style.color = 'red';
                } else {
                    responseDiv.style.color = 'green';
                }
            } catch (error) {
                responseDiv.textContent = 'Error: ' + error.message;
                responseDiv.style.color = 'red';
            }
        });
    </script>
</body>
</html>

Access this HTML file in your browser via http://localhost:3000/index.html (assuming your server is running on port 3000) and try uploading different sized files. Observe the console logs to see the memory usage and which storage type was chosen.

Key Benefits

• Automatic optimization: Small files use faster memory storage

• Stability: Large files or high memory usage triggers disk storage

• Resource monitoring: Built-in memory usage tracking

• Robust cleanup: Prevents disk space issues

• Error handling: Comprehensive error management

Best Practices

1. Monitor memory usage regularly in production

2. Set appropriate file size limits based on your server resources

3. Implement file type validation for security

4. Use cloud storage for production deployments

By following this detailed guide, you can confidently implement a robust and efficient file upload system in your Express.js applications using multer, leveraging both memory and disk storage for optimal performance and stability.