🇺🇸A n8n environment ready for scaling: installation and configuration on Clever Cloud.

n8n, what is it?

Are you familiar with software such as IFTTT or Zapier ?
They are workflow engines that chain tasks together to achieve a final result.
They allow you to automate tasks based on events that occur. In summary, when you perform an action, your workflow engine reacts accordingly.

n8n is also one of these engines. Unlike IFTTT and Zapier, it is an open-source software that you can install on one of your servers.

The main challenge for me with this software was that it was not easy to configure for handling high demand.
Apart from being able to scale it vertically, launching multiple instances simultaneously posed some problems with the database, and it was necessary to manage file sharing on a distributed block storage system.
This was not easy.
Fortunately, the people at n8n.io have solved the problem by introducing the concept of a worker:

As shown in this illustration, you have a main instance that handles the user interface, the creation of your workflows, and listening to incoming webhooks.
Then, it sends workflow execution commands to worker instances via a message bus system. Here, the excellent Redis has been chosen to fulfill this role.
Note that it is also possible to have webhook instances that will listen for events, but this article does not cover this functionality.
I plan to write an article on this feature in the coming months.

In the rest of this article, I propose deploying a main instance and three worker instances on the Clever Cloud PaaS.
We will see that only the workers will be responsible for executing our workflows.

Prerequisites

clever-tools

Make sure you have clever-tools installed on your workstation: Doc

Generating an execution key

We will need to generate a key that will be used by the n8n main instance as well as the workers.
Use the following command to generate it and note it down to use it later

uuidgen

I will refer to it as <VOTRE_CLE_EXECUTION>.

Primary instance

Creating the package.json file and the run.sh script

mkdir /var/tmp/n8n
cd /var/tmp/n8n
git init

Create the package.json file.

{
  "name": "n8n",
  "version": "0.221.1",
  "engines": {
    "node": "16.9"
 },
  "dependencies": {
    "n8n": "0.221.1"
  }
}

Create the run.sh file.

#!/bin/bash
set -x
./node_modules/.bin/n8n start

Add both files to the git repository:

chmod a+x run.sh
git add package.json run.sh
git commit -m "Init n8n launch"

Creating the primary instance

clever create cc-n8n --type node --org <YOUR_ORGA_ID> --region par
clever config update --enable-force-https
clever scale --alias cc-n8n --flavor XS
clever scale --alias cc-n8n --build-flavor S
clever addon create redis-addon cc-n8n-redis-queue --link cc-n8n --org <YOUR_ORGA_ID> --region par --plan s_mono --yes
clever addon create postgresql-addon cc-n8n-db --link cc-n8n --org <YOUR_ORGA_ID> --region par --plan xs_sml --yes
clever addon create fs-bucket n8n-fs-data --plan s --link cc-n8n --org <YOUR_ORGA_ID> --region par --yes
clever env set PORT 8080
clever env set N8N_VERSION 0.221.1
clever env set N8N_PORT `clever env | grep -w PORT| awk -F = '/PORT/ { print $2}'|sed s/\"//g`
clever env set N8N_PROTOCOL https
clever env set N8N_HOST `clever domain|awk '{print $1}'`
clever env set WEBHOOK_TUNNEL_URL "https://`clever domain|awk '{print $1}'`/"
clever env set VUE_APP_URL_BASE_APP "https://`clever domain|awk '{print $1}'`/"
clever env set N8N_ENCRYPTION_KEY <VOTRE_CLE_EXECUTION>
clever env set EXECUTIONS_DATA_SAVE_MANUAL_EXECUTIONS true
clever env set EXECUTIONS_DATA_SAVE_ON_ERROR all
clever env set N8N_USER_FOLDER /app/n8n-data/
clever env set EXECUTIONS_MODE queue
clever env set QUEUE_HEALTH_CHECK_ACTIVE true
clever env set QUEUE_BULL_REDIS_HOST `clever env | awk -F = '/REDIS_HOST/ { print $2}'|sed s/\"//g`
clever env set QUEUE_BULL_REDIS_PORT `clever env | awk -F = '/REDIS_PORT/ { print $2}'|sed s/\"//g`
clever env set QUEUE_BULL_REDIS_PASSWORD `clever env | awk -F = '/REDIS_PASSWORD/ { print $2}'|sed s/\"//g`
clever env set DB_TYPE postgresdb
clever env set DB_POSTGRESDB_DATABASE `clever env | awk -F = '/POSTGRESQL_ADDON_DB/ { print $2}'|sed s/\"//g`
clever env set DB_POSTGRESDB_HOST `clever env | awk -F = '/POSTGRESQL_ADDON_HOST/ { print $2}'|sed s/\"//g`
clever env set DB_POSTGRESDB_PORT `clever env | awk -F = '/POSTGRESQL_ADDON_PORT/ { print $2}'|sed s/\"//g`
clever env set DB_POSTGRESDB_USER `clever env | awk -F = '/POSTGRESQL_ADDON_USER/ { print $2}'|sed s/\"//g`
clever env set DB_POSTGRESDB_PASSWORD `clever env | awk -F = '/POSTGRESQL_ADDON_PASSWORD/ { print $2}'|sed s/\"//g`
clever env set GENERIC_TIMEZONE "Europe/Paris"
clever env set CC_FS_BUCKET /n8n-data:`clever env | awk -F = '/BUCKET_HOST/ { print $2}'|sed s/\"//g`
clever env set CC_RUN_COMMAND "./run.sh"

Deploying the application

Run the deployment command:

clever deploy

Workers instances

We are going to deploy 3 n8n worker instances to distribute the load across these workers.

Creating the package.json file and the run.sh script

mkdir /var/tmp/n8n-worker
cd /var/tmp/n8n-worker
git init

Create the package.json file.

{
  "name": "n8n",
  "version": "0.221.1",
  "engines": {
    "node": "16.9"
 },
  "dependencies": {
    "n8n": "0.221.1"
  }
}

Create the run.sh file.

#!/bin/bash
set -x
./node_modules/.bin/n8n worker

Add both files to the git repository:

chmod a+x run.sh
git add package.json run.sh
git commit -m "Init n8n worker launch"

Creating the worker instance

clever create cc-n8n-worker --type node --org <YOUR_ORGA_ID> --region par
clever config update --enable-force-https
clever scale --alias cc-n8n-worker --flavor XS --min-instances 3 --max-instances 3 
clever scale --alias cc-n8n-worker --build-flavor S
clever service link-addon cc-n8n-db
clever service link-addon cc-n8n-redis-queue
clever service link-addon n8n-fs-data
clever env set PORT 8080
clever env set N8N_VERSION 0.221.1
clever env set QUEUE_HEALTH_CHECK_PORT `clever env | grep -w PORT| awk -F = '/PORT/ { print $2}'|sed s/\"//g`
clever env set N8N_HOST `clever domain|awk '{print $1}'`
clever env set N8N_USER_FOLDER /app/n8n-data/
clever env set N8N_ENCRYPTION_KEY <VOTRE_CLE_EXECUTION>
clever env set EXECUTIONS_MODE queue
clever env set QUEUE_HEALTH_CHECK_ACTIVE true
clever env set QUEUE_BULL_REDIS_HOST `clever env | awk -F = '/REDIS_HOST/ { print $2}'|sed s/\"//g`
clever env set QUEUE_BULL_REDIS_PORT `clever env | awk -F = '/REDIS_PORT/ { print $2}'|sed s/\"//g`
clever env set QUEUE_BULL_REDIS_PASSWORD `clever env | awk -F = '/REDIS_PASSWORD/ { print $2}'|sed s/\"//g`
clever env set DB_TYPE postgresdb
clever env set DB_POSTGRESDB_DATABASE `clever env | awk -F = '/POSTGRESQL_ADDON_DB/ { print $2}'|sed s/\"//g`
clever env set DB_POSTGRESDB_HOST `clever env | awk -F = '/POSTGRESQL_ADDON_HOST/ { print $2}'|sed s/\"//g`
clever env set DB_POSTGRESDB_PORT `clever env | awk -F = '/POSTGRESQL_ADDON_PORT/ { print $2}'|sed s/\"//g`
clever env set DB_POSTGRESDB_USER `clever env | awk -F = '/POSTGRESQL_ADDON_USER/ { print $2}'|sed s/\"//g`
clever env set DB_POSTGRESDB_PASSWORD `clever env | awk -F = '/POSTGRESQL_ADDON_PASSWORD/ { print $2}'|sed s/\"//g`
clever env set GENERIC_TIMEZONE "Europe/Paris"
clever env set CC_FS_BUCKET /n8n-data:`clever env | awk -F = '/BUCKET_HOST/ { print $2}'|sed s/\"//g`
clever env set CC_RUN_COMMAND "./run.sh"

Deploying the application

Run the deployment command:

clever deploy

Let's play a little

References

TITLE	URL
Blog post by Loic Tosser: N8N on Clever Cloud	https://blog.kalvad.com/n8n-on-clever-cloud
Repo Git: n8n-example	https://github.com/CleverCloud/n8n-example
n8n Documentation: Queue mode	https://docs.n8n.io/hosting/scaling/queue-mode/