How to Use n8n Queue Mode to Scale Workers

steps

1. Prepare Your Server Environment: Ensure docker and docker-compose are installed on your server. Install Redis for queue management:

   bashCopy

   sudo apt-get update
   sudo apt-get install redis
   sudo systemctl start redis

2. Install n8n and Configure the Main Instance: Create a docker-compose.yml file for the main n8n instance. Configure Redis and database connections:

   yamlCopy

   version: '3'
   services:
     n8n:
       image: n8nio/n8n
       ports:
         - 5678:5678
       environment:
         - EXECUTIONS_PROCESS=main
         - QUEUE_MODE=true
         - REDIS_HOST=redis
         - DB_TYPE=postgresdb # Adapt if using a different DB
       restart: always
     redis:
       image: redis:alpine
       restart: always

   Start the services:

   bashCopy

   docker-compose up -d

3. Launch Worker Instances: Add worker containers in the same docker-compose.yml to facilitate parallel execution:

   yamlCopy

     worker-1:
       image: n8nio/n8n
       environment:
         - EXECUTIONS_PROCESS=queue
         - N8N_MODE=worker
         - REDIS_HOST=redis
       depends_on:
         - redis
       restart: always
   
     worker-2:
       image: n8nio/n8n
       environment:
         - EXECUTIONS_PROCESS=queue
         - N8N_MODE=worker
         - REDIS_HOST=redis
       depends_on:
         - redis
       restart: always

   Restart Docker to apply changes:

   bashCopy

   docker-compose down
   docker-compose up -d

4. Verify Worker Activity: Access your Redis instance to verify queued jobs:

   bashCopy

   redis-cli
   KEYS *

   Run test workflows to ensure proper execution and monitor resource usage with docker stats.

steps

1. Run a Workflow with Large Data: Configure a workflow to process large numbers of items (e.g., 10,000 rows). Monitor the execution time.

2. Check Parallel Execution: Log into Redis and view active job queues:

   bashCopy

   redis-cli
   LRANGE bull:queue_name:active 0 -1

3. Monitor Resource Utilization: Use the following command to inspect Docker container stats:

   bashCopy

   docker stats

   Confirm that each worker is actively processing jobs.