Laravel Queue System: Complete Guide

Laravel Queue System: Complete Basic to Advanced Guide

Laravel's queue system allows you to defer the processing of time-consuming tasks, such as sending emails or processing files, to a later time. This significantly improves your application's response time. In this comprehensive guide, we'll cover everything from basic setup to advanced queue techniques.

Table of Contents

• Introduction to Laravel Queues
• Setting Up Queues
• Creating Jobs
• Dispatching Jobs
• Running Queue Workers
• Queue Connections
• Handling Failed Jobs
• Job Middleware
• Job Chaining
• Batch Processing
• Laravel Horizon
• Best Practices

Introduction to Laravel Queues

Queues allow you to defer time-consuming tasks to be executed in the background, rather than making your users wait for them to complete during their web request. Common use cases include:

• Sending emails
• Processing uploaded files
• Performing database maintenance
• Generating reports
• Calling external APIs

Note: Queues are different from Laravel's scheduler. The scheduler is for running tasks at specific times (like cron jobs), while queues are for deferring tasks to be processed as soon as possible in the background.

Setting Up Queues

Configuration

Laravel supports several queue backends out of the box:

• Database
• Redis
• Amazon SQS
• Beanstalkd
• Sync (for local development, runs jobs immediately)

Configure your queue connection in config/queue.php:

'default' => env('QUEUE_CONNECTION', 'sync'),

'connections' => [
    'sync' => [
        'driver' => 'sync',
    ],

    'database' => [
        'driver' => 'database',
        'table' => 'jobs',
        'queue' => 'default',
        'retry_after' => 90,
    ],

    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => env('REDIS_QUEUE', 'default'),
        'retry_after' => 90,
        'block_for' => null,
    ],
],

Database Driver Setup

If using the database driver, you'll need to create the jobs table:

php artisan queue:table
php artisan migrate

Redis Driver Setup

For Redis, ensure you have the Redis PHP extension and Laravel's Redis package installed:

composer require predis/predis

Configure your Redis connection in config/database.php.

Creating Jobs

Jobs represent the tasks that will be queued. Generate a new job class:

php artisan make:job ProcessPodcast

This creates a file at app/Jobs/ProcessPodcast.php:

<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class ProcessPodcast implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct()
    {
        //
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        // Process the podcast...
    }
}

Job Structure

• ShouldQueue interface marks the job as queueable
• Dispatchable provides the dispatch method
• InteractsWithQueue allows job to interact with the queue
• Queueable provides configuration methods like onQueue
• SerializesModels safely serializes Eloquent models

Passing Data to Jobs

You can pass data to the job through its constructor:

public function __construct(public Podcast $podcast)
{
}

Warning: Be careful when passing large objects to jobs as they will be serialized and stored in your queue backend. Consider passing IDs and re-fetching the data in the job instead.

Dispatching Jobs

Dispatch jobs from anywhere in your application:

// Basic dispatch
ProcessPodcast::dispatch();

// With constructor parameters
ProcessPodcast::dispatch($podcast);

// Delay execution by 10 minutes
ProcessPodcast::dispatch($podcast)
    ->delay(now()->addMinutes(10));

// Specify a queue
ProcessPodcast::dispatch($podcast)
    ->onQueue('processing');

Dispatching From Controllers

public function store(Request $request)
{
    $podcast = Podcast::create($request->validated());
    
    ProcessPodcast::dispatch($podcast);
    
    return redirect()->route('podcasts.index');
}

Dispatching After Response

To dispatch after the HTTP response is sent to the browser:

public function store(Request $request)
{
    $podcast = Podcast::create($request->validated());
    
    ProcessPodcast::dispatchAfterResponse($podcast);
    
    return redirect()->route('podcasts.index');
}

Running Queue Workers

Workers process jobs from the queue. Start a worker:

php artisan queue:work

Worker Options

Option	Description
--queue=high,default	Process specific queues in order
--tries=3	Number of times to attempt a job
--timeout=60	Maximum seconds a job can run
--sleep=3	Seconds to sleep when no jobs available
--once	Only process the next job
--stop-when-empty	Stop when the queue is empty

Running Workers in Production

For production, use a process manager like Supervisor to keep workers running:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/artisan queue:work --sleep=3 --tries=3
autostart=true
autorestart=true
user=forge
numprocs=8
redirect_stderr=true
stdout_logfile=/path/to/worker.log

Queue Priorities

Process high priority jobs first:

php artisan queue:work --queue=high,default,low

Queue Connections

Laravel supports multiple queue connections. Configure them in config/queue.php:

'connections' => [
    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => '{default}',
        'retry_after' => 90,
    ],
    
    'sqs' => [
        'driver' => 'sqs',
        'key' => env('AWS_ACCESS_KEY_ID'),
        'secret' => env('AWS_SECRET_ACCESS_KEY'),
        'prefix' => env('SQS_PREFIX'),
        'queue' => env('SQS_QUEUE'),
        'region' => env('AWS_DEFAULT_REGION'),
    ],
],

Dispatch to a specific connection:

ProcessPodcast::dispatch($podcast)
    ->onConnection('sqs');

Handling Failed Jobs

When a job fails, it will be logged in the failed_jobs table. First, create the table:

php artisan queue:failed-table
php artisan migrate

Configuring Failure Handling

Define a failed method on your job class:

public function failed(Throwable $exception)
{
    // Notify admin of failure
}

Retrying Failed Jobs

View failed jobs:

php artisan queue:failed

Retry a failed job:

php artisan queue:retry 5  # Retry job with ID 5
php artisan queue:retry all # Retry all failed jobs

Delete a failed job:

php artisan queue:forget 5

Job Middleware

Middleware allows you to wrap custom logic around job execution. Create middleware:

php artisan make:middleware RateLimited

Implement the handle method:

public function handle($job, $next)
{
    Redis::throttle('key')
        ->block(0)->allow(1)->every(5)
        ->then(function () use ($job, $next) {
            $next($job);
        }, function () use ($job) {
            $job->release(5);
        });
}

Attach middleware to a job:

public function middleware()
{
    return [new RateLimited];
}

Job Chaining

Chain jobs to run sequentially:

Bus::chain([
    new ProcessPodcast($podcast),
    new OptimizePodcast($podcast),
    new ReleasePodcast($podcast),
])->dispatch();

If any job in the chain fails, the rest won't execute.

Chain with Callbacks

Bus::chain([
    new ProcessPodcast($podcast),
    new OptimizePodcast($podcast),
    function () use ($podcast) {
        $podcast->update(['status' => 'released']);
    },
])->dispatch();

Batch Processing

Laravel 8+ introduced job batching for executing a batch of jobs with completion callback.

$batch = Bus::batch([
    new ProcessPodcast(1),
    new ProcessPodcast(2),
    new ProcessPodcast(3),
])->then(function (Batch $batch) {
    // All jobs completed successfully
})->catch(function (Batch $batch, Throwable $e) {
    // First batch job failure
})->finally(function (Batch $batch) {
    // Executed after all callbacks
})->dispatch();

Batch Methods

Method	Description
totalJobs	Total jobs in the batch
pendingJobs	Jobs not yet processed
failedJobs	Number of failed jobs
processedJobs	Number of processed jobs
cancel()	Cancel the batch
delete()	Delete the batch

Laravel Horizon

Horizon provides a dashboard and configuration system for Redis queues.

Installation

composer require laravel/horizon
php artisan horizon:install

Configuration

Configure environments and worker settings in config/horizon.php:

'environments' => [
    'production' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['default'],
            'balance' => 'auto',
            'processes' => 10,
            'tries' => 3,
        ],
    ],
],

Running Horizon

php artisan horizon

For production, run Horizon via Supervisor:

[program:horizon]
process_name=%(program_name)s
command=php /path/to/artisan horizon
autostart=true
autorestart=true
user=forge
redirect_stderr=true
stdout_logfile=/path/to/horizon.log

Best Practices

• Keep jobs small and focused - Each job should do one thing well
• Use model IDs instead of full objects - Reduces serialization size
• Set appropriate timeouts - Prevent jobs from running too long
• Monitor your queues - Use Horizon or custom monitoring
• Handle failures gracefully - Implement failed job logic
• Use separate queues for different workloads - Prioritize important jobs
• Test your jobs - Write tests for your queueable jobs
• Consider idempotency - Design jobs to be safely retried

Pro Tip: For heavy queue workloads, consider using Laravel Octane with Swoole or RoadRunner for better performance.

Conclusion

Laravel's queue system is a powerful tool for improving application performance by deferring time-consuming tasks. From basic job dispatching to advanced features like batching and Horizon monitoring, Laravel provides everything you need to build robust background processing into your applications.

By following this guide and implementing queues where appropriate, you can significantly enhance your application's responsiveness and user experience.