How to Use Redis Lists (LPUSH, RPOP) as Message Queues

Redis Lists can make a useful small message queue when you need something lighter than RabbitMQ, Kafka, or a full background-job framework. The usual pattern is simple: producers add jobs with LPUSH, and workers take jobs with RPOP or BRPOP.

That simplicity is the reason people reach for it. A web request can drop an email job into Redis and return quickly. A worker can pick up that job a moment later. You do not need a broker topology, exchanges, topics, or a new operational stack. You do, however, need to be honest about the tradeoff: plain RPOP removes the message before the worker has finished the work. If the worker crashes at the wrong time, that job is gone unless you build an acknowledgement pattern around it.

Understanding Redis Lists as Queues

A Redis List is an ordered collection of strings. It can be seen as a sequence of elements, and Redis provides commands to add or remove elements from either the head or the tail of the list. This dual-ended nature makes lists inherently suitable for queue implementations.

• Enqueueing (Adding Messages): We can add new messages to the queue by pushing them onto one end of the list. The LPUSH command pushes elements to the head (left side) of a list.
• Dequeueing (Processing Messages): We can retrieve and remove messages from the queue by popping them from the other end of the list. The RPOP command pops elements from the tail (right side) of a list.

This specific combination (LPUSH for enqueuing and RPOP for dequeueing) creates a First-In, First-Out (FIFO) queue, which is the most common and expected behavior for a message queue.

Core Commands: LPUSH and RPOP

Let's delve into the two primary commands that form the backbone of our Redis message queue.

LPUSH key value [value ...]

The LPUSH command inserts one or more string values at the head (left side) of the list stored at key. If the key does not exist, a new list is created and the values are inserted.

Example:

Imagine you have a task that needs to be processed, such as sending an email. You can push this task as a message onto a Redis list named email_tasks.

# Push a single email task
LPUSH email_tasks "{'to': '[email protected]', 'subject': 'Welcome!', 'body': 'Thanks for signing up!'}"

# Push another task, it will be placed before the previous one
LPUSH email_tasks "{'to': '[email protected]', 'subject': 'New User Registration', 'body': 'A new user has registered.'}"

After these commands, the email_tasks list will look like this (from head to tail):

1) "{'to': '[email protected]', 'subject': 'New User Registration', 'body': 'A new user has registered.'}"
2) "{'to': '[email protected]', 'subject': 'Welcome!', 'body': 'Thanks for signing up!'}"

RPOP key

The RPOP command removes and returns the last element (from the tail, right side) of the list stored at key. If the list is empty, it returns nil.

Example:

A worker process can periodically poll the email_tasks list for new tasks using RPOP.

# A worker attempts to retrieve a task
RPOP email_tasks

If the list is not empty, RPOP will return the last element pushed (which is the first element from the tail). In our example above, the first call to RPOP would return:

"{'to': '[email protected]', 'subject': 'Welcome!', 'body': 'Thanks for signing up!'}"

Subsequent calls would then retrieve the next available task from the tail.

Building a Basic Message Queue System

Let's outline the typical flow of a simple message queue using LPUSH and RPOP.

1. Producer (Task Enqueueing)

Any part of your application that needs to offload work can act as a producer. It constructs a message (often a JSON string representing the task details) and pushes it onto a Redis list using LPUSH.

Producer Logic (Conceptual Python Example):

import redis
import json

r = redis.Redis(host='localhost', port=6379, db=0)

def send_email_task(to_email, subject, body):
    task_message = {
        'type': 'send_email',
        'payload': {
            'to': to_email,
            'subject': subject,
            'body': body
        }
    }
    # LPUSH adds to the head of the list 'email_queue'
    r.lpush('email_queue', json.dumps(task_message))
    print(f"Pushed email task to queue: {to_email}")

# Example usage:
send_email_task('[email protected]', 'Hello from Producer', 'This is a test message.')
send_email_task('[email protected]', 'Important Update', 'New features available.')

2. Consumer (Task Dequeueing and Processing)

Worker processes, running independently, will continuously monitor the Redis list for new messages. They use RPOP to fetch and remove a message from the queue.

Consumer Logic (Conceptual Python Example):

import redis
import json
import time

r = redis.Redis(host='localhost', port=6379, db=0)

def process_tasks():
    while True:
        # RPOP attempts to get a message from the tail of the list 'email_queue'
        message_bytes = r.rpop('email_queue')
        if message_bytes:
            message_str = message_bytes.decode('utf-8')
            try:
                task = json.loads(message_str)
                print(f"Processing task: {task}")
                # Simulate task processing
                if task.get('type') == 'send_email':
                    print(f"  -> Sending email to {task['payload']['to']}...")
                    # Replace with actual email sending logic
                    time.sleep(1) # Simulate work
                    print(f"  -> Email sent to {task['payload']['to']}.")
                else:
                    print(f"  -> Unknown task type: {task.get('type')}")
            except json.JSONDecodeError:
                print(f"Error decoding JSON: {message_str}")
            except Exception as e:
                print(f"Error processing task {message_str}: {e}")
        else:
            # No message, wait a bit before polling again
            # print("No tasks available, waiting...")
            time.sleep(0.5)

if __name__ == "__main__":
    print("Worker started. Waiting for tasks...")
    process_tasks()

When you run the producer, it pushes messages. When you run the consumer, it will start picking them up and processing them. The order of processing will correspond to the order they were pushed (FIFO) because LPUSH adds to the head and RPOP removes from the tail.

Reliability Considerations

While LPUSH and RPOP provide a basic queueing mechanism, building a production queue means deciding what should happen when workers crash, jobs fail, or producers send malformed payloads.

1. Message Loss During Processing

If a worker process crashes after RPOP has removed the message but before it has finished processing, that message is lost. To prevent this:

• Use BRPOP instead of tight polling: BRPOP blocks until a list has an element or until a timeout occurs. This does not make processing reliable by itself, but it does stop workers from waking up every few milliseconds just to find an empty queue.

  # Blocking pop from the right, with a 0 timeout (block indefinitely)
  BRPOP email_queue 0
  

• Use a processing list for acknowledgement: A common pattern is to atomically move a message from email_queue to email_processing, process it, and then remove it from email_processing only after the work succeeds. If a worker dies, a separate reaper process can look for stale items in the processing list and move them back to the main queue. RPOPLPUSH is the classic command for this pattern, and newer Redis versions also provide LMOVE/BLMOVE.

2. Handling Failed Tasks

What happens if a task fails during processing (e.g., due to a temporary network issue or bad data)?

• Retry Mechanisms: Implement retry logic within the worker. After a few failures, move the task to a 'failed_tasks' list for manual inspection.
• Dead Letter Queue (DLQ): A dedicated Redis list (or other storage) where messages that repeatedly fail processing are sent. This is crucial for debugging and recovery.

3. Multiple Consumers

If you have multiple worker instances consuming from the same queue, RPOP (and BRPOP) ensures that each message is processed by only one worker. This is because RPOP atomically removes the element.

4. Message Ordering

While LPUSH and RPOP create a FIFO queue, this guarantee is only as strong as your processing logic. If consumers re-queue failed messages without proper handling, or if you introduce other operations, the strict FIFO order might be compromised.

5. Payload Format and Idempotency

Treat the message body as a small contract. JSON is common because it is easy to inspect in redis-cli, but use valid JSON rather than Python-style single-quoted dictionaries:

{"type":"send_email","id":"email-1842","payload":{"to":"[email protected]","template":"welcome"}}

The id field matters. If a worker retries after a timeout, or if a stale job gets re-queued from a processing list, the same logical job may run more than once. Design the handler so a duplicate is harmless. For an email worker, that might mean recording email-1842 in the application database before sending, then checking that record before any retry sends another message.

6. Queue Length and Backpressure

Watch queue length with LLEN email_queue. A growing queue is not automatically bad; it may simply mean workers are catching up after a traffic spike. A queue that grows for hours usually means producers are faster than consumers, workers are failing, or one slow dependency is holding everything back.

In practice, I like to alert on age as well as length. Redis Lists do not store enqueue time separately, so put a timestamp in the payload if job age matters:

{"type":"resize_image","id":"img-991","created_at":"2026-05-24T08:15:00Z","payload":{"image_id":991}}

Then your worker logs can tell you whether jobs are being processed seconds late or hours late. That is much more useful than length alone when you are debugging a real incident.

Advanced Techniques (Briefly)

• RPOPLPUSH: Atomically pops a message from one list and pushes it onto another (e.g., a 'processing' list). This is a key command for implementing reliable processing with acknowledgements.
• BLPOP / BRPOP with Multiple Keys: Block and pop from the first list that becomes non-empty. Useful for consuming from multiple queues.
• Lua Scripting: For complex atomic operations that RPOPLPUSH doesn't cover, Lua scripts can be used to ensure critical sequences of commands execute without interruption.

A More Reliable Worker Shape

For anything more important than a best-effort notification, avoid the plain "pop and hope" worker. A safer shape looks like this:

RPOPLPUSH email_queue email_processing

The worker receives the message and Redis has already moved it to email_processing. After the email is sent and the application records success, the worker removes that exact payload from the processing list:

LREM email_processing 1 '{"type":"send_email","id":"email-1842"}'

That still is not a perfect enterprise queue. LREM must match the payload, large processing lists can become awkward, and you need a reaper process that knows when a message is old enough to retry. But it changes the failure mode in a useful way. A worker crash no longer deletes the only copy of the job.

If you use this approach, put retry metadata in the message or store it beside the message ID in another key. For example, a reaper can move a stale message back to email_queue the first few times, then move it to email_failed after the retry limit. That gives you a place to inspect poison messages instead of watching the same bad payload fail forever.

When Redis Lists Are the Wrong Queue

Redis Lists are easy to understand, but they are not always the right tool. If you need delayed jobs, job priorities, scheduled retries, workflow visibility, or long-term audit history, a job library or a dedicated broker may be less work in the end. Redis Streams are also worth considering because they have consumer groups and acknowledgement semantics built into the data type.

I still like Lists for small internal queues: thumbnail generation, cache warming, webhook fan-out where the source system can retry, or simple background jobs owned by one application. The moment multiple teams depend on the queue contract, write down the delivery expectations. "At least once", "at most once", and "best effort" are not academic terms during an incident. They decide whether duplicates are acceptable, whether lost messages are tolerable, and how much recovery machinery you need.

Redis Lists are a good fit when you need a small, understandable queue and you already operate Redis. Start with LPUSH and BRPOP for simple background work. Add a processing list, retry count, and dead-letter list once losing a job would matter. If you need delayed scheduling, priorities, fan-out, long retention, or strong delivery guarantees across many services, that is usually the point where a purpose-built queue becomes easier to live with than a growing pile of Redis conventions.