RabbitMQ and Its Competitors: A Practical Guide to Choosing a Message Broker

An API receives a request to generate a report. The report takes 30 seconds. Should the user wait, and what happens if the report service is temporarily unavailable?

A message broker lets the API hand work to a background worker. The API can return a job ID, while the worker processes the request separately.

RabbitMQ is a strong choice for background jobs and flexible message routing. Kafka fits retained event histories, NATS fits service messaging, and managed queues can reduce infrastructure work. The right choice depends on what must happen when consumers are slow, offline, or restarted.

Quick Selection: When to Use What

These are practical starting points, not performance rankings.

If you only need an immediate response, HTTP or gRPC may be enough. For communication between processes on one Linux machine, consider Linux IPC before introducing a network broker.

What Is RabbitMQ?

RabbitMQ is a message broker: applications publish messages, and the broker routes them to consumers. The examples here use AMQP 0-9-1, a messaging protocol supported by RabbitMQ.

Producer --> Exchange --> Queue --> Consumer
                 |
                 +-----> Another queue --> Another consumer

The producer publishes to an exchange. Even publishing with exchange="" uses RabbitMQ’s built-in default exchange, which routes by queue name. See the exchange documentation.

Work Queue vs Publish/Subscribe

• One queue, several workers: workers share the workload. A delivery goes to one worker, but failures can cause redelivery.
• Several queues bound to an exchange: each matching queue receives a copy. Billing and shipping can process the same event independently.

Common mistake: putting billing and shipping consumers on the same queue. They compete for deliveries instead of each receiving every order event. See the work queue tutorial.

Exchanges: Routing Without Application Spaghetti

For topic exchanges, * matches exactly one word and # matches zero or more words. For example, factory.# matches both factory.temperature and factory.line1.temperature. These rules are documented in the topic tutorial.

RabbitMQ: Pros and Cons

Advantages

• Flexible routing lets services subscribe to the events they need.
• Acknowledgments support recovery when workers fail.
• Prefetch limits how many unacknowledged deliveries a worker receives.
• Quorum queues provide replicated storage for important jobs.
• You can run it yourself or use a managed RabbitMQ service.

Tradeoffs

• You must operate storage, upgrades, monitoring, and recovery, or pay a provider to do so.
• Slow consumers create backlogs; a queue does not create processing capacity.
• Consumers must handle duplicates and partial failures.
• Parallel workers can finish messages in a different order from publication.
• Acknowledged queue messages are removed; historical replay requires a different storage model.

Classic Queues, Quorum Queues, or Streams?

A three-member quorum queue can tolerate one member being unavailable. A single-node development broker cannot provide that protection. Simply joining nodes into a cluster does not replicate every queue. See quorum queues and clustering.

RabbitMQ also supports streams and partitioned superstreams, so “RabbitMQ cannot replay messages” is too broad. Ordinary queues and streams serve different needs.

RabbitMQ vs Its Main Competitors

Apache Kafka: Retained Event History

Kafka stores events in partitioned logs. Consumers track their positions and can reread retained events. Separate consumer groups can process the same history independently. Ordering is within a partition, not across a whole multi-partition topic. See Kafka’s introduction.

Pros: replay, independent consumers, and a broad ecosystem for data pipelines.

Cons: partitioning, retention, and consumer lag need planning. Delayed retries and individual job handling often require additional application design.

Choose Kafka: order events must feed analytics, fraud detection, and search, and a new consumer must process last week’s retained events.

Choose RabbitMQ: the main requirement is distributing tasks to workers with selective routing and per-message acknowledgments.

NATS and JetStream: Service Messaging

Core NATS provides live publish/subscribe and request/reply with at-most-once delivery. JetStream adds persistence, replay, and acknowledged consumption. They are different reliability choices. See the NATS JetStream documentation.

Pros: straightforward subject-based messaging; useful for communication between services.

Cons: Core NATS does not retain missed messages for disconnected subscribers. JetStream adds storage, replication, and consumer configuration to operate.

Choose NATS: request/reply and subject-based communication fit your services. Enable JetStream when messages must survive disconnections.

Choose RabbitMQ: its exchange/queue model fits the workflow, or your team already operates it successfully.

Redis Streams: Useful When Redis Is Already There

Redis Streams provides an append-only stream, consumer groups, and pending-message tracking. It differs from Redis Pub/Sub, whose messages are lost to subscribers that are disconnected. See Redis Streams and Redis Pub/Sub delivery semantics.

Pros: convenient when Redis is already part of your application; supports reading history and sharing work within a group.

Cons: persistence, failover, stream trimming, and reclaiming abandoned deliveries need deliberate configuration. Sharing resources with a busy cache can affect the messaging workload.

Choose Redis Streams: a relatively small pipeline benefits from your existing Redis expertise.

Choose RabbitMQ: you want a dedicated broker with exchange-based routing. Existing Redis infrastructure alone is not a reliability argument.

Amazon SQS: Managed Queues

SQS removes broker-server administration. Standard queues provide at-least-once delivery and best-effort ordering. FIFO queues add ordering within message groups and send deduplication. See Standard queues and FIFO queues.

Pros: managed operation and convenient AWS integration.

Cons: AWS dependency, request costs, and service limits; it does not expose RabbitMQ’s exchange model. Fanout commonly involves SNS or EventBridge alongside SQS.

Choose SQS: your workers already run in AWS and reducing operational work matters most.

Choose RabbitMQ: self-hosting, portability, or its routing model matters more. With either system, make business operations safe to retry; FIFO deduplication does not make a database update or payment automatically exactly once.

Apache Artemis: Enterprise Java Integration

Apache Artemis supports Jakarta Messaging/JMS and several wire protocols. It is a separate broker from ActiveMQ Classic. See the Artemis documentation.

Pros: a natural candidate for existing Java messaging applications and enterprise integration.

Cons: broker configuration and operations still require expertise; compatibility depends on the client API and protocol your application uses.

Choose Artemis: an existing JMS/Jakarta Messaging requirement drives the decision. For a new system, compare actual client needs rather than choosing by language alone.

What About MQTT?

MQTT is a protocol, while RabbitMQ is a broker product. Mosquitto and other MQTT brokers commonly serve devices; RabbitMQ can also accept MQTT through its plugin.

A practical IoT architecture is:

Sensors --> MQTT broker --> Bridge service --> RabbitMQ --> Backend workers

Use this combination only when backend workflows justify the extra component. See the MQTT practical guide for device-side examples.

Practical Example: A RabbitMQ Job Queue in Python

This local demo sends a report job and processes it with a worker. It uses a quorum queue, publisher confirms, and manual acknowledgments.

1. Start RabbitMQ

Prerequisites: Docker, Python 3, and Python’s venv support.

docker run -d --name rabbitmq-demo --hostname rabbitmq-demo \
  -p 127.0.0.1:5672:5672 \
  -p 127.0.0.1:15672:15672 \
  -e RABBITMQ_DEFAULT_USER=demo \
  -e RABBITMQ_DEFAULT_PASS=demo-local-only \
  -v rabbitmq-demo-data:/var/lib/rabbitmq \
  rabbitmq:4-management

# Repeat until the broker reports that it is running.
docker exec rabbitmq-demo rabbitmq-diagnostics -q check_running

python3 -m venv .venv
. .venv/bin/activate
python -m pip install pika

Open http://localhost:15672 and log in with demo / demo-local-only. Port 5672 serves AMQP; 15672 serves the management UI. The credentials and unencrypted connection are for this localhost demo. The 4-management tag follows RabbitMQ 4.x; pin a tested version or digest for reproducible deployments. See RabbitMQ installation.

2. Share Connection and Queue Setup

Save as common.py:

import pika

QUEUE = "report.jobs"


def connect():
    connection = pika.BlockingConnection(pika.ConnectionParameters(
        host="127.0.0.1",
        credentials=pika.PlainCredentials("demo", "demo-local-only"),
        heartbeat=60,
        blocked_connection_timeout=30,
    ))
    channel = connection.channel()
    channel.queue_declare(
        queue=QUEUE, durable=True, arguments={"x-queue-type": "quorum"}
    )
    return connection, channel

3. Publish One Job

Save as send.py:

import json
import uuid
import pika
from common import QUEUE, connect

connection, channel = connect()
try:
    channel.confirm_delivery()
    job_id = str(uuid.uuid4())
    channel.basic_publish(
        exchange="",
        routing_key=QUEUE,
        body=json.dumps({"job_id": job_id, "report_id": 42}),
        properties=pika.BasicProperties(
            content_type="application/json",
            delivery_mode=2,
            message_id=job_id,
        ),
        mandatory=True,
    )
    print("Broker confirmed:", job_id)
finally:
    connection.close()

With confirms enabled, Pika raises an exception for a negative acknowledgment or an unroutable mandatory publication. A successful call confirms broker acceptance, not completion of the report. See Pika publisher confirmations.

4. Consume the Job

Save as worker.py:

import json
from common import QUEUE, connect

connection, channel = connect()
channel.basic_qos(prefetch_count=1)


def process(ch, method, properties, body):
    try:
        job = json.loads(body)
        if not isinstance(job, dict):
            raise ValueError("Expected a JSON object")
        if not isinstance(job.get("job_id"), str):
            raise ValueError("Missing job_id")
        if type(job.get("report_id")) is not int:
            raise ValueError("Expected an integer report_id")
    except (ValueError, UnicodeDecodeError) as error:
        print("Rejected invalid job:", error, flush=True)
        # No dead-letter exchange in this demo: this discards the message.
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)
        return

    # Replace this print with your actual, idempotent report operation.
    print(f"Processed report {job['report_id']}, job {job['job_id']}", flush=True)
    ch.basic_ack(delivery_tag=method.delivery_tag)


channel.basic_consume(queue=QUEUE, on_message_callback=process, auto_ack=False)
try:
    print("Waiting for report jobs. Press Ctrl+C to stop.", flush=True)
    channel.start_consuming()
except KeyboardInterrupt:
    channel.stop_consuming()
finally:
    if connection.is_open:
        connection.close()

Run these in separate terminals, activating the same virtual environment in each:

# Terminal 1
python worker.py

# Terminal 2
python send.py

The sender prints Broker confirmed: ...; the worker prints Processed report 42, job ....

Start another worker and publish several jobs to see work sharing. Stop both workers, publish a job, and restart one: the queued job waits for consumption. prefetch_count=1 limits each worker to one unacknowledged delivery. See RabbitMQ work queues.

This example has no automatic reconnection or retry scheduler. Unexpected processing errors close the connection, leaving unacknowledged work eligible for redelivery. Long-running work also needs a design that keeps the client’s I/O loop servicing heartbeats.

Stop and remove the demo container when finished; the named data volume remains:

docker stop rabbitmq-demo
docker rm rabbitmq-demo

Reliability: The Parts That Matter in Production

Confirms and Acknowledgments Solve Different Problems

Confirm handling and consumer acknowledgments are independent. A connection loss while waiting for a confirm creates uncertainty: the broker may already have accepted the message. Retrying can produce a duplicate. See acknowledgments and publisher confirms.

Make Operations Safe to Repeat

Imagine a worker updates a database, then crashes before acknowledging. Another worker receives the same message.

Use a stable job_id. For database-only effects, record that ID and perform the business update in the same transaction, backed by a unique constraint. If the ID already exists, acknowledge without repeating the update. External services need their own idempotency mechanism, such as a payment API’s idempotency key.

A message ID alone does not make RabbitMQ deduplicate ordinary queue messages. Preserve the same ID when retrying the same logical job. See the reliability guide.

Bound Retries and Keep Failed Messages

Do not immediately requeue a permanently invalid message forever. A practical policy is:

Transient failure --> delayed retry --> limited attempts --> dead-letter queue
Invalid payload ----------------------------------------> dead-letter queue

A dead-letter exchange routes rejected or expired messages to a configured destination. It does not automatically create a retry schedule. Configure the exchange, destination queue, bindings, delay, and attempt limit explicitly. Without a dead-letter destination, requeue=False discards the message, as in the demo. Critical workloads also need an appropriate dead-letter transfer safety configuration. See dead-letter exchanges.

Avoid the Database/Publish Gap

Saving an order and publishing orders.created are two separate operations. A crash between them can leave an order with no event.

The transactional outbox pattern saves the order and an outgoing-event row in one database transaction. A relay publishes pending rows and marks them sent after confirmation. The relay can still publish twice after a crash, so consumers still need deduplication.

Practical Production Checklist

• Backlog: monitor ready messages, unacknowledged messages, job age, processing failures, and dead-letter queue growth.
• Capacity: monitor disk, memory, and publisher blocking. Define queue limits and an overflow policy before storage fills.
• Ordering: parallel execution and retries can change completion order. Serialize work per entity when order matters.
• Connections: reuse connections and channels; configure heartbeats and reconnect with backoff.
• Payloads: include an event ID, schema version, and correlation ID. Store large files elsewhere and queue their reference.
• Access: use TLS across networks, separate application users, and restricted virtual-host permissions.
• Recovery: test worker crashes, broker restarts, and replica loss with your actual workload.

Benchmark with your message sizes, durability settings, routing, and consumer behavior. A throughput number measured without replication or acknowledgments says little about your production configuration.

For an ordinary background-job system, start by evaluating RabbitMQ. For a retained event pipeline, evaluate Kafka. For AWS jobs, compare the operational cost of SQS. Choose the simplest option that meets your recovery, routing, and replay requirements.

Related Reading

• Monolithic vs Microservices vs SOA
• MQTT Protocol: Practical Guide
• Linux IPC in Embedded Systems