When a Queue Table Isn't Enough: Moving to RabbitMQ

A surprising number of production systems handle background work the exact same way. They use a jobs table, a status column (pending, processing, done, failed), and one or more workers that poll it on a loop, picking up pending rows and updating their status as they go.

This is a completely reasonable way to start. It needs zero new infrastructure. It is incredibly easy to inspect (SELECT * FROM jobs WHERE status = 'failed' is a debugging tool everyone already knows how to use). And for low volumes, it honestly works without issue for a very long time.

The signs that a system has outgrown this approach are not usually "too much volume" in the abstract. They are much more specific than that.

The polling itself becomes the load

Every single worker checking the table every few seconds is a query, regardless of whether there is actually any work to do. At low volume, this is completely negligible.

As the table grows and more workers get added to handle the throughput, that constant polling becomes a steady, grinding background load on the database. It actively competes with the actual application queries it is meant to support. Even using SELECT ... FOR UPDATE SKIP LOCKED (the correct way to do this safely) still means every single worker is hammering the exact same table on every cycle.

Retries and failure handling get reinvented, badly

A queue table's retry logic is usually a retry_count column and some application code that increments it and re-queues the job, until it doesn't.

Edge cases slowly start to accumulate as special cases over time. Jobs that partially completed before failing, jobs that get picked up twice because two workers polled at the exact same moment, or failed jobs that pile up with no clear way to inspect why. These are not impossible problems, but they are problems a dedicated message queue solved years ago. Re-solving them in application code simply means painfully re-solving all their edge cases too.

What RabbitMQ actually changes

RabbitMQ completely flips the model from polling to pushing. A producer publishes a message to an exchange. RabbitMQ routes it to a queue based on bindings. And consumers receive messages as they arrive rather than repeatedly asking for them:

# Producer
channel.basic_publish(
    exchange='',
    routing_key='email_notifications',
    body=json.dumps({'user_id': 123, 'template': 'welcome'}),
    properties=pika.BasicProperties(delivery_mode=2)  # persistent
)

# Consumer
def callback(ch, method, properties, body):
    process_notification(json.loads(body))
    ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_consume(queue='email_notifications', on_message_callback=callback)

basic_ack only fires after the message is successfully processed. If the consumer crashes mid-processing, RabbitMQ instantly redelivers the message because it was never acknowledged. Failed messages can route to a dead-letter exchange automatically. This gives you a completely separate queue of "things that failed" to cleanly inspect, without writing a single line of that logic yourself.

What stays the same, and what doesn't transfer

The mental model of "a job has a payload and a status" does not go away. RabbitMQ does not replace the concept; it simply replaces the delivery mechanism.

What does not transfer directly is your debugging habit. SELECT * FROM jobs WHERE status = 'failed' becomes checking a dead-letter queue instead. This has a different (though arguably better, as messages there are genuinely undeliverable rather than just slow) shape. Also, anything that relied on the queue table being queryable alongside other application data in the same transaction—like atomically inserting a row and a related queue entry together—needs serious rethinking, since the queue is now a physically separate system.

When the table is still the right call

If the volume is low, the jobs are incredibly simple, and a handful of workers polling every few seconds does not register as a load on the database, a queue table is not a mistake. It is appropriately sized for the exact problem you have.

The switch to RabbitMQ is worth making when polling load becomes visibly painful in database metrics. It is worth it when retry and failure handling in application code has grown its own massive set of edge cases. Or, it is worth it when multiple different services suddenly need to produce or consume the exact same events. At that point, a message broker designed exactly for that stops being "extra infrastructure" and starts being the far simpler option.