November 12, 2023
Using PostgreSQL as a Job Queue
Skip Redis and Celery. How to build a robust, transactional background job queue using standard PostgreSQL features.
When building background processing pipelines, the default instinct is to reach for Redis and a dedicated queueing library. For 95% of applications, this introduces unnecessary infrastructure complexity.
You can build a highly concurrent, transactional queue directly in PostgreSQL using FOR UPDATE SKIP LOCKED.
SELECT * FROM jobsWHERE status = 'pending'
ORDER BY created_at ASC
FOR UPDATE SKIP LOCKED
LIMIT 1;
This query ensures that multiple worker nodes can concurrently poll the database without ever pulling the same job twice. It is transactional, persistent, and requires zero additional infrastructure.