Hm, this post is from 2014 so many of the arguments presented don't stand anymore.
Today you can containerize and autoscale Celery workers, so having task specific queues and worker groups will solve your resource utilization problems. For even better utilization, you can have gevent and process based workers depending if the task is resource or I/O heavy.
In years I have never seen Celery workers hang, we process millions of tasks (hundreds of task types) every day. Since the post is very old, I'm sure that they had to deal with a few nasty bugs back then.
However, Celery does have real scalability issues: If you run over 1000 worker nodes Celery gossip alone will be well over 1000 messages per second. Having 1000s of gossip queues isn't ideal either given that RabbitMQ is not easily scalable. You might end up with multiple clusters which can be a bit of a pain in a single codebase.
I'm not sure if running Celery on SQS solves some of these issues.
Today you can containerize and autoscale Celery workers, so having task specific queues and worker groups will solve your resource utilization problems. For even better utilization, you can have gevent and process based workers depending if the task is resource or I/O heavy.
In years I have never seen Celery workers hang, we process millions of tasks (hundreds of task types) every day. Since the post is very old, I'm sure that they had to deal with a few nasty bugs back then.
However, Celery does have real scalability issues: If you run over 1000 worker nodes Celery gossip alone will be well over 1000 messages per second. Having 1000s of gossip queues isn't ideal either given that RabbitMQ is not easily scalable. You might end up with multiple clusters which can be a bit of a pain in a single codebase. I'm not sure if running Celery on SQS solves some of these issues.