Using with Celery¶
Celery is an asynchronous task queue/job queue based on distributed message passing.
Usage in a Worker¶
Workers in celery may be created by spawning new processes or threads from the celeryd process. The multiprocessing module is used to spawn new worker processes, while eventlet is used to spawn new worker green threads.
multiprocessing
¶
The ConnectionPool
class is not multiprocessing
-safe. Because
celery evaluates globals prior to spawning new worker processes, a global
ConnectionPool
will be shared among multiple processes. This is
inherently unsafe and will result in race conditions.
Instead of having celery spawn multiple child processes, it is recommended that
you set
CELERYD_CONCURRENCY
to 1 and start multiple separate celery processes. The process argument
--pool=solo
may also be used when starting the celery processes.
See also
eventlet
¶
Because the ConnectionPool
class uses concurrency primitives from
the threading
module, you can use eventlet
worker threads after
monkey patching
the standard library. Specifically, the threading
and socket
modules must monkey-patched.
Be aware that you may need to install dnspython in order to connect to your nodes.
See also
Usage as a Broker Backend¶
pycassa is not currently a broker backend option.