[ How to kill celery worker process for process restart ]
I have a celery worker process that creates one socket to an external server in celery.signals.worker_process_init
. If the connection can't be established and an exception is raised the celery worker fails to start, which is the intended behaviour.
That socket class also sends occasional heartbeat packets using a threading.Timer
. Unfortunately, because the Timer thread has no special exception handling, my worker doesn't quit if something happens to the connection.
I started catching exceptions from my socket read and write operations and have tried a bunch of things to gracefully stop the current worker process:
sys.exit(1) # Doesn't kill the worker process (because it's running in a thread?)
next attempt, I saw it mentioned somewhere in celery/billiard
issues:
raise SystemExit("Socket error") # Same, I guess
this works:
os.kill(os.getpid(), signal.SIGHUP)
BUT the celery worker supervisor process goes into a fast spin creating celery worker processes that exit immediately, instead of giving up and dying after a few attempts.
Is there a nicer way to signal the death of a celery worker process from outside a celery task?
Answer 1
I had similar problem. I ended up creating a separate script to manage workers.
Create an infinite while loop, sleep for required time and then create/kill worker process.
while True:
time.sleep(5)
# create workers if possible
# kill stale workers if not required.