[ How to kill celery worker process for process restart ]

I have a celery worker process that creates one socket to an external server in celery.signals.worker_process_init. If the connection can't be established and an exception is raised the celery worker fails to start, which is the intended behaviour.

That socket class also sends occasional heartbeat packets using a threading.Timer. Unfortunately, because the Timer thread has no special exception handling, my worker doesn't quit if something happens to the connection.

I started catching exceptions from my socket read and write operations and have tried a bunch of things to gracefully stop the current worker process:

sys.exit(1)  # Doesn't kill the worker process (because it's running in a thread?)

next attempt, I saw it mentioned somewhere in celery/billiard issues:

raise SystemExit("Socket error")  # Same, I guess

this works:

os.kill(os.getpid(), signal.SIGHUP)

BUT the celery worker supervisor process goes into a fast spin creating celery worker processes that exit immediately, instead of giving up and dying after a few attempts.

Is there a nicer way to signal the death of a celery worker process from outside a celery task?

Answer 1

I had similar problem. I ended up creating a separate script to manage workers.

Create an infinite while loop, sleep for required time and then create/kill worker process.

while True:
    time.sleep(5)
    # create workers if possible
    # kill stale workers if not required.