Skip to content

fix: Update runservice worker scripts to disable unneeded features - #1371

Merged
czgu merged 1 commit into
pinterest:masterfrom
baumandm:external/worker-redis
Nov 14, 2023
Merged

fix: Update runservice worker scripts to disable unneeded features#1371
czgu merged 1 commit into
pinterest:masterfrom
baumandm:external/worker-redis

Conversation

@baumandm

Copy link
Copy Markdown
Contributor

We've encountered issues where workers would stop receiving tasks after an error message like this:

consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 332, in start
    blueprint.start(self)

I was able to reproduce this locally via Docker by manually bouncing the Redis container.

After some research, we came across suggestions to disable the heartbeat/gossip/mingle features of Celery. In my local testing it appears this change eliminates (or at least reduces) the issue where the workers cannot reconnect to Redis after a connection failure.

We have been running this in production for over a month and everything works as expected, including Flower.

@czgu
czgu merged commit 09ffc40 into pinterest:master Nov 14, 2023
aidenprice pushed a commit to arrowtail-precision/querybook that referenced this pull request Jan 3, 2024
@baumandm
baumandm deleted the external/worker-redis branch January 10, 2024 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants