Add default celery configurations #1591
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
moisses89
commented
Jul 27, 2023
config/settings/base.py
Outdated
| @@ -213,7 +213,21 @@ | |||
| # https://docs.celeryproject.org/en/stable/userguide/optimizing.html#broker-connection-pools | |||
| # https://docs.celeryq.dev/en/latest/userguide/optimizing.html#broker-connection-pools | |||
| CELERY_BROKER_POOL_LIMIT = env( | |||
| "CELERY_BROKER_POOL_LIMIT", default=env("CELERYD_CONCURRENCY", default=1000) | |||
| "CELERY_BROKER_POOL_LIMIT", default=env("CELERYD_CONCURRENCY", default=0) | |||
There was a problem hiding this comment.
@Uxio0 I changed this value to 0 to open and close connections every time it is needed, maybe could be better a value less than 1000. I'm not sure about to put this to 0, because connections with rabbitMq were designed to be durable.
Uxio0
reviewed
Jul 27, 2023
config/settings/base.py
Outdated
| default=env("CELERY_BROKER_CONNECTION_MAX_RETRIES", default=0), | ||
| ) | ||
| # https://docs.celeryq.dev/en/stable/userguide/configuration.html#broker-channel-error-retry | ||
| CELERY_BROKER_CHANNEL_ERROR_RETRY = env.int( |
moisses89
force-pushed
the
configure_celery_recommended_parameters
branch
2 times, most recently
from
967be8b
to
79d3131
Compare
Uxio0
requested changes
Jul 27, 2023
| @@ -13,4 +13,6 @@ fi | |||
| sleep 10 | |||
|
|
|||
| echo "==> $(date +%H:%M:%S) ==> Running Celery beat <==" | |||
| exec celery -C -A config.celery_app beat -S django_celery_beat.schedulers:DatabaseScheduler --loglevel $log_level | |||
| exec celery -C -A config.celery_app beat \ | |||
There was a problem hiding this comment.
without-heartbeat and so on are not required for the scheduler?
There was a problem hiding this comment.
looks like is a flag accepted only for workers.
safe-transaction-service-scheduler-1 | Error: No such option: --without-heartbeat
moisses89
force-pushed
the
configure_celery_recommended_parameters
branch
from
79d3131
to
d7cac92
Compare
Uxio0
approved these changes
Jul 27, 2023
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Several issues are appearing on
rabbitMqconnections, ("connection reset by peer", "missing heartbeat"...)One possible reason could be due the high number of messages that celery are sending to keep the heartbeat, mingle and gossip.
The following links explains a little bit more about it:
https://www.cloudamqp.com/docs/celery.html#commandline-arguments
celery/celery#4980
Also we change some default parameter that can be related:
Issues related
#1566