Docker hub raise an error while trying to deploy a new machine with mastodon.

[Mahmoud-Emad]

While I deploy a new machine with mastodon flist, my instance takes a much minute then I logged in to the machine then tracked the machine logs then got it.

[+] mastodon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

at the same time, this issue happened with @maayarosama and @Omarabdul3ziz

so after some research, I got the reason from StackOverflow

The pull limit is a rolling limit that should reset parts of the quota 6 hours after that part of the quota was used. E.g. of you do 25 pulls every hour, then after the 4th hour, you need to wait 2 hours for the first 25 pulls to be added back to your quota.

Anonymous pulls are based on the IP performing the pull, and if you are behind a proxy or NAT, that may mean others on the same network are included in your limit. So if you see the limit continue to be reached after 6 hours, there are most likely others on the network pulling from the hub with your same source IP from the NAT.

Logging in with a free Hub account doubles this limit and is based on login rather than source IP, allowing different users behind a NAT to pull without conflicting with each other.

Therefore you should include credentials with your pull commands, using docker login or the equivalent for the tool you use to pull.

so I just asking, what if we are alive with many clients, then +200 clients deployed machines at the same time or at least within 6 hours, as I understand we will see this issue again.

[xmonader]

@robvanmieghem / @delandtj what do you suggest in general for solutions requiring docker images ?

[robvanmieghem]

fuck docker hub

While I deploy a new machine with mastodon flist

why does this still need to pull from docker hub?

[Mahmoud-Emad]

why does this still need to pull from docker hub?

because we have maybe around 5 images, we have to pull all of them in the docker-compose to have a stable mastodon instance.

[robvanmieghem]

which ones?

[Mahmoud-Emad]

• postgres:14-alpine
• redis:7-alpine
• threefolddev/mastodon:latest

we use threefolddev/mastodon:latest in the sidekiq, streaming, and web services.

[robvanmieghem]

There is a pretty straight forward solution to set up a Registry as a pull through cache

It does say

Mirrors of Docker Hub are still subject to Docker’s fair usage policy.

So I have no clue if this is going to solve the problem.

[robvanmieghem]

• postgres:14-alpine
• redis:7-alpine
• threefolddev/mastodon:latest

we use threefolddev/mastodon:latest in the sidekiq, streaming, and web services.

threefolddev/mastodon should not be on docker hub imho but on ghcr.io

[ashraffouda]

we need to find a solution to this issue as this keeps happening

[Mahmoud-Emad]

@ashraffouda Deploying our registry can't solve this issue?

[scottyeager]

Setting up a registry mirror as a pull through cache (or two for good measure since Docker supports adding multiple) would be simple enough. Then the Docker config for our images would just need an update to use those.

The potential problem I see is that in the naive configuration these are open to the public and could be abused. I don't see a solution for limiting their use to Zos nodes. What might be feasible would be limiting the images served by the mirrors to those we actually use in solution deployments.

I'm not a huge fan of asking users to input Docker credentials before deployment.