Docker Hub pull rate limits

[rnorth]

N.B. This issue description will be updated as new information becomes available.

As of 2020-08-13, Docker have updated their terms of service and pricing page, indicating that:

• unauthenticated pulls will be rate limited to 100 per 6h
• authenticated pulls will be rate limited to 200 per 6h

The page on rate limits clarifies that this applies to layers being pulled, not images. Since most images comprise multiple layers, the effective image pull rate limit will be very low.

The rate limit page is inconsistent about the number, but also states that these rate limits are being introduced gradually.

Based on a recent build it does not appear that these rate limits are being enforced yet. However, it seems as though they will be in effect by 2020-11-01, so we need to understand the implications for various groups.

Implications

• For environments where images are persisted between builds, the rate limit may not be hit. For ephemeral CI environments where images are discarded between builds, it seems more likely that the rate limit will be hit.

• In general, all user groups may need to either:

  • use an authenticated Docker Hub account for builds. This page describes how this can be set up.
    • testcontainers-java has good support for authenticated image pulls. Other language forks will need work done.
    • Docker Hub's Free/Pro/Team plans will allow 300/unlimited/unlimited layer pulls per 6h.
  • OR, users may wish to set up their own registry to mirror/copy images which they require. This is already possible, and work is underway within Testcontainers to continue to make this easier.

• user groups with low image pulling requirements (e.g. few images or a low rate of builds) will presumably not be impacted.

For open source projects using Testcontainers within their builds (including Testcontainers itself)

• per https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/, an open source project plan will be available with free unlimited pulls, and Docker have an application form here: forms.gle/vvKURDTYwok7Pc4r5
• For projects using GitHub Actions: GitHub Actions will have mitigation in place to prevent rate limits from affecting builds.
• For projects not using GitHub Actions: For projects that set up authenticated pulls in their CI infrastructure, it is not clear what approach should be followed for repository forks. Using build secrets to hold Docker Hub auth tokens means it will not be available to forks.

For companies using Testcontainers within their builds

• If Hub's rate limiting is based on IP address, and if developers share an IP address, it seems possible that rate limits could apply to developers as well as CI infrastructure.
• For projects that set up authenticated pulls in their CI infrastructure, it is also not clear what approach should be followed for repository forks. Using build secrets to hold Docker Hub auth tokens means it will not be available to forks, although in a company environment this may be more manageable.
• Companies may be more likely to have private docker registries (e.g. ECR, GCR) which they can copy required images to. Testcontainers will implement an image substitution mechanism to allow private registry image names to be used as substitutes for public Docker Hub images (see below).

Actions for users

• Be mindful that rate limits may apply soon, and some preparation will be needed. Consider the above options and implement.
• Please help clarify open questions if you have additional information.

Actions for Testcontainers team

• Set up authenticated pulls for Testcontainers' own builds (N.B. this refers to adding docker login to our CI scripts. Authenticated pull support is already implemented in testcontainers-java)

  • GitHub Actions - skip, not needed
  • CircleCI, Azure Pipelines for Linux, Travis - switch to master branch only and use secrets
  • Azure Pipelines for Windows (triggered by maintainer PR comments) - use secrets

• Implement transparent 'image substitution' (whereby Testcontainers will consult a user-provided piece of code to obtain an alias for an image, which may reside on a private registry). PR: Image substitution #3102

• Investigate the possibility of providing an implementation of image substitution that treats GitHub Packages as a cache.

• Retrofit to language forks: authenticated image pulls and image substitution

• Improve documentation around authenticated image pulls and options for mirroring registries / copying images to non-Hub registries

Open questions

• When will rate limits noticeably kick in? Unclear: nominally 1st November, but some Docker/Testcontainers users have reported rate limiting already

  • Docker have said that IPs pulling the highest volume will be experiencing the rate limit earlier. All other signs point to 1st November being the start.

• Are CI platforms going to attempt to mitigate this?

  • GitHub Actions: YES
  • Others, no information yet

• Is there a separate Free plan for Open Source projects that allows unlimited pulls? Yes, see https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/

• How should repo forks manage the need to provide Hub credentials?

  • Aside from GitHub Actions providing a different approach to mitigation, it appears there is no solution to this problem for other CI platforms.
  • teams using their own CI executors may find other mitigations, such as configuring a local docker registry server to act as an authenticated mirror.

[rnorth]

One idea: for fork builds on GitHub Actions, we could look at using GitHub Packages as a cache for required images. Images pushed to GitHub Packages should be readable on forks, so the main (automatable) work would be:

• to ensure that an OSS project's docker dependencies are published to GitHub Packages
• to ensure that image substitution is performed to rewrite references to the original image name with a corresponding GitHub Package URI.

Both could be accomplished using the pluggable image substitution mechanism that we have in mind.

I'll add an action above for us to investigate.

[vlsi]

AFAIK, GitHub registry requires API token even for image pulls from public repositories, so I'm not sure images would be readable on forks.

[rnorth]

@vlsi you might be right, so I'd like to do some investigation before putting a lot of effort in. My understanding of the docs was that forks can access a GITHUB_API_TOKEN secret which is a different, read-only, token. It's quite possible that I'm wrong.

[WtfJoke]

@vlsi, @rnorth is correct the GITHUB_TOKEN provided by github action has read only access on forked repos.

See https://docs.github.com/en/actions/configuring-and-managing-workflows/authenticating-with-the-github_token#permissions-for-the-github_token

[fongie]

just fyi, this started happening for us today in aws codebuild in our pipeline, tests fail because limit exceeded on docker hub, so some sort of rates are being enforced already (not waiting until november)

[rnorth]

@fongie that's alarming and disappointing - my understanding was that this was not being applied yet.

Based on a test branch (#3098) I get the impression that GitHub Actions and CircleCI might not be getting rate limited yet.

The advice would remain the same - to use an authenticated account for pulling or to use a mirror/copy in another registry. I'm saddened that we've not had time to release our image substitution yet, though.

[rnorth]

Updated above with link to https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/ which clarifies:

• that pulls will be based on image pulls, not layers.
• an open source project plan will be available and Docker have an application form here: https://forms.gle/vvKURDTYwok7Pc4r5

From my perspective, two major topics remain unresolved:

• for open source projects, how can Docker Hub auth credentials be shared with forks of repos for PR builds? Will CI platform vendors step in with a solution?
• we want to ship transparent image substitution in Testcontainers. If we have until November I'd be confident in this; if rate limiting is being applied much sooner then we do not have time.

@fongie I've reached out to a contact at Docker to see if we can get clarification on when the rate limiting is supposed to be taking effect.

[fongie]

Great! Sorry for not being more specific last post, I was busy trying to get our release out hehe

This is the error message we got:
Caused by: com.github.dockerjava.api.exception.DockerClientException: Could not pull image: error pulling image configuration: toomanyrequests: Too Many Requests. Please see https://docs.docker.com/docker-hub/download-rate-limit/

in AWS CodeBuild, trying to pull 'postgres:10.7'

It worked on the third attempt simply by retrying. I understand that there is the option to set up an authenticated account, I just dont have the time for that right now, but I would guess this is happening to more people and you might get more questions about this issue, so it might be nice to prepare thorough instructions for how to do it. If I log in to another docker registry in my codebuild environment, will testcontainers automatically use that to pull from when I do new PostgreSQLContainer("postgres:10.7"), or do I need to instruct testcontainers on which registry to use?

[rnorth]

@fongie sorry for the slow response. If you log in to Docker Hub then Testcontainers will be able to use that hub account for pulling.

If you log in to another docker registry (e.g. ECR) then that does not help directly, but it could be made to work. ECR does not function as a pull-through cache (AFAIK), so you'd have to copy the required image into ECR and use new PostgreSQLContainer("your.ecr.registry.amazonaws.com/somepath/postgres:10.7") to create the container. This is a bit of a pain, which is why I'm working on #3102 to at least help with the second aspect.

Logging in to Docker Hub requires fewer changes.

[rnorth]

For GitHub Actions users, this issue looks worth following: actions/runner-images#1445