Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Ensure default branch cache persists #100457

Merged
merged 1 commit into from
Mar 5, 2025

Conversation

Repiteo
Copy link
Contributor

@Repiteo Repiteo commented Dec 16, 2024

I've been brainstorming solutions to our cache system & this is what I've come up with: a peristent cache for the default branch. Presently, after a merge batch happens on main, we're lucky if that cache lasts the entire day; when traffic is high, we're lucky if it lasts mere hours. This comes down to a double-whammy of caches being immutable (can't override existing versions) & fallback keys not counting as "accessing" the cache. While the former is something we just have to live with, this PR works around the latter with this new system:

  1. Before the local branch's cache is restored, attempt a read-only detection of the default branch's cache via fallback key.
  2. If the previous step succeeded, use its output to cache-hit the key explicitly; this registers that cache as "accessed".

I've cleaned out my cache to demonstrate how integrating this affects a cache list. When quota is reached, the oldest cache accessed are what will go away first.

ID     KEY                                                                                           SIZE        CREATED                 ACCESSED
24446  windows-editor|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520                  222.32 MiB  less than a minute ago  less than a minute ago
24445  windows-template|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520                120.11 MiB  less than a minute ago  less than a minute ago
24444  linux-editor-double-sanitizers|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520  430.62 MiB  about 3 minutes ago     about 3 minutes ago
24443  windows-template-gcc|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520            58.41 MiB   about 4 minutes ago     about 4 minutes ago
24442  web-nothreads-template|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520          157.14 MiB  about 4 minutes ago     about 4 minutes ago
24441  web-template|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520                    130.58 MiB  about 4 minutes ago     about 4 minutes ago
24440  windows-editor-clang|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520            102.66 MiB  about 5 minutes ago     about 5 minutes ago
24439  linux-template-minimal|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520          25.51 MiB   about 5 minutes ago     about 5 minutes ago
24438  linux-editor-mono|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520               95.17 MiB   about 6 minutes ago     about 6 minutes ago
24437  macos-editor|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520                    229.29 MiB  about 6 minutes ago     about 6 minutes ago
24436  macos-template|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520                  88.45 MiB   about 6 minutes ago     about 6 minutes ago
24435  linux-editor-llvm-sanitizers|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520    623.16 MiB  about 6 minutes ago     about 6 minutes ago
24411  linux-template-minimal|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089          25.51 MiB   about 28 minutes ago    about 6 minutes ago
24389  linux-template-minimal|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                        25.51 MiB   about 1 hour ago        about 6 minutes ago
24434  linux-template-mono|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520             48.76 MiB   about 6 minutes ago     about 6 minutes ago
24426  linux-editor-double-sanitizers|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089  420.56 MiB  about 23 minutes ago    about 7 minutes ago
24414  linux-editor-mono|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089               93.63 MiB   about 27 minutes ago    about 7 minutes ago
24390  linux-editor-mono|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                             90.61 MiB   about 1 hour ago        about 7 minutes ago
24418  linux-template-mono|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089             48.75 MiB   about 26 minutes ago    about 7 minutes ago
24388  linux-editor-double-sanitizers|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                400.08 MiB  about 1 hour ago        about 7 minutes ago
24392  linux-template-mono|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                           48.75 MiB   about 1 hour ago        about 7 minutes ago
24433  ios-template|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520                    218.70 MiB  about 7 minutes ago     about 7 minutes ago
24432  linux-editor-thread-sanitizer|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520   333.24 MiB  about 7 minutes ago     about 7 minutes ago
24431  android-editor|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520                  313.83 MiB  about 7 minutes ago     about 7 minutes ago
24430  android-template-arm32|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520          273.23 MiB  about 7 minutes ago     about 7 minutes ago
24429  android-template-arm64|ci/cache-persistence|65babb99028518cf92ddb0e404c66f67b5b89520          261.74 MiB  about 7 minutes ago     about 7 minutes ago
24421  linux-editor-llvm-sanitizers|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089    620.74 MiB  about 25 minutes ago    about 8 minutes ago
24425  web-nothreads-template|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089          148.83 MiB  about 24 minutes ago    about 8 minutes ago
24423  web-template|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089                    130.56 MiB  about 24 minutes ago    about 8 minutes ago
24385  web-nothreads-template|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                        148.81 MiB  about 1 hour ago        about 8 minutes ago
24386  web-template|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                                  131.25 MiB  about 1 hour ago        about 8 minutes ago
24419  linux-editor-thread-sanitizer|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089   331.30 MiB  about 26 minutes ago    about 8 minutes ago
24428  windows-editor|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089                  217.08 MiB  about 19 minutes ago    about 8 minutes ago
24387  linux-editor-llvm-sanitizers|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                  615.57 MiB  about 1 hour ago        about 8 minutes ago
24427  windows-template|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089                119.36 MiB  about 20 minutes ago    about 8 minutes ago
24391  linux-editor-thread-sanitizer|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                 327.60 MiB  about 1 hour ago        about 8 minutes ago
24424  windows-editor-clang|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089            101.15 MiB  about 24 minutes ago    about 8 minutes ago
24422  windows-template-gcc|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089            57.81 MiB   about 24 minutes ago    about 8 minutes ago
24382  windows-template|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                              117.73 MiB  about 1 hour ago        about 8 minutes ago
24416  ios-template|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089                    218.08 MiB  about 27 minutes ago    about 8 minutes ago
24383  windows-template-gcc|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                          56.57 MiB   about 1 hour ago        about 8 minutes ago
24381  windows-editor-clang|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                          98.19 MiB   about 1 hour ago        about 8 minutes ago
24420  macos-editor|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089                    219.86 MiB  about 25 minutes ago    about 8 minutes ago
24384  windows-editor|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                                205.93 MiB  about 1 hour ago        about 8 minutes ago
24412  android-template-arm32|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089          273.18 MiB  about 27 minutes ago    about 8 minutes ago
24417  macos-template|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089                  87.41 MiB   about 26 minutes ago    about 8 minutes ago
24415  android-editor|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089                  313.85 MiB  about 27 minutes ago    about 8 minutes ago
24380  ios-template|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                                  216.99 MiB  about 1 hour ago        about 8 minutes ago
24378  macos-editor|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                                  201.12 MiB  about 1 hour ago        about 8 minutes ago
24413  android-template-arm64|ci/cache-persistence|d2767e8fbdf32b5170bfaa8b4fa675b510586089          243.38 MiB  about 27 minutes ago    about 8 minutes ago
24375  macos-template|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                                85.36 MiB   about 1 hour ago        about 8 minutes ago
24377  android-template-arm32|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                        232.45 MiB  about 1 hour ago        about 8 minutes ago
24379  android-editor|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                                310.96 MiB  about 1 hour ago        about 8 minutes ago
24376  android-template-arm64|master|adc54f127d3e66be5f5b07e8c65b7886fb74089b                        206.61 MiB  about 1 hour ago        about 8 minutes ago
24147  pre-commit-3||607cb7f10e088734ecdb7dfc5497153b214291e33cf4f3a1936bb64509aef747                51.87 MiB   about 10 hours ago      about 9 minutes ago
24146  cache-apt-pkgs_90e68416d72c033bca307201aa6958f4                                               43.80 KiB   about 10 hours ago      about 9 minutes ago
24399  linux-template-mono|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9             48.75 MiB   about 1 hour ago        about 26 minutes ago
24407  linux-editor-double-sanitizers|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9  410.31 MiB  about 1 hour ago        about 27 minutes ago
24408  linux-editor-thread-sanitizer|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9   329.51 MiB  about 1 hour ago        about 27 minutes ago
24406  linux-editor-llvm-sanitizers|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9    618.06 MiB  about 1 hour ago        about 27 minutes ago
24404  web-nothreads-template|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9          140.44 MiB  about 1 hour ago        about 28 minutes ago
24402  web-template|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9                    130.52 MiB  about 1 hour ago        about 28 minutes ago
24410  windows-editor|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9                  211.88 MiB  about 1 hour ago        about 28 minutes ago
24395  linux-editor-mono|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9               92.12 MiB   about 1 hour ago        about 28 minutes ago
24405  windows-template-gcc|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9            57.18 MiB   about 1 hour ago        about 28 minutes ago
24409  windows-template|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9                118.59 MiB  about 1 hour ago        about 28 minutes ago
24398  linux-template-minimal|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9          25.51 MiB   about 1 hour ago        about 28 minutes ago
24403  windows-editor-clang|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9            99.66 MiB   about 1 hour ago        about 28 minutes ago
24401  macos-editor|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9                    210.53 MiB  about 1 hour ago        about 28 minutes ago
24397  ios-template|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9                    217.56 MiB  about 1 hour ago        about 28 minutes ago
24393  android-template-arm64|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9          225.03 MiB  about 1 hour ago        about 28 minutes ago
24400  macos-template|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9                  86.37 MiB   about 1 hour ago        about 28 minutes ago
24394  android-editor|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9                  312.41 MiB  about 1 hour ago        about 28 minutes ago
24396  android-template-arm32|ci/cache-persistence|5f8d7e2219a0eadda0223475559a16c4150d3fa9          252.88 MiB  about 1 hour ago        about 28 minutes ago

@Repiteo Repiteo added this to the 4.4 milestone Dec 16, 2024
@Repiteo Repiteo requested a review from a team as a code owner December 16, 2024 02:55
@akien-mga
Copy link
Member

Looks good overall, but how does this deal with stable branches like 4.3, or dev branches like 3.x?

We don't want all of those to use caches from master, they should use a cache from their own base branch.

@Ivorforce
Copy link
Contributor

Ivorforce commented Dec 16, 2024

We don't want all of those to use caches from master, they should use a cache from their own base branch.

Commenting something I've come across for earlier PRs: It may be useful to use ${{github.event.pull_request.base.sha}} (ref) to (somehow) figure out the target of the pull request, and ping and / or pull from the appropriate cache.

@AThousandShips AThousandShips self-requested a review December 16, 2024 12:53
@AThousandShips
Copy link
Member

At a glance this looks promising! Was considering something like it in my testing, will give it a review in the next few days when I find time!

@Repiteo
Copy link
Contributor Author

Repiteo commented Dec 16, 2024

Looks good overall, but how does this deal with stable branches like 4.3, or dev branches like 3.x?

In the short-term, it doesn't. Those branches already have their own Action syntax/names, which don't overlap with this PR's new cache names at all. If anything, they'll behave better, as now there's zero risk of falling back to master before their PR cache is setup.

As for if this gets cherry-picked and/or when 4.4 becomes a stable branch... That's a little trickier. It might be possible to exclude master as a fallback for PRs not targeting it, but I'd have to look into it further. Regardless, the "pinging" of the master cache should always occur, because it persisting is important even if unused

Commenting something I've come across for earlier PRs: It may be useful to use ${{github.event.pull_request.base.sha}} (ref) to (somehow) figure out the target of the pull request, and ping and / or pull from the appropriate cache.

It's extremely important to note that only the default branch of a repo can have its cache accessed by other branches. The idea that all caches could be shared across any PR was a HUGE misconception of the earlier implementation; they're deliberately isolated. That's why I removed the last fallback key case, it never came into play.

If we wanted something that could reference any PR from any PR, the only things I can think of are:

  1. Recreating the caches on master via callback Action.
  2. Upload caches as artifacts & detect/download the right one arbitrarily.

I've not looked into either option thoroughly, nor am I certain if they're even feasible, but it's a start. I'm absolutely gonna pursue them further in other PRs, but they're almost certainly outside the scope of this one.

@AThousandShips
Copy link
Member

This solution looks good, can't extensively test it as such but it should work based on what I can see

@Repiteo Repiteo force-pushed the ci/cache-persistence branch from 65babb9 to 865ef15 Compare January 10, 2025 03:42
@Repiteo Repiteo modified the milestones: 4.4, 4.5 Jan 25, 2025
@Repiteo Repiteo force-pushed the ci/cache-persistence branch from 865ef15 to 5265503 Compare January 25, 2025 18:16
Copy link
Member

@AThousandShips AThousandShips left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't confirm the behavior itself but the change makes sense and I think we can give this a try and see if it helps

@akien-mga
Copy link
Member

Needs rebase to resolve merge conflicts.

@Repiteo Repiteo force-pushed the ci/cache-persistence branch from 5265503 to 0b98715 Compare March 5, 2025 16:34
@Repiteo Repiteo merged commit 937fccf into godotengine:master Mar 5, 2025
1 check passed
@Repiteo Repiteo deleted the ci/cache-persistence branch March 5, 2025 16:35
@AThousandShips
Copy link
Member

There is now a warning when running:

Warning: Failed to restore: Failed to GetCacheEntryDownloadURL: Received non-retryable error: Failed request: (400) Bad Request: key invalid length for cache entry key: must be between 1 and 512 characters

Unsure if this inhibits the fetch or not, will have to test by re-running once CI has finished

@AThousandShips
Copy link
Member

Would have been good to have been able to verify this before the PR was merged, but will see if a fix is needed once CI is done on master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants