Buildbot: Checksum is sometimes out-of-sync from the file #89

New Issue

Sergey Sharybin · 2024-07-04T10:59:32+02:00

Sergey Sharybin commented

2024-07-04 10:59:32 +02:00

The checksums are stored in a separate file (<build_name>.sha256). They are served via CDN, similar to the actual Blender builds.

The issue comes from the fact that sometimes builds do change without change in the file name. For example, LTS tracks keeps being build every night, or a vdev build can be re-triggered with the new libraries.

This leads to situation when CDN cache is in inconsistent state due to build and checksum being cached separately.
Perhaps there is a way to provide proper cache-control from our side, so that CDN can more immediately react to the changes to the content.

The checksums are stored in a separate file (`<build_name>.sha256`). They are served via CDN, similar to the actual Blender builds. The issue comes from the fact that sometimes builds do change without change in the file name. For example, LTS tracks keeps being build every night, or a vdev build can be re-triggered with the new libraries. This leads to situation when CDN cache is in inconsistent state due to build and checksum being cached separately. Perhaps there is a way to provide proper cache-control from our side, so that CDN can more immediately react to the changes to the content.

Sergey Sharybin added the

Service

Buildbot

label 2024-07-04 10:59:32 +02:00

Bart van der Braak commented

2024-07-15 18:31:03 +02:00

I have only just gotten CDN77 access, so I will compare caching strategies for these endpoints and the responses our origin provide to make this more consistent:

Cache-Control Headers: For example, set the Cache-Control header to no-cache or max-age=0 for the checksum files. This will force the CDN to check for updates each time the file is requested.
Invalidate Cache: Implement a cache invalidation strategy where we programmatically purge the specific files from the CDN cache whenever a new build is pushed.
Use ETags: Utilize ETags (entity tags) for these resources. The CDN can then use these tags to determine whether the cached version is still valid or if it should fetch a new version.
CDN Configuration: E.g. different cache rules for different file types or paths.

I have only just gotten CDN77 access, so I will compare caching strategies for these endpoints and the responses our origin provide to make this more consistent: 1. **Cache-Control Headers**: For example, set the `Cache-Control` header to `no-cache` or `max-age=0` for the checksum files. This will force the CDN to check for updates each time the file is requested. 2. **Invalidate Cache**: Implement a cache invalidation strategy where we programmatically purge the specific files from the CDN cache whenever a new build is pushed. 3. **Use ETags**: Utilize ETags (entity tags) for these resources. The CDN can then use these tags to determine whether the cached version is still valid or if it should fetch a new version. 4. **CDN Configuration**: E.g. different cache rules for different file types or paths.

Bart van der Braak added this to the DevOps Progress Board project 2024-07-16 12:49:22 +02:00

Bart van der Braak self-assigned this 2024-07-16 13:38:35 +02:00

Bart van der Braak commented

2024-07-17 14:43:15 +02:00

This issue was prevalent for download.blender.org. I changed the Nginx configuration as follows to make sure that Cloudflare revalidates if anything changes at the origin:

@@ -170,13 +170,9 @@
             add_header Cache-Control "no-cache, no-store, must-revalidate, proxy-revalidate, max-age=60";
         }
 
-        location ~* \.(xz|msi|msix|exe|gz|bz2|dmg)$ {
-            # Make sure that Cloudflare revalidates such that files don't drift from the .md5sum counterpart
-            add_header Cache-Control "public, no-cache, must-revalidate";
-        }
-
-        location ~* \.(css|js|jpg|jpeg|png|gif|ico|svg|woff|woff2|ttf|otf|mov|avi|zip)$ {
+       location ~* \.(css|js|jpg|jpeg|png|gif|ico|svg|woff|woff2|ttf|otf|mov|avi|zip|xz|msi|msix|exe|gz|bz2|dmg)$ {
             # Set the expiration time to 1 month
+            expires 1M;
             add_header Cache-Control "public, max-age=2592000, immutable";
         }

Using the following sources:

This issue was prevalent for `download.blender.org`. I changed the Nginx configuration as follows to make sure that Cloudflare revalidates if anything changes at the origin: ``` @@ -170,13 +170,9 @@ add_header Cache-Control "no-cache, no-store, must-revalidate, proxy-revalidate, max-age=60"; } - location ~* \.(xz|msi|msix|exe|gz|bz2|dmg)$ { - # Make sure that Cloudflare revalidates such that files don't drift from the .md5sum counterpart - add_header Cache-Control "public, no-cache, must-revalidate"; - } - - location ~* \.(css|js|jpg|jpeg|png|gif|ico|svg|woff|woff2|ttf|otf|mov|avi|zip)$ { + location ~* \.(css|js|jpg|jpeg|png|gif|ico|svg|woff|woff2|ttf|otf|mov|avi|zip|xz|msi|msix|exe|gz|bz2|dmg)$ { # Set the expiration time to 1 month + expires 1M; add_header Cache-Control "public, max-age=2592000, immutable"; } ``` Using the following sources: - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control - https://developers.cloudflare.com/cache/concepts/cache-control/ - https://blog.cloudflare.com/introducing-smart-edge-revalidation/

Bart van der Braak commented

2024-07-24 12:57:05 +02:00

Fixing this issue with revalidation for cdn.builder.blender.org by using:

@@ -27,6 +27,13 @@
                 SetHandler application/x-httpd-php
         </FilesMatch>
 
+        <IfModule mod_headers.c>
+               # Make sure that CDN77 revalidates every 60 sec to prevent cache drift between builds and their shasum counterpart
+                <FilesMatch "\.(xz|msi|msix|exe|gz|bz2|dmg|sha256)$">
+                        Header set Cache-Control "public, max-age=60, must-revalidate"
+                </FilesMatch>
+        </IfModule>
+

Fixing this issue with revalidation for `cdn.builder.blender.org` by using: ``` @@ -27,6 +27,13 @@ SetHandler application/x-httpd-php </FilesMatch> + <IfModule mod_headers.c> + # Make sure that CDN77 revalidates every 60 sec to prevent cache drift between builds and their shasum counterpart + <FilesMatch "\.(xz|msi|msix|exe|gz|bz2|dmg|sha256)$"> + Header set Cache-Control "public, max-age=60, must-revalidate" + </FilesMatch> + </IfModule> + ```

👍 1

Bart van der Braak referenced this issue from infrastructure/blender-buildbot-www

2024-07-24 13:28:10 +02:00

Cache: Add `cache-control` headers with revalidation to ensure checksum consistency #9

Bart van der Braak referenced this issue from a commit

2024-07-25 16:01:35 +02:00

Cache: Add `cache-control` headers with revalidation to ensure checksum consistency (#9)

Bart van der Braak commented

2024-07-25 17:36:52 +02:00

Change has been deployed to UATEST and Production.

Bart van der Braak closed this issue

2024-07-25 17:37:13 +02:00

Oleg-Komarov commented

2024-07-29 18:22:51 +02:00

should .pdf be also included in this rule?

should `.pdf` be also included in this rule?

Oleg-Komarov commented

2024-07-29 19:02:30 +02:00

use case: https://download.blender.org/institute/blender-annual-report-2023-v1.4-web.pdf

Sergey Sharybin commented

2024-07-30 00:29:12 +02:00

@Oleg-Komarov This task is about builds served from builder.b.o, which uses CDN77, which is quite different from download.b.o. Also, some rules on download.b.o has been already tweaked during 4.2 release.
If there are still issues with the download.b.o please submit a new report.

@Oleg-Komarov This task is about builds served from builder.b.o, which uses CDN77, which is quite different from download.b.o. Also, some rules on download.b.o has been already tweaked during 4.2 release. If there are still issues with the download.b.o please submit a new report.

Bart van der Braak reopened this issue

2024-07-30 01:19:53 +02:00

Bart van der Braak closed this issue

2024-07-30 01:20:09 +02:00

Bart van der Braak referenced this issue

2024-07-30 01:27:20 +02:00

Buildbot: Determine appropriate caching strategy for all file extension types served from `download.blender.org` #112

Bart van der Braak commented

2024-07-30 01:28:40 +02:00

Created #112

Sign in to join this conversation.