Buildbot: Determine appropriate caching strategy for all file extension types served from download.blender.org #112

New Issue

Bart van der Braak · 2024-07-30T01:27:20+02:00

2024-07-30 01:27:20 +02:00

Optimize the caching strategy for files served from download.blender.org to ensure efficient loading times and resource management. This task stems from an issue where the Blender Annual Report PDF file is not being cached properly:

[pablovazquez]
is the annual report like really heavy? It's taking a long while to load here https://download.blender.org/institute/blender-annual-report-2023-v1.4-web.pdf

[Oleg-Komarov]
it appears that it is not being cached on CloudFlare:
cache-control: no-cache, no-store, must-revalidate, proxy-revalidate, max-age=60
cf-cache-status: BYPASS

[pablovazquez]
Maybe we want to cache it, it'll get a bit more traffic now that is available in the homepage of b.o

[Oleg-Komarov]
looks related #89 (comment)

Optimize the caching strategy for files served from `download.blender.org` to ensure efficient loading times and resource management. This task stems from an issue where the Blender Annual Report PDF file is not being cached properly: > **[pablovazquez]** > is the annual report like really heavy? It's taking a long while to load here https://download.blender.org/institute/blender-annual-report-2023-v1.4-web.pdf > **[Oleg-Komarov]** > it appears that it is not being cached on CloudFlare: > ``` > cache-control: no-cache, no-store, must-revalidate, proxy-revalidate, max-age=60 > cf-cache-status: BYPASS > ``` > **[pablovazquez]** > Maybe we want to cache it, it'll get a bit more traffic now that is available in the homepage of b.o > **[Oleg-Komarov]** > looks related https://projects.blender.org/infrastructure/blender-projects-platform/issues/89#issuecomment-1253718 >

Bart van der Braak changed title from ~~Buildbot: Retrieve served file extensions `builder.blender.org`, determine caching strategy with appropriate defaults~~ to Buildbot: Determine appropriate caching strategy for all file extension types served from `builder.blender.org`

2024-07-30 01:28:04 +02:00

Bart van der Braak referenced this issue

2024-07-30 01:28:48 +02:00

Buildbot: Checksum is sometimes out-of-sync from the file #89

Bart van der Braak added this to the DevOps Progress Board project 2024-07-30 01:38:53 +02:00

Bart van der Braak changed title from ~~Buildbot: Determine appropriate caching strategy for all file extension types served from `builder.blender.org`~~ to Buildbot: Determine appropriate caching strategy for all file extension types served from `download.blender.org`

2024-07-30 10:02:01 +02:00

Bart van der Braak commented

2024-07-30 12:41:32 +02:00

$ find . -type f | awk -F. '{if (NF>1) print $NF}' | sort | uniq -c | sort -nr
12500 png
12226 html
5752 jpg
2164 tga
 984 c
 888 zip
 419 txt
 382 bz2
 355 pdf
 320 gz
 294 h
 232 wiki
 230 xz
 227 dmg
 220 tif
 211 md5sum
 197 gif
 185 blend
 161 md5
 137 JPG
 134 avi
 133 msi
 131 exe
 110 sha256
 109 mov
 100 htm
  95 bak
  91 js
  88 msix
  69 wml
  69 srt
  56 tgz
  52 deb
  49 css
  36 tbz
  33 mp4
  33 cpp
  31 ogg
  27 7z
  24 Z
  23 Extra
  22 mkv
  19 mpg
  19 flv
  15 mp3
  14 svg
  13 ttf
  13 ogv
  13 odp
  12 webp
  12 swf
  12 doc
  12 aif
  11 rpm
   8 xvpics/CVS/Root
   8 xvpics/CVS/Repository
   8 xvpics/CVS/Entries
   8 apk
   7 woff
   7 wav
   7 pptx
   7 json
   7 jpeg
   7 eps
   7 eot
   7 dll
   7 buildinfo
   7 1
   6 xcf
   6 py
   6 ppt
   6 m4v
   6 exr
   6 dvd
   6 dsc
   6 changes
   6 DS_Store
   5 webm
   5 st
   5 ac3
   4 sh
   4 log
   4 ai
   4 MOV
   3 wml#
   3 tar
   3 conf
   3 asc
   3 Bpib
   2 xml
   2 woff2
   2 wat
   2 vms
   2 stroke
   2 sas
   2 rtf
   2 rar
   2 qt
   2 pl
   2 php
   2 pas
   2 odt
   2 mc6
   2 map
   2 manx
   2 mak
   2 lib
   2 jar
   2 iso
   2 inv
   2 inc
   2 ico
   2 ho
   2 flac
   2 dmp
   2 dj
   2 divx
   2 cfg
   2 cab
   2 bcc
   2 Makefile
   2 EXE
   1 xpi
   1 usdc
   1 unix
   1 txt#
   1 torrent
   1 tiff
   1 sxi
   1 svgz
   1 status
   1 ssh/authorized_keys
   1 signature
   1 sha256sum
   1 rc
   1 ppm
   1 pn
   1 pc
   1 patch
   1 opt
   1 ods
   1 odg
   1 obj
   1 new
   1 mxf
   1 msg
   1 mms
   1 lyx#
   1 kdenlive
   1 jpe
   1 hdr
   1 fd
   1 dpx
   1 docx
   1 def
   1 db
   1 bphys
   1 asm
   1 ansi
   1 TXT
   1 SHA512
   1 PNG
   1 MTS
   1 Log
   1 LNK
   1 EPS
   1 Bfs
   1 Bfont
   1 BCkey
...

``` $ find . -type f | awk -F. '{if (NF>1) print $NF}' | sort | uniq -c | sort -nr 12500 png 12226 html 5752 jpg 2164 tga 984 c 888 zip 419 txt 382 bz2 355 pdf 320 gz 294 h 232 wiki 230 xz 227 dmg 220 tif 211 md5sum 197 gif 185 blend 161 md5 137 JPG 134 avi 133 msi 131 exe 110 sha256 109 mov 100 htm 95 bak 91 js 88 msix 69 wml 69 srt 56 tgz 52 deb 49 css 36 tbz 33 mp4 33 cpp 31 ogg 27 7z 24 Z 23 Extra 22 mkv 19 mpg 19 flv 15 mp3 14 svg 13 ttf 13 ogv 13 odp 12 webp 12 swf 12 doc 12 aif 11 rpm 8 xvpics/CVS/Root 8 xvpics/CVS/Repository 8 xvpics/CVS/Entries 8 apk 7 woff 7 wav 7 pptx 7 json 7 jpeg 7 eps 7 eot 7 dll 7 buildinfo 7 1 6 xcf 6 py 6 ppt 6 m4v 6 exr 6 dvd 6 dsc 6 changes 6 DS_Store 5 webm 5 st 5 ac3 4 sh 4 log 4 ai 4 MOV 3 wml# 3 tar 3 conf 3 asc 3 Bpib 2 xml 2 woff2 2 wat 2 vms 2 stroke 2 sas 2 rtf 2 rar 2 qt 2 pl 2 php 2 pas 2 odt 2 mc6 2 map 2 manx 2 mak 2 lib 2 jar 2 iso 2 inv 2 inc 2 ico 2 ho 2 flac 2 dmp 2 dj 2 divx 2 cfg 2 cab 2 bcc 2 Makefile 2 EXE 1 xpi 1 usdc 1 unix 1 txt# 1 torrent 1 tiff 1 sxi 1 svgz 1 status 1 ssh/authorized_keys 1 signature 1 sha256sum 1 rc 1 ppm 1 pn 1 pc 1 patch 1 opt 1 ods 1 odg 1 obj 1 new 1 mxf 1 msg 1 mms 1 lyx# 1 kdenlive 1 jpe 1 hdr 1 fd 1 dpx 1 docx 1 def 1 db 1 bphys 1 asm 1 ansi 1 TXT 1 SHA512 1 PNG 1 MTS 1 Log 1 LNK 1 EPS 1 Bfs 1 Bfont 1 BCkey ... ```

Bart van der Braak added the

Service

Buildbot

label 2024-08-01 11:00:11 +02:00

Bart van der Braak added the

Type

Config

label 2024-08-01 11:23:10 +02:00

Bart van der Braak closed this issue

2024-08-14 10:31:37 +02:00

Bart van der Braak reopened this issue

2024-08-14 10:33:10 +02:00