Deployment: Split 'data' storage into seperate pools #34

New Issue

Arnd Marijnissen · 2023-02-14T15:47:07+01:00

Arnd Marijnissen commented

2023-02-14 15:47:07 +01:00

Given the fact that the repo-archive disk filled up super-quick all of a sudden, it's clear that there's quite som potential to DdoS the site by simply uploading a lot, creating a large repo or , in the previous case, doing a lot of download-requests.

It'd be good to split up the current 'data' pool into three pools with different semantics (and storage-requirements). Some of this can be done by using seperate ceph-mounts or by using quota's instead.

database: small, speed critical, uptime is paramount
Repositories-blender: the main repositories that the system's designed to handle, important, large
Repositories-other: The community projects that can be on a different speed and backup regime (optional, can be done later, too)
repo-archive: this is storage that is ephemeral , can be lower-speed or should at least be quota'd.

Given the fact that the repo-archive disk filled up super-quick all of a sudden, it's clear that there's quite som potential to DdoS the site by simply uploading a lot, creating a large repo or , in the previous case, doing a lot of download-requests. It'd be good to split up the current 'data' pool into three pools with different semantics (and storage-requirements). Some of this can be done by using seperate ceph-mounts or by using quota's instead. - database: small, speed critical, uptime is paramount - Repositories-blender: the main repositories that the system's designed to handle, important, large - Repositories-other: The community projects that can be on a different speed and backup regime (optional, can be done later, too) - repo-archive: this is storage that is ephemeral , can be lower-speed or should at least be quota'd.

Arnd Marijnissen added the

deployment

label 2023-02-14 15:47:07 +01:00

Brecht Van Lommel commented

2023-02-14 16:09:31 +01:00

One thing I wonder about is if blender and other repositories are in separate pools, are hard links between them still possible to avoid every fork taking up space? Or are hard links already not working?

Matti-Ranta commented

2023-02-15 03:58:44 +01:00

One thing I wonder about is if blender and other repositories are in separate pools, are hard links between them still possible to avoid every fork taking up space?

With gitea.com, hardlinks across pools did not work, although that could just be because we didn't look into it much and it may have well been possible with appropriate tuning.

For repo archive, issue attachments, LFS etc.. you may wish to look into using ceph's radosgw and using their S3 compatible API and https://docs.gitea.io/en-us/config-cheat-sheet/#storage-storage that way you can manage that specific storage outside of VM config.

S3 storage can't be used for the issues search index, code search index, and git repos themselves (and a few other minor things).

> One thing I wonder about is if blender and other repositories are in separate pools, are hard links between them still possible to avoid every fork taking up space? With gitea.com, hardlinks across pools did not work, although that could just be because we didn't look into it much and it may have well been possible with appropriate tuning. For repo archive, issue attachments, LFS etc.. you may wish to look into using ceph's radosgw and using their S3 compatible API and https://docs.gitea.io/en-us/config-cheat-sheet/#storage-storage that way you can manage that specific storage outside of VM config. S3 storage can't be used for the issues search index, code search index, and git repos themselves (and a few other minor things).

Sign in to join this conversation.