Gitea server running out of disk space due to archive downloads #29

New Issue

Brecht Van Lommel · 2023-02-13T23:53:28+01:00

Brecht Van Lommel commented

2023-02-13 23:53:28 +01:00

What appears to be happening is that something is downloading a lot of archives. That is a .tar.gz or .bundle as can be found for example on this page by clicking on the "..." menu next to the Git URL.
https://projects.blender.org/blender/blender/

Such packages are being downloaded for various revisions, and each is about 500MB for the Blender repository. Each archive is cached for some time and adds up quickly.

This is supposed to be cleared regularly by cron.archive_cleanup, but that doesn't seem to be working and leaving behind older files.

Even if it was working, generating 100s of GB every day just to discard it is not good either. So we should block whatever is downloading these archives, or at least block it from downloading projects.blender.org/*/*/archive/*.

What appears to be happening is that something is downloading a lot of archives. That is a .tar.gz or .bundle as can be found for example on this page by clicking on the "..." menu next to the Git URL. https://projects.blender.org/blender/blender/ Such packages are being downloaded for various revisions, and each is about 500MB for the Blender repository. Each archive is cached for some time and adds up quickly. This is supposed to be cleared regularly by `cron.archive_cleanup`, but that doesn't seem to be working and leaving behind older files. Even if it was working, generating 100s of GB every day just to discard it is not good either. So we should block whatever is downloading these archives, or at least block it from downloading `projects.blender.org/*/*/archive/*`.

Brecht Van Lommel added the

labels 2023-02-13 23:53:28 +01:00

techknowlogick commented

2023-02-14 00:05:10 +01:00

Are you able to check the logs and see what is triggering these archive generting events? Is it a sewtch crawler, some other automated tooling, or is it user behaviour?

Based on the answers to the above, perhaps some ratelimiting could be added, and known crawlers be blocked from the archive endpoint.

Are you able to check the logs and see what is triggering these archive generting events? Is it a sewtch crawler, some other automated tooling, or is it user behaviour? Based on the answers to the above, perhaps some ratelimiting could be added, and known crawlers be blocked from the archive endpoint.

Brecht Van Lommel commented

2023-02-14 13:50:52 +01:00

Closing as a duplicate of #32.

Brecht Van Lommel closed this issue

2023-02-14 13:50:52 +01:00

Brecht Van Lommel referenced this issue

2023-02-14 13:52:05 +01:00

Gitea: Deal with repo-archives bloat #32

Sign in to join this conversation.