Self-hosting external libraries packages #86124

Open
opened 2021-03-01 12:14:35 +01:00 by Dalai Felinto · 30 comments

This is a project/proposal to self-host all (most anyways) libraries Blender links to (png, jpg, ...).

This makes Blender less dependent on external hosting solutions. It also makes it easy to comply to GPL source code request.

  • Host all packages in the svn
  • make deps to support that
  • Create a new make source_archive_complete to include those
  • make deps from the downloaded full package to also work with the local packages

Note: make source_archive_complete should create the following compressed file structure:

$ tar -Jtvf blender-with-libraries-2.92.0.tar.xz

*blender_2.92.0/*blender_2.92.0/source/blender/blenkernel
(...)
*packages/*packages/tiff-4.1.0.tar.gz
(...)
This is a project/proposal to self-host all (most anyways) libraries Blender links to (png, jpg, ...). This makes Blender less dependent on external hosting solutions. It also makes it easy to comply to GPL source code request. * Host all packages in the svn * `make deps` to support that * Create a new `make source_archive_complete` to include those * `make deps` from the downloaded full package to also work with the local packages --- Note: `make source_archive_complete` should create the following compressed file structure: $ `tar -Jtvf blender-with-libraries-2.92.0.tar.xz` ``` *blender_2.92.0/*blender_2.92.0/source/blender/blenkernel (...) *packages/*packages/tiff-4.1.0.tar.gz (...) ```
Author
Owner

Changed status from 'Needs Triage' to: 'Confirmed'

Changed status from 'Needs Triage' to: 'Confirmed'
Author
Owner

Added subscriber: @dfelinto

Added subscriber: @dfelinto
Ray molenkamp was assigned by Dalai Felinto 2021-03-01 16:32:37 +01:00
Author
Owner

I talked to Ray, he will take from here.

I talked to Ray, he will take from here.
Author
Owner

Also, @LazyDodo I prepared some cleanup patches: P2005. They are all yours now ;)

Also, @LazyDodo I prepared some cleanup patches: [P2005](https://archive.blender.org/developer/P2005.txt). They are all yours now ;)
Author
Owner

Added subscriber: @ponderz

Added subscriber: @ponderz
Author
Owner

James wants to keep an eye on it since he will incorporate make package_archive_full to the auto-deployment rig once this is available.

James wants to keep an eye on it since he will incorporate `make package_archive_full` to the auto-deployment rig once this is available.
Author
Owner

Just to add some extra suggestions that are related to the issue here.
As far as the external world is concerned, those changes mean Blender will make available:

My proposal is that on major updates (2.90.0, 2.91.0) both packages are provided. But on incremental minor updates (2.90.1, 2.91.3) only the small package is provided (unless in the rare case where a critical bug was fixed in a library).

Just to add some extra suggestions that are related to the issue here. As far as the external world is concerned, those changes mean Blender will make available: * https://download.blender.org/source/blender-2.92.0.tar.xz (40MB) * https://download.blender.org/source/blender-with-libraries-2.92.0.tar.xz (~2GB) - not yet there My proposal is that on major updates (2.90.0, 2.91.0) both packages are provided. But on incremental minor updates (2.90.1, 2.91.3) only the small package is provided (unless in the rare case where a critical bug was fixed in a library).
Member

Just as a sanity check you are not expecting people to download this tarball, run some magic make command and end up with a fully functional blender i hope? Cause that's not the sort of thing we can support.

This may not be common knowledge, but make deps is made by our platform devs, for our platform devs, it's sole purpose is to populate our SVN repo. its target audience is only our platform devs, we offer no support on it and it's highly sensitive to our build environment. if we want to populate the SVN repo with the tar-balls of the 3rd party libs for compliance purposes that be no problem, but it seems unlikely at this point we can/should expand its intended target audience beyond what it currently is.

Just as a sanity check you are not expecting people to download this tarball, run some magic make command and end up with a fully functional blender i hope? Cause that's not the sort of thing we can support. This may not be common knowledge, but `make deps` is made *by* our platform devs, *for* our platform devs, it's sole purpose is to populate our SVN repo. its target audience is only our platform devs, we offer no support on it and it's highly sensitive to *our* build environment. if we want to populate the SVN repo with the tar-balls of the 3rd party libs for compliance purposes that be no problem, but it seems unlikely at this point we can/should expand its intended target audience beyond what it currently is.
Author
Owner
  • I would like make deps to work as well as it does now for the platform maintainers.
  • I would love if make deps would work out of the box in my Ubuntu computer, but it is a separate topic.
* I would like `make deps` to work as well as it does now for the platform maintainers. * I would love if `make deps` would work out of the box in my Ubuntu computer, but it is a separate topic.
Member

I would like make deps to work as well as it does now for the platform maintainers.

I'm unsure what you mean?

I would love if make deps would work out of the box in my Ubuntu computer, but it is a separate topic.

I am unlikely to let make deps grow into a second install_deps.sh by trying to target every possible environment on the planet, it's working well for us now due to its limited scope, and doing a full run of lib updates is still a multi week multi person endeavor.

> I would like make deps to work as well as it does now for the platform maintainers. I'm unsure what you mean? >I would love if make deps would work out of the box in my Ubuntu computer, but it is a separate topic. I am unlikely to let make deps grow into a second `install_deps.sh` by trying to target every possible environment on the planet, it's working well for us now due to its limited scope, and doing a full run of lib updates is still a multi week multi person endeavor.
Author
Owner

What I meant with my first point is that I don't expect make deps to be easier to use in other platforms just because we are distributing tarballs. But I also don't expect it to be harder to use with the tarballs than make deps is at the moment either.

What I meant with my first point is that I don't expect `make deps` to be easier to use in other platforms just because we are distributing tarballs. But I also don't expect it to be harder to use with the tarballs than `make deps` is at the moment either.
Member

allright! i'll get on it!

allright! i'll get on it!
Member

Added subscriber: @dr.sybren

Added subscriber: @dr.sybren
Member

Initial diff in D10598, I prefer to keep the review there on the technical aspects only, lets keep any non technical discussion here.

makes Blender less dependent on external hosting solutions.

That's a little bit of a stretch, we rely on external hosting when we build the deps, storing the copy that we downloaded at that point in SVN afterwards changes nothing given we still need to get that initial copy.
The only time this is valid is if we ever need to rebuild the libs for an older blender version and some of the library sources are no longer available, seems like a very niche use-case and if this was the sole buy-in, i would have likely not have gone along with this.

It also makes it easy to comply to GPL source code request.

Now this I buy and support. (tip: lead with the better arguments next time)

Host all packages in the svn

I winged the URI where the packages will be in SVN, may need some tweaking and perhaps some checking with operations (is there an official team there?) the initial set will be around 550 MB also unsure if we want just a giant folder with all sources or split it out by blender version?

make deps to support that

Done, but see note above regarding the URL

Create a new make package_archive_full to include those

i'm not familiar with package_archive unsure what this does, source_archive_full seems to make sense in my head, but no idea what package_archive does either way makefiles are not my forte and @dr.sybren will have to assist here.

make deps from the downloaded full package to also work with the local packages

Done.

Initial diff in [D10598](https://archive.blender.org/developer/D10598), I prefer to keep the review there on the technical aspects only, lets keep any non technical discussion here. >makes Blender less dependent on external hosting solutions. That's a little bit of a stretch, we rely on external hosting when we build the deps, storing the copy that we downloaded at that point in SVN afterwards changes nothing given we still need to get that initial copy. The only time this is valid is if we ever need to rebuild the libs for an older blender version and some of the library sources are no longer available, seems like a very niche use-case and if this was the sole buy-in, i would have likely not have gone along with this. >It also makes it easy to comply to GPL source code request. Now this I buy and support. (tip: lead with the better arguments next time) > Host all packages in the svn I winged the URI where the packages will be in SVN, may need some tweaking and perhaps some checking with operations (is there an official team there?) the initial set will be around 550 MB also unsure if we want just a giant folder with all sources or split it out by blender version? > make deps to support that Done, but see note above regarding the URL >Create a new make package_archive_full to include those i'm not familiar with `package_archive` unsure what this does, `source_archive_full` seems to make sense in my head, but no idea what `package_archive` does either way makefiles are not my forte and @dr.sybren will have to assist here. >make deps from the downloaded full package to also work with the local packages Done.
Author
Owner
  1. It was indeed source_archive I was referring to. Which is the one we distribute. Sorry about the confusion (updated the task description).

  2. I will check with operations what would be the impact in the SVN server to confirm that it can handle it. I believe it will be way more than 550MB though. I got ~500MB with a third of the libraries.

1. It was indeed `source_archive` I was referring to. Which is the one we distribute. Sorry about the confusion (updated the task description). 2. I will check with operations what would be the impact in the SVN server to confirm that it can handle it. I believe it will be way more than 550MB though. I got ~500MB with a third of the libraries.
Member

Oh yeah the windows USD libs are bigger than that :) still it be nice to ping them given we're doing something new

Oh yeah the windows USD libs are bigger than that :) still it be nice to ping them given we're doing something new
Member

Added subscriber: @dmcgrath

Added subscriber: @dmcgrath
Member

In #86124#1123228, @dfelinto wrote:
2. I will check with operations what would be the impact in the SVN server to confirm that it can handle it. I believe it will be way more than 550MB though. I got ~500MB with a third of the libraries.

As far as the server storage is concerned, SVN is a rather small jail running on the host, and should have plenty of space for 500MB of files (~1-2TB free iirc).

It's more the backups that are a concern, but 500mb is not a problem. It's only when you get into daily GB's of change that the current system is squeezed tight due to deltas, but we have a new system and HDD's waiting to go into the DC to solve that in the near future.

> In #86124#1123228, @dfelinto wrote: > 2. I will check with operations what would be the impact in the SVN server to confirm that it can handle it. I believe it will be way more than 550MB though. I got ~500MB with a third of the libraries. As far as the server storage is concerned, SVN is a rather small jail running on the host, and should have plenty of space for 500MB of files (~1-2TB free iirc). It's more the backups that are a concern, but 500mb is not a problem. It's only when you get into daily GB's of change that the current system is squeezed tight due to deltas, but we have a new system and HDD's waiting to go into the DC to solve that in the near future.
Member

For the source packages i'm expecting to be around 500-700 megs per release, likely arriving all at once so backups may be an issue there. however....

For the current library use the same problem arises (but worse), with the libs tightly coupled you end up updating a bunch of them at a time leading to a few hundred megs to a few GB spikes and then long periods of no activity at all

For the source packages i'm expecting to be around 500-700 megs per release, likely arriving all at once so backups may be an issue there. however.... For the current library use the same problem arises (but worse), with the libs tightly coupled you end up updating a bunch of them at a time leading to a few hundred megs to a few GB spikes and then long periods of no activity at all
Member

In #86124#1123401, @LazyDodo wrote:
For the source packages i'm expecting to be around 500-700 megs per release, likely arriving all at once so backups may be an issue there. however....

This is fine. It's only when you have 50G a day that get into a daily pool that sticks around for a month that you start to run into issues. Even then, the backup settings can be tweaked, but the downside is that Bacula requires a full run every time you change it.

For the current library use the same problem arises (but worse), with the libs tightly coupled you end up updating a bunch of them at a time leading to a few hundred megs to a few GB spikes and then long periods of no activity at all

Again, not a problem for once a month or so type stuff, so long as it's not daily. If you do things like with docs where 50G a day changes, and you don't exclude it from backups (since it's generated anyway), you end up with 50G x 30 days = ~ 1.5TB, which is almost as big as the entire backup pool, currently.

> In #86124#1123401, @LazyDodo wrote: > For the source packages i'm expecting to be around 500-700 megs per release, likely arriving all at once so backups may be an issue there. however.... This is fine. It's only when you have 50G a day that get into a daily pool that sticks around for a month that you start to run into issues. Even then, the backup settings can be tweaked, but the downside is that Bacula requires a full run every time you change it. > For the current library use the same problem arises (but worse), with the libs tightly coupled you end up updating a bunch of them at a time leading to a few hundred megs to a few GB spikes and then long periods of no activity at all Again, not a problem for once a month or so type stuff, so long as it's not daily. If you do things like with docs where 50G a day changes, and you don't exclude it from backups (since it's generated anyway), you end up with 50G x 30 days = ~ 1.5TB, which is almost as big as the entire backup pool, currently.
Member

Again, not a problem for once a month or so type stuff, so long as it's not daily. If you do things like with docs where 50G a day changes,

oh my, no, it will be nothing like that

>Again, not a problem for once a month or so type stuff, so long as it's not daily. If you do things like with docs where 50G a day changes, oh my, no, it will be nothing like that
Author
Owner
  • I have 2.83 and 2.92 ready to be uploaded (if you don't mind that I did them mostly automatically but partially manually). [they are indeed ~550MB]
  • I have all the 2.93 libraries ready to be svn committed if you rather me to do it from here (assuming we don't change their names).
  • For 2.93 onwards we still need the make source_archive_full.
  • New libraries (after 2.93) should remove the previous one in svn - so at any given time we only have what is required by master.
* I have 2.83 and 2.92 ready to be uploaded (if you don't mind that I did them mostly automatically but partially manually). [they are indeed ~550MB] * I have all the 2.93 libraries ready to be svn committed if you rather me to do it from here (assuming we don't change their names). * For 2.93 onwards we still need the `make source_archive_full`. * New libraries (after 2.93) should remove the previous one in svn - so at any given time we only have what is required by master.

This issue was referenced by d07a7697cf

This issue was referenced by d07a7697cfb064f5129469018082f43a8a86d92f

Changed status from 'Confirmed' to: 'Resolved'

Changed status from 'Confirmed' to: 'Resolved'
Author
Owner

Changed status from 'Resolved' to: 'Confirmed'

Changed status from 'Resolved' to: 'Confirmed'
Author
Owner

Leaving this open until D10727 is committed.

Leaving this open until [D10727](https://archive.blender.org/developer/D10727) is committed.

This issue was referenced by b8479a70c9

This issue was referenced by b8479a70c936c5c59f199a847d679c80196e6c0b

Added subscriber: @brecht

Added subscriber: @brecht

Can we document this? The appropriate wiki page is here:
https://wiki.blender.org/wiki/Building_Blender/Dependencies

Two things are unclear to me:

  • The policy is for library upgrades. Does bumping a version in versions.cmake need to go along with putting a new source package in svn?
  • How to build with these packages? Seems like you'd need to symlink from the build to the lib/packages folder or something? Or are these packages not intended for building, just for bundling in a source archive?
Can we document this? The appropriate wiki page is here: https://wiki.blender.org/wiki/Building_Blender/Dependencies Two things are unclear to me: * The policy is for library upgrades. Does bumping a version in `versions.cmake` need to go along with putting a new source package in svn? * How to build with these packages? Seems like you'd need to symlink from the build to the lib/packages folder or something? Or are these packages not intended for building, just for bundling in a source archive?
Member

The policy is for library upgrades. Does bumping a version in versions.cmake need to go along with putting a new source package in svn?

Yes, and removal of the old package

How to build with these packages? Seems like you'd need to symlink from the build to the lib/packages folder or something? Or are these packages not intended for building, just for bundling in a source archive?

As of now it's just for bundling, Imeant to do a small tweak to the build scripts to detect the archives in the source bundle and use them, but given it's just a syntactic sugar for the mythical user that builds blender and all deps from source tarball it was pretty low on my prio list and have not gotten to it for 2.93

This whole change is just for compliance reasons, make deps 's target audience is still just our platforms devs, and as of now i'm not aware of any plans to widen that audience.

> The policy is for library upgrades. Does bumping a version in versions.cmake need to go along with putting a new source package in svn? Yes, and removal of the old package > How to build with these packages? Seems like you'd need to symlink from the build to the lib/packages folder or something? Or are these packages not intended for building, just for bundling in a source archive? As of now it's just for bundling, Imeant to do a small tweak to the build scripts to detect the archives in the source bundle and use them, but given it's just a syntactic sugar for the mythical user that builds blender and all deps from source tarball it was pretty low on my prio list and have not gotten to it for 2.93 This whole change is just for compliance reasons, `make deps` 's target audience is still just our platforms devs, and as of now i'm not aware of any plans to widen that audience.
Philipp Oeser removed the
Interest
Platforms, Builds & Tests
label 2023-02-10 08:58:24 +01:00
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
6 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#86124
No description provided.