Migration of libraries to Git LFS #108978
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Viewport & EEVEE
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Asset Browser Project
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Module
Viewport & EEVEE
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Severity
High
Severity
Low
Severity
Normal
Severity
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#108978
Loading…
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This is a quick design task to communicate and coordinate transition of libraries from Subversion to Git LFS.
The are currently the following folders in the
lib
folder of Subversion:Floating repositories
The following repositories are converted from Subversion to Git LFS, but not attached as submodules:
Submodules
The remaining repositories are converted to Git LFS but also are attached to the blender.git as a submodule.
The main difference from out previous usage of submodules is that within this proposal the submodules are actually tracking a specific commit.
This leads to an extra steps when it comes to committing updates: the submodule has in the
blender.git
has to be updated after commit to a submodule, but on another hand, it allows to ensure libraries are tests are at their proper state when bisecting.Directory layout
The following directory layout is proposed:
Saving disk space
It is undesirable that all checkouts of
blender.git
will checkout libraries for all platforms.it seems to be possible to utilize the
update
submodule configuration.From the local prototype the
.gitmodules
could look similar to this (note theupdate = none
for the libraries):With such configuration doing
git submodule update
will skip the libraries:In order to make the libraries updated the following command (or similar to other platform) can be used:
git config --local submodule."lib/linux".update checkout
:Updating after commit to the submodule
When a commit is done to a submodule a parent repository (
blender.git
) needs to be updated as well.The easiest workflow seems to be:
For example, if a new reference image is to be committed it is to be done from
tests/data/files
folders.blender.git
checkout:git add <path/to/module>
.For example,
git add tests/data/files
.git commit
andgit push
as usualUpdate automation
For the
make update
will take care of the following things:update
tocheckout
)Caveats
Which can actually be good in some cases, because it allows to have
main
andblender-v3.6.release
branches to use their own libraries.make update
might put it to a detached head. A manual step to put it to a desired branch before commit might be needed.Plan
blender.git
make update
One issue this does not solve is that pull requests can't include new or updated test files, which means they can't be tested on the buildbot. A solution would be to store those files in the main Blender repository with LFS.
However using LFS for the main repository does imply that you can't easily switch git commits while offline. Through there must be ways to do it, by either downloading the entire history or skipping LFS updates somehow. Actually I guess the same problem exist when using libraries as submodules, that would also require internet access when switching to a commit with libraries changes. But those are updated less frequently.
It would be interesting to hear other's opinions on this. Personally I'd happily make this trade-off to be able to include tests in pull requests, but it's not a change to make lightly.
Also to summarize some of the advantages of this migration:
I'm not experienced enough with git-lfs to make any worth while contribution to this discussion
I do have a concern this setup will break my current workflow, which is like this:
I keep several releases checked out simultaneously in their own folders and symlink
lib
to what i need depending on the branch I am in, as I literally can't afford to wait an hour for the libs to download/update every time I switch a branch. I would get nothing done if i switch between the release branch and main a few times a day.Additional todo: if we're going to have a significant amount of binaries in the source tree, the
make source_archive/source_archive_complete
likely needs some work to exclude these folders.Two potential solutions to that problem:
LIBDIR
CMake variable to a different value for every build folder, and don't use the submodules.We could of course not use submodules too, and instead do what we do for
make update
of addons but with a specified commit hash so somewhere. I'm not sure that's fundamentally different, but maybe there is some benefit to keeping this under more manual control rather than having git switch things for us.@LazyDodo I guess you already not relying on the
make update
, otherwise it will "ruin" the SVN checkout which symlink is pointing to. If that's true, I think there is a third option:update = none
(which is the default), and make itlib/windows_x64
be a symlink to one of git-lfs checkouts done somewhere else. Not sure whatgit status
performed in the parent repository will tell you in this case though :)I have to admit, i'm not married to my workflow, as long as I can quickly (< 5 mins) switch libs I'm fine, HDD space and bandwidth are essentially infinite these days my time however is not. That being said, users on metered connections will likely not share that sentiment.
Started the Blender-side changes needed to implement the proposal: #117946
There are a couple of things which can not easily be submitted as PR:
Attached scripts which I used locally to perform those steps.
The SVN to Git-LFS conversion is done taking the following things in mind:
For the next steps:
The repository names use underscores, but we use dashes for existing repositories. I think we should be consistent.
We also use the
blender-
prefix usually, but I can see the motivation for leaving it our here as it's a bit redundant with the organization and the libs are used by Cycles standalone. Maybe some would still benefit from the prefix, likeblender-test-data.git
andblender-assets.git
?Maybe the tests directory structure could be a bit flatter for convenience?
tests/data
->blender-test-data.git
tests/benchmarks
->benchmarks.git
For these others we could consider:
asset-bundles.git
lib-source.git
icon_geom
inassets.git
?Replacing underscore with dash shouldn't be a bit of a deal.
Could be a bit weird though:
lib-windows-x64
?The tests directory can indeed become flatter.
I've put the scripts to the PR now.
We can also sit together, and do the adjustments. Will be faster than bouncing comments.
While windows libs have been ABI stable ever since vs2015, there is no guarantee it will stay this way,
lib-windows-x64
may be too generic? on the other hand, it hasn't broken in 9 years, we could just cross that bridge when we get there...@LazyDodo For that I though it will be easier and flexible to use branches. Similar to how eventually a newer glibc ABI will replace Linux libraries.
For the ABI, I guess the question is if we ever want to support two ABIs and two sets of libraries at the same time. Probably best to deal with that if it becomes a problem.
For the underscores vs dashes, I forgot to mention that we probably want to keep the folder name with underscore for consistency, just the repo name would use dashes I think. The repo name for libs could also be e.g.
lib-window_x64
.I'm not immediately sure. Currently I don't think we care about these being tightly connected to any particular Git revision, so not having them as a submodule seems most convenient.
Though I can imagine that happening for the asset one in the future, if we rely more on shipping with node groups and testing them. So maybe the asset should be connected, but the benchmarks not? It would be a bit less convenient that a Blender developer would have to do something when studio artists update these. But I guess it's justified.
Sounds good to me. Can also think of it as an alternative to "lib/windows_x64". Not sure it is important or just me seeing patters :)
Or when some node group becomes incompatible with some Blender version, having stronger coupling seems to reduce risk of things getting out of sync.
Unfortunately, it seems that we'd better spent a bit more educated guess now, as changing between "floating module" and git-submodule is not impossible, but is disruptive, especially for bisect.
Perhaps we can have a quick talk with Dalai, Julien, and Simon and see if they feel strongly about having them "floating".
For the benchmarks, it seems easiest to keep it "floating". Will save us from fitting it into the
make_update.py
, we'd just need to git-ignore, and then it will be manual checkout for the initial svn->git switch, and then we can add it to the helper scripts. Kinda to scope things a bit more narrow.One thing I don't really understand, is whether we need to do anything with the
asset_bundles
:Ah, one more thing I forgot to mention: we'd need to set-up permissions on the repositories. Is easy, but we should not forget about it :)
As far as I know it's just files that are available for download here, and not connected to Blender build and packaging.
https://www.blender.org/download/demo-files/
Once the extensions website is online, these assets should become available there. So maybe it's a more a general question, if we have any officially maintained asset extensions, where do we put the Git repository?
Since it's effectively done by the Blender Studio right now, maybe it belongs in the
studio/
organization?