Re-design of submodules used in blender.git #104573

Closed
opened 2023-02-10 15:38:26 +01:00 by Sergey Sharybin · 5 comments

This is a proposal to improve the situation with the usage of submodules in Blender.

Introduction

The submodules are designed to be used to track dependencies which do not change that often: for example, an external library. However, our usage requires behavior closer to svn:external where the latest state of an external is being tracked, without state stored in the main repository.

The behavior of Git tooling around submodules and their ignore setting also changed since the initial setup of the submodules done back in 2013.

Back in that time we did not have "smart" make update tool which could have taken care of certain aspects of setup.

There was an attempt to improve the situation aab707ab70, but the initial though of making it more clear what's going on behind the scenes caused even more confusion. People are confused what they need to do with the submodules, and some tools still stage those automatically.

One more thing to mention before jumping to the proposal to help understanding some of the parts of the proposal and the timeline. There is planned project for the year to introduce the Blender Extensions platform, which eventually will take care of community maintained addons.

Proposal

The proposal is divided into 3 different actions.

Addons

Remove addons and addons_contrib submodules, replace them with a Git checkout to a directory within a local working tree, which is ignored by .gitignore.

To avoid conflict the proposal is to check them out to a top level scripts directory. The addons are not part of the release process anyway, and such move will solve the problam of possible conflicts when doing bisect between two points in time before and after the change.

The make update script will take care of initializing the checkout of addons and addons_contrib, and keeping those up-to-date.

The proposal is to not automatically de-initialize the submodule, to avoid possible loss of local branches. If there is a way to automatically safely unregister submodule we can do that instead.

Evenetually, those folders will be removed from the blender.git once the extension platform is in a production state. Additionally, the tricky logic for such checkout will be removed from the make update script.

The downside of this change is that all release builders would need to use make update pipeline for the time being. It is not a problem for us as we already do so, but it is something to clearly communicate to the external projects like Linux distribution.

Locales & Developer tools

The proposal is to move them to the blender.git. They will go to the top level folrders locales and tools respectively. Their submodules will be removed (deinitialized).

The initial idea of the tools being applicable for all releases does not seem to work well, and they are already tagged and packaged at a per-release basis anyway.

The locales are because they are maintained by just Bastien, and they are not too big to warrant the dedicated repo.

Timeline

  • Until the final design is signed-off and implemented, revert the aab707ab70.
  • Write a short instruction for developers and contributors how to make it so changes in submodules do not appear in pull requests.
  • On Tuesday the BF Admins will have a meeting, and will come to a signed-off decision
  • The make update script changes will be made/finished soon after that, and the change to the repository will be implemented.
  • Hopefully the documentation will be as straightforward as git pull && make update. If it is not, it will be written and shared as well.
This is a proposal to improve the situation with the usage of submodules in Blender. # Introduction The submodules are designed to be used to track dependencies which do not change that often: for example, an external library. However, our usage requires behavior closer to `svn:external` where the latest state of an external is being tracked, without state stored in the main repository. The behavior of Git tooling around submodules and their `ignore` setting also changed since the initial setup of the submodules done back in 2013. Back in that time we did not have "smart" `make update` tool which could have taken care of certain aspects of setup. There was an attempt to improve the situation aab707ab70e, but the initial though of making it more clear what's going on behind the scenes caused even more confusion. People are confused what they need to do with the submodules, and some tools still stage those automatically. One more thing to mention before jumping to the proposal to help understanding some of the parts of the proposal and the timeline. There is planned project for the year to introduce the Blender Extensions platform, which eventually will take care of community maintained addons. # Proposal The proposal is divided into 3 different actions. ## Addons Remove `addons` and `addons_contrib` submodules, replace them with a Git checkout to a directory within a local working tree, which is ignored by `.gitignore`. To avoid conflict the proposal is to check them out to a top level `scripts` directory. The addons are not part of the release process anyway, and such move will solve the problam of possible conflicts when doing bisect between two points in time before and after the change. The `make update` script will take care of initializing the checkout of `addons` and `addons_contrib`, and keeping those up-to-date. The proposal is to not automatically de-initialize the submodule, to avoid possible loss of local branches. If there is a way to automatically safely unregister submodule we can do that instead. Evenetually, those folders will be removed from the `blender.git` once the extension platform is in a production state. Additionally, the tricky logic for such checkout will be removed from the `make update` script. The downside of this change is that all release builders would need to use `make update` pipeline for the time being. It is not a problem for us as we already do so, but it is something to clearly communicate to the external projects like Linux distribution. ## Locales & Developer tools The proposal is to move them to the `blender.git`. They will go to the top level folrders `locales` and `tools` respectively. Their submodules will be removed (deinitialized). The initial idea of the tools being applicable for all releases does not seem to work well, and they are already tagged and packaged at a per-release basis anyway. The locales are because they are maintained by just Bastien, and they are not too big to warrant the dedicated repo. # Timeline * Until the final design is signed-off and implemented, revert the aab707ab70e. * Write a short instruction for developers and contributors how to make it so changes in submodules do not appear in pull requests. * On Tuesday the BF Admins will have a meeting, and will come to a signed-off decision * The `make update` script changes will be made/finished soon after that, and the change to the repository will be implemented. * Hopefully the documentation will be as straightforward as `git pull && make update`. If it is not, it will be written and shared as well.
Sergey Sharybin added the
Type
Design
label 2023-02-10 15:38:26 +01:00
Brecht Van Lommel was assigned by Sergey Sharybin 2023-02-10 15:38:27 +01:00
Dalai Felinto was assigned by Sergey Sharybin 2023-02-10 15:38:27 +01:00
Campbell Barton was assigned by Sergey Sharybin 2023-02-10 15:38:27 +01:00
Bastien Montagne was assigned by Sergey Sharybin 2023-02-10 15:38:27 +01:00
Sergey Sharybin self-assigned this 2023-02-10 15:38:27 +01:00

I like most of the idea.

My only remark is that core functionality that happens to be implemented as add-ons (I/O mostly, but also OpenXR and Grease Pencil features) should be brought to inside Blender at some point.

I like most of the idea. My only remark is that core functionality that happens to be implemented as add-ons (I/O mostly, but also OpenXR and Grease Pencil features) should be brought to inside Blender at some point.

Perhaps it would be better to align how the addon directories are handled with the proposed bundled assets #104474?

That is, instead of hosting the addon repositories as "ghost" repos inside of the blender git folder like this:

lib/
  └── assets/
blender/
  ├── addons/
  └── addons_contrib/

We have something like this:

lib/
  ├── assets/
  ├── addons/
  └── addons_contrib/
blender/

With this I think we are more consistent in how we place optional/non-vital addons and data. IE, you can still build and run Blender without the three directories in the lib folder. However you will miss out on features.

It should also make it very clear that these folders are not part of the main blender repo. If we create "ghost" repos inside the main repo with the update script, I think it might be easy for people to forget sometimes and make changes in them expecting them to be picked up when commiting to the main repo.

Perhaps it would be better to align how the addon directories are handled with the proposed bundled assets https://projects.blender.org/blender/blender/pulls/104474? That is, instead of hosting the addon repositories as "ghost" repos inside of the blender git folder like this: ``` lib/ └── assets/ blender/ ├── addons/ └── addons_contrib/ ``` We have something like this: ``` lib/ ├── assets/ ├── addons/ └── addons_contrib/ blender/ ``` With this I think we are more consistent in how we place optional/non-vital addons and data. IE, you can still build and run Blender without the three directories in the `lib` folder. However you will miss out on features. It should also make it very clear that these folders are not part of the main blender repo. If we create "ghost" repos inside the main repo with the update script, I think it might be easy for people to forget sometimes and make changes in them expecting them to be picked up when commiting to the main repo.

Mostly I don't have issues with this proposal although I don't personally have such a problem with submodules either (it seems others do, PR's are often including submodule changes accidentally)


I like most of the idea.

My only remark is that core functionality that happens to be implemented as add-ons (I/O mostly, but also OpenXR and Grease Pencil features) should be brought to inside Blender at some point.

I would prefer to instead have a small number of core add-ons in Blender's main repository, as there can still be advantages in keeping functionality modular & self contained - regarding maintenance, deprecating & upgrading functionality. There may be some exceptions to this where it makes sense to make an add-on built-in, I don't see the need to automatically make all of these core add-ons built-in.

Mostly I don't have issues with this proposal although I don't personally have such a problem with submodules either (it seems others do, PR's are often including submodule changes accidentally) ---- > I like most of the idea. > > My only remark is that core functionality that happens to be implemented as add-ons (I/O mostly, but also OpenXR and Grease Pencil features) should be brought to inside Blender at some point. I would prefer to instead have a small number of core add-ons in Blender's main repository, as there can still be advantages in keeping functionality modular & self contained - regarding maintenance, deprecating & upgrading functionality. There may be some exceptions to this where it makes sense to make an add-on built-in, I don't see the need to automatically make all of these core add-ons built-in.
Author
Owner

That is, instead of hosting the addon repositories as "ghost" repos inside of the blender git folder like this

I am not sure what is the advantage of using lib folder would be, but the disadvantages are:

  • Semantically, add-ons are not libraries
  • Such mix will break another semantic: the lib folder can be seen as a "sparse" checkout of SVN.
  • It is not how a lot of suggestions about implementing svn:external in a Git way.
  • Makes it more tricky to setup IDE and code editors, and do global search when one does Python API changes.
  • Makes it more tricky ensuring the same pyproject.toml is used for both built-in Blender scripts and add-ons. Sure, this is not something we can enforce with the add-ons moved to the extensions platform, but until this is done we need to keep things working in a consistent way.

For Campbell and Dalai:

Indeed i forgot to state in the proposal that the built-in scripts will stay within the scripts directory in the main repository. This includes add-ons maintained by BF. At least in the short term.

In a longer term I think those should become more of a built-in scripts like the bl_operators. It is still possible to keep code organized in a modular manner, but often there is extra maintenance for both us and add-on developers to check that certain other addon is enabled. But, baby steps!

> That is, instead of hosting the addon repositories as "ghost" repos inside of the blender git folder like this I am not sure what is the advantage of using `lib` folder would be, but the disadvantages are: * Semantically, add-ons are not libraries * Such mix will break another semantic: the `lib` folder can be seen as a "sparse" checkout of SVN. * It is not how a lot of suggestions about implementing `svn:external` in a Git way. * Makes it more tricky to setup IDE and code editors, and do global search when one does Python API changes. * Makes it more tricky ensuring the same `pyproject.toml` is used for both built-in Blender scripts and add-ons. Sure, this is not something we can enforce with the add-ons moved to the extensions platform, but until this is done we need to keep things working in a consistent way. --- For Campbell and Dalai: Indeed i forgot to state in the proposal that the built-in scripts will stay within the `scripts` directory in the main repository. This includes add-ons maintained by BF. At least in the short term. In a longer term I think those should become more of a built-in scripts like the `bl_operators`. It is still possible to keep code organized in a modular manner, but often there is extra maintenance for both us and add-on developers to check that certain other addon is enabled. But, baby steps!
Author
Owner

The PR #104755 which implements this design has been merged to blender-v3.5-release and main branches.

The PR #104755 which implements this design has been merged to `blender-v3.5-release` and `main` branches.
Blender Bot added the
Status
Archived
label 2023-02-21 17:27:11 +01:00
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#104573
No description provided.