Extensions: Deploying add-ons for offline work and automation #122512

Closed
opened 2024-05-30 19:23:29 +02:00 by Dalai Felinto · 18 comments

Motivation

There are situations where downloading extensions from Blender is not possible or practical.

There are various existing inconveniences with this, and with the removal of some add-ons from shipping with Blender, this has become a more pressing issue.

Broadly there are two use cases.

Animation Studios

  • Workstation machines are offline
  • Want to automatically deploy updates and bug fixes to everyone
  • Any software must be approved, security checked and installed by the IT department
  • Blender version and add-ons are locked for each show, with ability to switch between shows
  • User manually configuring things is error prone, want to avoid debugging those problems
  • Environment variables are commonly used to centrally configure VFX software

Courses and schools

  • Machines may be offline or online
  • Want to have a shared set of add-ons and other configuration
  • Varying levels of automation, some manual install steps typically ok

Previous Workflows

Manual

  • Share add-on files with users
  • Tell users to install and enable them

Shared Script Directories

  • Deploy add-ons in a script directory on a shared, read-only network drive
  • May be automated with a script that checks and unzips some files
  • Tell users to add script directories in preferences

Current Workflows with Extensions

Manual

  • Create add-ons bundle zip file (what are the exact steps?)
  • Share add-ons bundle zip with users
  • Tell users to install bundle, by default will all be enabled.

Probably fine for courses when it works. Just not entirely clear how to create those bundles, maybe needs docs or an operator to create a bundle from a repository.

Could be some value in making application templates easy to create and share like this, if more than just add-ons are needed.

Shared Extension Repository

  • Create shared repository folder on a shared, read-only network drive
  • For automated deployment:
    • Install Blender on deployment machine
    • Script installation of extension zips to this repository
      • Is this possible through command line?
  • Tell users to add this shared repository in preferences
    • Does this work well with read-only network drive?
    • Note this would not be like an online repository from which you install updates

Running Blender as part of deployment is not ideal, and may be a deal breaker. This workflow is also not documented and obvious.

Potential Workflows

Shared Script Directories

  • Same as before, but make it work for extensions. (#122688)
  • Make it work properly with environment variables. (#122689)
  • May consider a mechanism without environment variables, like a bundle directory next to the Blender executable.

Some inconvenience since extension zips have an uncommon compression, and may or may not include the a folder. But not terribly difficult to automate, even if we don't change the extensions spec to simplify this.

This is the simplest for studios that are already using Blender in a way, since it's basically the old mechanism that they may already be using. And it means that in the transition period, there is no need to deal with the distinction between legacy and new add-ons. But we may want to get rid of script addons directories in the future if it's considered part of what is legacy?

Note that script directories are also used for presets, startup and application templates currently. So it may not get fully replaced by extensions anyway.

Shared Extension Repository

  • Support installing extension without running Blender
    • Can a repository be made to work without an index file.
  • Make sure these read-only repositories work well.
    • Users should not be able to edit them.
    • User should not need to install updates manually.
  • Support adding repositories through environment variable for central configuration.
## Motivation There are situations where downloading extensions from Blender is not possible or practical. There are various existing inconveniences with this, and with the removal of some add-ons from shipping with Blender, this has become a more pressing issue. Broadly there are two use cases. ### Animation Studios * Workstation machines are offline * Want to automatically deploy updates and bug fixes to everyone * Any software must be approved, security checked and installed by the IT department * Blender version and add-ons are locked for each show, with ability to switch between shows * User manually configuring things is error prone, want to avoid debugging those problems * Environment variables are commonly used to centrally configure VFX software ### Courses and schools * Machines may be offline or online * Want to have a shared set of add-ons and other configuration * Varying levels of automation, some manual install steps typically ok ## Previous Workflows ### Manual * Share add-on files with users * Tell users to install and enable them ### Shared Script Directories * Deploy add-ons in a script directory on a shared, read-only network drive * May be automated with a script that checks and unzips some files * Tell users to add script directories in preferences ## Current Workflows with Extensions ### Manual * Create add-ons bundle zip file (what are the exact steps?) * Share add-ons bundle zip with users * Tell users to install bundle, by default will all be enabled. Probably fine for courses when it works. Just not entirely clear how to create those bundles, maybe needs docs or an operator to create a bundle from a repository. Could be some value in making application templates easy to create and share like this, if more than just add-ons are needed. ### Shared Extension Repository * Create shared repository folder on a shared, read-only network drive * For automated deployment: * Install Blender on deployment machine * Script installation of extension zips to this repository * Is this possible through command line? * Tell users to add this shared repository in preferences * Does this work well with read-only network drive? * Note this would not be like an online repository from which you install updates Running Blender as part of deployment is not ideal, and may be a deal breaker. This workflow is also not documented and obvious. ## Potential Workflows ### Shared Script Directories * Same as before, but make it work for extensions. (#122688) * Make it work properly with environment variables. (#122689) * May consider a mechanism without environment variables, like a `bundle` directory next to the Blender executable. Some inconvenience since extension zips ~~have an uncommon compression, and~~ may or may not include the a folder. But not terribly difficult to automate, even if we don't change the extensions spec to simplify this. This is the simplest for studios that are already using Blender in a way, since it's basically the old mechanism that they may already be using. And it means that in the transition period, there is no need to deal with the distinction between legacy and new add-ons. But we may want to get rid of script addons directories in the future if it's considered part of what is legacy? Note that script directories are also used for presets, startup and application templates currently. So it may not get fully replaced by extensions anyway. ### Shared Extension Repository * [x] Support installing extension without running Blender * [x] Can a repository be made to work without an index file. * [ ] Make sure these read-only repositories work well. * Users should not be able to edit them. * User should not need to install updates manually. * [x] Support adding repositories through environment variable for central configuration.
Dalai Felinto added the
Type
To Do
label 2024-05-30 19:23:29 +02:00
Dalai Felinto added this to the Python API project 2024-05-30 19:23:30 +02:00
Brecht Van Lommel was assigned by Dalai Felinto 2024-05-30 19:23:41 +02:00

I'm thinking perhaps this doesn't have to be an actual repository, but work more like an additional script path like the core or legacy add-ons. And showing "Bundled:" instead of "Core:" in the list. It seems simpler not having to manage some virtual repository and add various exceptions for that throughout the code.

Also thinking perhaps it's not worth supporting the case where you dump a bunch of zip files in a directory and Blender extracts them for you. But less sure about that. The main thing that makes the zip files inconvenient is that they are not guaranteed to contain a folder, so files can conflict. If they did it wouldn't be as big an ask to just unzip them in place.

Some further thoughts and request for feedback on devtalk.
https://devtalk.blender.org/t/changes-to-add-on-and-themes-bundling-4-2-onwards/34593/166?u=brecht

I'm thinking perhaps this doesn't have to be an actual repository, but work more like an additional script path like the core or legacy add-ons. And showing "Bundled:" instead of "Core:" in the list. It seems simpler not having to manage some virtual repository and add various exceptions for that throughout the code. Also thinking perhaps it's not worth supporting the case where you dump a bunch of zip files in a directory and Blender extracts them for you. But less sure about that. The main thing that makes the zip files inconvenient is that they are not guaranteed to contain a folder, so files can conflict. If they did it wouldn't be as big an ask to just unzip them in place. Some further thoughts and request for feedback on devtalk. https://devtalk.blender.org/t/changes-to-add-on-and-themes-bundling-4-2-onwards/34593/166?u=brecht
Author
Owner

Re: zips: we could always guarantee a folder when unzipping, and making the folder name match the extension_id. I believe this is what we do when installing extensions from disk.

Re: zips: we could always guarantee a folder when unzipping, and making the folder name match the extension_id. I believe this is what we do when installing extensions from disk.

To be clear, are you talking about making this folder a requirement for all extension zips? That's what would be needed to make unzipping it manually or through a script easy. In our own code we can and do check this of course, that's not really an issue.

To be clear, are you talking about making this folder a requirement for all extension zips? That's what would be needed to make unzipping it manually or through a script easy. In our own code we can and do check this of course, that's not really an issue.
Author
Owner

@brecht I see what you mean, but I'm not proposing any changes to the extension zips. They can be in a subfolder or flat inside the .zip file. I was mentioning what to do when unzipping them.

I think it makes sense to follow the same principles we use for the installed from disk. A folder inside the .zip is not required, but the moment you have the extension unzipped you have:

  • Each extension goes to a folder.
  • The folder names has to match the extension id.

You can't have different extensions outside a folder anyways because their manifest would clash.

PS.: If we want to support manual unzipping as part of this workflow, we should also stop using that special compression flag when creating extensions using Blender cli. Which I think we should anyways if you ask me.

@brecht I see what you mean, but I'm not proposing any changes to the extension zips. They can be in a subfolder or flat inside the .zip file. I was mentioning what to do when unzipping them. I think it makes sense to follow the same principles we use for the installed from disk. A folder inside the .zip is not required, but the moment you have the extension unzipped you have: * Each extension goes to a folder. * The folder names has to match the extension id. You can't have different extensions outside a folder anyways because their manifest would clash. PS.: If we want to support manual unzipping as part of this workflow, we should also stop using that special compression flag when creating extensions using Blender cli. Which I think we should anyways if you ask me.

I made some PRs with relatively straightforward changes.

I didn't do the bundle directory yet. Thinking about this, if you are modifying the Blender install folder you can just dump stuff in the 4.2/scripts folder anyway. For studio setups environment variables are the typical way to set up this kind of thing. It's trivial to add if we want to though.

Perhaps beyond this the main thing is to write some things in the user manual:

  • How to create a bundle of extensions to send to someone to install, for example for an offline or online course.
  • How to extract extensions and set up environment variables to use them, for a more automated studio environment.
I made some PRs with relatively straightforward changes. I didn't do the `bundle` directory yet. Thinking about this, if you are modifying the Blender install folder you can just dump stuff in the `4.2/scripts` folder anyway. For studio setups environment variables are the typical way to set up this kind of thing. It's trivial to add if we want to though. Perhaps beyond this the main thing is to write some things in the user manual: * How to create a bundle of extensions to send to someone to install, for example for an offline or online course. * How to extract extensions and set up environment variables to use them, for a more automated studio environment.

PS.: If we want to support manual unzipping as part of this workflow, we should also stop using that special compression flag when creating extensions using Blender cli. Which I think we should anyways if you ask me.

I don't know what this refers to, I could unzip the files fine on Windows, macOS and Linux with the default tools.

> PS.: If we want to support manual unzipping as part of this workflow, we should also stop using that special compression flag when creating extensions using Blender cli. Which I think we should anyways if you ask me. I don't know what this refers to, I could unzip the files fine on Windows, macOS and Linux with the default tools.
Brecht Van Lommel changed title from Extensions: Portable bundle to Extensions: Deploying add-ons for offline work and automation 2024-06-04 15:24:44 +02:00
Author
Owner

PS.: If we want to support manual unzipping as part of this workflow, we should also stop using that special compression flag when creating extensions using Blender cli. Which I think we should anyways if you ask me.

I don't know what this refers to, I could unzip the files fine on Windows, macOS and Linux with the default tools.

For records, this will no longer be a problem. But more information can be found here: #122710

>> PS.: If we want to support manual unzipping as part of this workflow, we should also stop using that special compression flag when creating extensions using Blender cli. Which I think we should anyways if you ask me. > I don't know what this refers to, I could unzip the files fine on Windows, macOS and Linux with the default tools. For records, this will no longer be a problem. But more information can be found here: https://projects.blender.org/blender/blender/issues/122710

The solutions that rely on "Shared Script Directories" have implications for long term maintenance of extensions - requiring supporting extensions both as top level modules and sub-modules indefinitely, this means we can't assume extensions have their own name-space and there may be bugs when developers don't account for both ways of loading their extensions.

This feels very much like short-term-easy-solution that incurs long term overhead and brings back problems extensions intended to solve.

Local repositories should be usable to access bundled extensions, some details need to be figured out but I think it can be made to work.

The solutions that rely on "Shared Script Directories" have implications for long term maintenance of extensions - requiring supporting extensions both as top level modules and sub-modules indefinitely, this means we can't assume extensions have their own name-space and there may be bugs when developers don't account for both ways of loading their extensions. This feels very much like short-term-easy-solution that incurs long term overhead and brings back problems extensions intended to solve. Local repositories should be usable to access bundled extensions, some details need to be figured out but I think it can be made to work.
Author
Owner

Can't the bundle script folder be its own namespace? Having scripts on the global namespace does sound like a step backwards.

I think it is fair to build a solution that is aimed at (local) extensions (which in turn relies on relative import OR hacks ;). @brecht does that goes against the audience you were targeting?

Can't the bundle script folder be its own namespace? Having scripts on the global namespace does sound like a step backwards. I think it is fair to build a solution that is aimed at (local) extensions (which in turn relies on relative import OR hacks ;). @brecht does that goes against the audience you were targeting?

Yes it can be it's own name-space, and if it could be supported with it's own name-space - this addresses my concerns for the mostpart.

But in that case we would need to have some mechanism for inserting it into the extensions-module with a unique sub-module name, make sure extensions properly update when adding/removing the script directories. At this point it's practically a second kind of local-repository that needs to co-exist with other repositories ... which begs the question "why not just use local-repositories?".

Yes it can be it's own name-space, and if it could be supported with it's own name-space - this addresses my concerns for the mostpart. But in that case we would need to have some mechanism for inserting it into the extensions-module with a unique sub-module name, make sure extensions properly update when adding/removing the script directories. At this point it's practically a second kind of local-repository that needs to co-exist with other repositories ... which begs the question "why not just use local-repositories?".

Talked with Brecht about the design for extensions repositories being used for bundle. Here are the changes we agreed on.

Notes:

  • Add BLENDER_SYSTEM_EXTENSIONS matching BLENDER_USER_EXTENSIONS.

  • Repositories support "System" directory,
    where the directory string is appended to BLENDER_SYSTEM_EXTENSIONS.

  • The user defaults will include a system repository.
    This includes an empty directory which would be included in portable blender distributions (so users are able to know where to set up their bundled extensions).

  • The default for:

    • BLENDER_SYSTEM_EXTENSIONS would be ./4.2/extensions.
    • Default repository would be system.

    (this relies on changes to the portable install).
    Otherwise these names would cause BLENDER_SYSTEM_EXTENSIONS to be inside BLENDER_USER_EXTENSIONS which could be difficult to support.

  • System repositories will need a way to be accessed and remain read-only:
    We will use a path as if the repository were local
    since this is wont collide with built-in locations, e.g.
    ~/.config/blender/4.2/extensions/{repo_module_id}/
    ... which could contain any repository cache even if the source files are located on a system repository.
    We could always use this, even for custom-directory repositories since these may also be read-only.

Open topics:

These aren't required to be solved for the above items to be handled. Noting them as areas we will want to solve.

  • Where extensions can write their own configuration
    because system directory may be read-only
    we could consider bpy.utils.extension_resource(package=__package__) which generates an extension-specific.
    This doesn't have to be solved now, but system extensions raises this as something we may need to solve soon.

  • How to populate this repository.

Talked with Brecht about the design for extensions repositories being used for bundle. Here are the changes we agreed on. **Notes:** - Add `BLENDER_SYSTEM_EXTENSIONS` matching `BLENDER_USER_EXTENSIONS`. - Repositories support "System" directory, where the directory string is appended to `BLENDER_SYSTEM_EXTENSIONS`. - The user defaults will include a system repository. This includes an empty directory which would be included in portable blender distributions (so users are able to know where to set up their bundled extensions). - The default for: - `BLENDER_SYSTEM_EXTENSIONS` would be `./4.2/extensions`. - Default repository would be `system`. (this relies on changes to the portable install). Otherwise these names would cause `BLENDER_SYSTEM_EXTENSIONS` to be inside `BLENDER_USER_EXTENSIONS` which could be difficult to support. - System repositories will need a way to be accessed and remain read-only: We will use a path as if the repository were local since this is wont collide with built-in locations, e.g. `~/.config/blender/4.2/extensions/{repo_module_id}/` ... which could contain any repository cache even if the source files are located on a system repository. _We could always use this, even for custom-directory repositories since these may also be read-only._ **Open topics:** _These aren't required to be solved for the above items to be handled. Noting them as areas we will want to solve._ - Where extensions can write their own configuration because system directory may be read-only we could consider `bpy.utils.extension_resource(package=__package__)` which generates an extension-specific. This doesn't have to be solved now, but system extensions raises this as something we may need to solve soon. - How to populate this repository.

I started writing some docs about this subject in blender/blender-manual!104811, but will wait to land until this system repository is implemented.

I started writing some docs about this subject in blender/blender-manual!104811, but will wait to land until this system repository is implemented.

Support for system-extension repositories !122832, although the overlap between local & system extensions should be resolved first (otherwise user/system extensions may overlap).

Support for system-extension repositories !122832, although the overlap between local & system extensions should be resolved first (otherwise user/system extensions may overlap).

The portable config changes to avoid the overlap landed in #122778.

The portable config changes to avoid the overlap landed in #122778.

Support for "system" repositories has landed !122832.


Added remaining TODO's to #119521:

  • Support read-only system repositories.
  • Document the process of bundling extensions.
Support for "system" repositories has landed !122832. --- Added remaining TODO's to #119521: - Support read-only system repositories. - Document the process of bundling extensions.

@brecht & @dfelinto: there is a detail regarding supporting read-only "System" repositories.

Besides making system repositories support a read-only file-system, as the tasks mentions users shouldn't be able to edit them.

This would mean removing the following functionality:

  • Installing from disk (where system repositories wouldn't be listed).
  • Uninstall extensions.
  • Delete files when removing the repository.

When the options aren't available, the buttons can be greyed out with a message in the tooltip noting that system repositories are read-only.

@brecht & @dfelinto: there is a detail regarding supporting read-only "System" repositories. Besides making system repositories support a read-only file-system, as the tasks mentions users shouldn't be able to edit them. This would mean removing the following functionality: - Installing from disk (where system repositories wouldn't be listed). - Uninstall extensions. - Delete files when removing the repository. When the options aren't available, the buttons can be greyed out with a message in the tooltip noting that system repositories are read-only.

Yes, disabling that functionality sounds rights.

Yes, disabling that functionality sounds rights.

Committed the suggested changes as well as support for read-only repositories, a24add4a66, closing.


The need for docs is noted in: #119521.

Committed the suggested changes as well as support for read-only repositories, a24add4a669a9281fcde4e7bfc3fbb2ac1093e18, closing. ---- The need for docs is noted in: #119521.
Blender Bot added the
Status
Archived
label 2024-06-11 08:07:53 +02:00
Campbell Barton added
Status
Resolved
and removed
Status
Archived
labels 2024-06-11 08:08:11 +02:00
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Asset Browser Project
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#122512
No description provided.