Proposal for extensible CLI sub-commands #117936

Closed
opened 2024-02-07 08:26:18 +01:00 by Campbell Barton · 19 comments

Motivation

For the Blender extensions project we would like to make command-line tools available for validating extensions, building extension archives and running automated updates.

To prevent adding too many top level command-line arguments this proposal suggests an extensible way to add sub-commands that can be used by built-in tools as well as add-ons.

See extensions task: #117286

Proposed Solution

Provide an interface for scripts to register CLI tools and a convenient way to access them.

Note that the exact details for how this is implemented is an open topic,
a suggested implementation is included here.

Usage:

  • Add a new --tool command-line argument.
  • All trailing arguments would be passed to the handler function, so: blender --tool extensions update
    would call the "extensions" handler passing in ["update"].
  • By convention all tools are expected to have their own --help message, e.g. blender --tool extensions --help.
  • Blender's --help message would list all registered --tool arguments.
  • Scripts (including add-ons) may register their own tools.

Implementation:

  • Two Python API functions would be added:
    bpy.utils.register_cli_tool("name", handler_fn) along with an unregister_cli_tool.
    These would wrap C++ functions, allowing native C++ sub-commands too.
  • Using --tool implies --background.
  • Using --tool loads the user preferences but not the users startup.blend.
    Avoid tools running slowly if the user happens to have a heavy startup.
  • Blender's built-in argument parsing stops after --tool {ARGUMENT} (no need to use --).
  • The handler takes a list of strings and returns an integer (used for the exit-code).

Pros:

  • Convenient to access from the Blender command-line.
  • Extensible (add-ons can easily add their own).
  • Discoverable (shown in --help).

Cons:

  • Calling blender --tool extensions is less convenient than a commend - blender-extensions for instance.
    For developers who might call these frequently we could recommend they setup aliases to tools they use frequently
    with the convention of blender --tool example -> blender-example.
  • We can't take advantage of knowledge about what the tool requires in order avoid redundant work on startup.
    As add-ons need to be loaded, the user preferences also need to be loaded.
    In general it's impractical to declare that a tool doesn't require a Blender session temporary directory or any other details relating to Blender's environment which is typically initialized before loading add-ons.

    While this is a solvable problem, I don't see how we could solve this without involved auto-discovery of tools and their requirements which runs before low-level systems are initialized but after all enabled add-ons are known. This adds considerable overhead/complexity for something I believe we can manage without.
  • Added overhead for running --help which needs to load all add-ons to display the tools.
    If loading Blender's internals to display help text is a problem: --tool help could be used to discover tools,
    matching --engine functionality.

Alternative Solutions

For completeness, I've listed some alternative solutions with their pros/cons.

The main advantage with them is we don't need to create & maintain an API for CLI tools in Blender
although I find the results are often less convenient.

Wrapper Scripts

It is already possible to create a wrapper script that runs a command-line tool that's bundled with an add-on.

If we wanted to use this solution, scripts for Linux/macOS/Windows would need to be included with Blender,
so Blender would come with additional tools which would be available along-side Blender.

This example shows how 'scripts/addons/my_addon/cli/my_custom_tool.py can be run from Blender on the command-line.

#!/bin/sh
BF_DIST_BIN=$(dirname "$0")
BF_PROGRAM="blender"

exec "$BF_DIST_BIN/$BF_PROGRAM" \
    --background \
    -noaudio \
    --python-expr \
    "import os, sys, subprocess, bpy; sys.exit(subprocess.call([sys.executable, os.path.join(bpy.utils.system_resource('SCRIPTS'), 'addons, 'my_addon, 'cli', 'my_custom_tool.py'), *sys.argv[sys.argv.index('--') + 1:]]))" -- "$@"

Pros:

  • Doesn't require a specialized API (simply rely on wrapper script).
  • Commands are discoverable and convenient to access.

Cons:

  • 3rd party add-ons would not be able to register their own tools.
  • With added visibility & inclusion in the PATH (for Linux at least) there is an incentive to avoid including obscure functionality in Blender's base directory. Using a sub-command has the advantage we can include more obscure actions without making a strong argument for adding additional top-level commands.
    Arguably a blender-extensions command could justify it's own top-level wrapper script, however we might take advantage of sub-commands in places where top-level wrapper scripts aren't as appropriate.
  • Potential portability issues as we would have to support wrapper scripts on Linux/macOS/Windows, snap packages.

Support Modules as Scripts

Python supports -m module-name which terminates argument parsing and is used to implement sub-commands.

This example starts a local file server using http.server:

python3 -m http.server 3333

We could add an equivalent --python-module argument which allows for any module to be accessed from the command-line.

Pros:

  • Doesn't require a specialized API (simply loads any module).
  • It's making use of an existing convention - even allowing built-in Python modules to be used via Blender.

Cons:

  • The full module path must be known, with newly added extensions,
    these are name-spaced so the module name depends on the extension repositories preferences.
  • Not discoverable as any Python module might provide a command-line interface (solvable via documentation).
  • Less convenient as it requires --background to be passed in for typical usage.

NOTE: we could support this whatever the outcome of this proposal, although it doesn't seem like a priority either.

Stand Alone Tools

We could release a command-line tool to deal with Blender extensions, this has a few down-sides though.


Pros:

  • Doesn't require any integration with Blender.
  • Once installed & in the users PATH, accessing is more convenient.
  • May run faster (avoid the overhead of running Blender's binary to access a command-line tool).

Cons:

  • A separate tool would need a way to reference the blender binary for Blender specific actions such as upgrading packages (take a --blender argument for instance).
  • A separate tool could become outdated or incompatible with the version of Blender in use.
  • The additional overhead for us to manage releases of a 3rd party tools as well as users to install and update a separate packaging tool.
  • Adding additional tools requires a separate release process, users to install etc.
  • Add-ons could not easily add their own tools.
Motivation ========== For the Blender extensions project we would like to make command-line tools available for validating extensions, building extension archives and running automated updates. To prevent adding too many top level command-line arguments this proposal suggests an extensible way to add sub-commands that can be used by built-in tools as well as add-ons. See extensions task: #117286 Proposed Solution ================= Provide an interface for scripts to register CLI tools and a convenient way to access them. Note that the exact details for how this is implemented is an open topic, a suggested implementation is included here. **Usage:** - Add a new `--tool` command-line argument. - All trailing arguments would be passed to the handler function, so: `blender --tool extensions update` would call the "extensions" handler passing in `["update"]`. - By convention all tools are expected to have their own `--help` message, e.g. `blender --tool extensions --help`. - Blender's `--help` message would list all registered `--tool` arguments.<br> - Scripts (including add-ons) may register their own tools. **Implementation:** - Two Python API functions would be added: `bpy.utils.register_cli_tool("name", handler_fn)` along with an `unregister_cli_tool`.<br>*These would wrap C++ functions, allowing native C++ sub-commands too.* - Using `--tool` implies `--background`. - Using `--tool` loads the user preferences but **not** the users `startup.blend`. *Avoid tools running slowly if the user happens to have a heavy startup.* - Blender's built-in argument parsing stops after `--tool {ARGUMENT}` (no need to use `--`). - The handler takes a list of strings and returns an integer (used for the exit-code). --- **Pros:** - Convenient to access from the Blender command-line. - Extensible (add-ons can easily add their own). - Discoverable (shown in `--help`). **Cons:** - Calling `blender --tool extensions` is less convenient than a commend - `blender-extensions` for instance. For developers who might call these frequently we could recommend they setup aliases to tools they use frequently with the convention of `blender --tool example` -> `blender-example`. - We can't take advantage of knowledge about what the tool requires in order avoid redundant work on startup. As add-ons need to be loaded, the user preferences also need to be loaded. In general it's impractical to declare that a tool doesn't require a Blender session temporary directory or any other details relating to Blender's environment which is typically initialized before loading add-ons.<br> *While this is a solvable problem, I don't see how we could solve this without involved auto-discovery of tools and their requirements which runs before low-level systems are initialized but after all enabled add-ons are known. This adds considerable overhead/complexity for something I believe we can manage without.* - Added overhead for running `--help` which needs to load all add-ons to display the tools. If loading Blender's internals to display help text is a problem: `--tool help` could be used to discover tools, matching `--engine` functionality. Alternative Solutions ===================== For completeness, I've listed some alternative solutions with their pros/cons. The main advantage with them is we don't need to create & maintain an API for CLI tools in Blender although I find the results are often less convenient. ### Wrapper Scripts It is already possible to create a wrapper script that runs a command-line tool that's bundled with an add-on. If we wanted to use this solution, scripts for Linux/macOS/Windows would need to be included with Blender, so Blender would come with additional tools which would be available along-side Blender. This example shows how `'scripts/addons/my_addon/cli/my_custom_tool.py` can be run from Blender on the command-line. ```.sh #!/bin/sh BF_DIST_BIN=$(dirname "$0") BF_PROGRAM="blender" exec "$BF_DIST_BIN/$BF_PROGRAM" \ --background \ -noaudio \ --python-expr \ "import os, sys, subprocess, bpy; sys.exit(subprocess.call([sys.executable, os.path.join(bpy.utils.system_resource('SCRIPTS'), 'addons, 'my_addon, 'cli', 'my_custom_tool.py'), *sys.argv[sys.argv.index('--') + 1:]]))" -- "$@" ``` **Pros:** - Doesn't require a specialized API (simply rely on wrapper script). - Commands are discoverable and convenient to access. **Cons:** - 3rd party add-ons would not be able to register their own tools. - With added visibility & inclusion in the PATH (for Linux at least) there is an incentive to avoid including obscure functionality in Blender's base directory. Using a sub-command has the advantage we can include more obscure actions without making a strong argument for adding additional top-level commands. *Arguably a `blender-extensions` command **could** justify it's own top-level wrapper script, however we might take advantage of sub-commands in places where top-level wrapper scripts aren't as appropriate.* - Potential portability issues as we would have to support wrapper scripts on Linux/macOS/Windows, snap packages. ### Support Modules as Scripts Python supports `-m module-name` which terminates argument parsing and is used to implement sub-commands. This example starts a local file server using `http.server`: ``` python3 -m http.server 3333 ``` We could add an equivalent `--python-module` argument which allows for any module to be accessed from the command-line. **Pros:** - Doesn't require a specialized API (simply loads any module). - It's making use of an existing convention - even allowing built-in Python modules to be used via Blender. **Cons:** - The full module path must be known, with newly added extensions, these are name-spaced so the module name depends on the extension repositories preferences. - Not discoverable as any Python module *might* provide a command-line interface (solvable via documentation). - Less convenient as it requires `--background` to be passed in for typical usage. *NOTE: we could support this whatever the outcome of this proposal, although it doesn't seem like a priority either.* ### Stand Alone Tools We could release a command-line tool to deal with Blender extensions, this has a few down-sides though. --- **Pros:** - Doesn't require any integration with Blender. - Once installed & in the users PATH, accessing is more convenient. - May run faster (avoid the overhead of running Blender's binary to access a command-line tool). **Cons:** - A separate tool would need a way to reference the `blender` binary for Blender specific actions such as upgrading packages (take a `--blender` argument for instance). - A separate tool could become outdated or incompatible with the version of Blender in use. - The additional overhead for us to manage releases of a 3rd party tools as well as users to install and update a separate packaging tool. - Adding additional tools requires a separate release process, users to install etc. - Add-ons could not easily add their own tools.
Campbell Barton added the
Interest
Python API
Type
Design
Module
Core
labels 2024-02-07 08:26:19 +01:00

To me this seems like we are trying to cram too much into one binary. Especially since the functionality is kinda orthogonal to Blender (3D creation software != package manager). IE the extensions platform is essentially only managing a folder structure on the end users computer. This folder structure is then used by Blender to load the extensions. But there shouldn't be anything that strongly ties this together with the actual blender program/binary.

I personally think that this should be a stand alone tool. More akin to how the Vim and Emacs plugin system(s) are setup.

This means that fetching, updating and installing plugins is handled by third party plugins/tools that can be invoked from outside and inside of the program. (Of course additional functionality is there if used from inside of Blender)
For Blender, we would of course ship the plugin and the tools with our releases.

Pros:

  • You will never need the blender binary to package, fetch, install, or update addons.
    (However if you execute any of these steps from within Blender the tools will notify Blender when to update/refresh the updated parts)
  • More flexibility. If someone wants to change how the fetching or updating works or add new functionality, they don't need to recompile blender. It is also much easier to test this from the command line.
  • By making this a standalone thing it is easier for us to focus on providing stable APIs that plugin manager systems can use.

Cons:

  • Less tight integration with Blender (however I would argue that this is actually a pro)
  • Could be cumbersome if our APIs on the Blender side are very unstable and changes a lot. (This should not be the case though as the APIs should be as simple and minimal as possible)
To me this seems like we are trying to cram too much into one binary. Especially since the functionality is kinda orthogonal to Blender (3D creation software != package manager). IE the extensions platform is essentially only managing a folder structure on the end users computer. This folder structure is then used by Blender to load the extensions. But there shouldn't be anything that strongly ties this together with the actual blender program/binary. I personally think that this should be a stand alone tool. More akin to how the Vim and Emacs plugin system(s) are setup. This means that fetching, updating and installing plugins is handled by third party plugins/tools that can be invoked from outside and inside of the program. (Of course additional functionality is there if used from inside of Blender) For Blender, we would of course ship the plugin and the tools with our releases. Pros: - You will never need the blender binary to package, fetch, install, or update addons. (However if you execute any of these steps from within Blender the tools will notify Blender when to update/refresh the updated parts) - More flexibility. If someone wants to change how the fetching or updating works or add new functionality, they don't need to recompile blender. It is also much easier to test this from the command line. - By making this a standalone thing it is easier for us to focus on providing stable APIs that plugin manager systems can use. Cons: - Less tight integration with Blender (however I would argue that this is actually a pro) - Could be cumbersome if our APIs on the Blender side are very unstable and changes a lot. (This should not be the case though as the APIs should be as simple and minimal as possible)
Member

My initial feeling is that trying to separate it from the Blender binary might become more annoying later on. Might be true that in the beginning these tools don't actually need the Blender binary, but it feels likely that we want to reuse some parts of the binary at some point.

Instead of blender --tool <tool-name>, I'd probably use something like blender run <tool-name>. It feels friendlier to me, but that's very subjective. It also makes it more obvious that you can't pass other arguments to Blender that you'd normally pass if you wanted to start a Blender session.

My initial feeling is that trying to separate it from the Blender binary might become more annoying later on. Might be true that in the beginning these tools don't actually need the Blender binary, but it feels likely that we want to reuse some parts of the binary at some point. Instead of `blender --tool <tool-name>`, I'd probably use something like `blender run <tool-name>`. It feels friendlier to me, but that's very subjective. It also makes it more obvious that you can't pass other arguments to Blender that you'd normally pass if you wanted to start a Blender session.
Author
Owner

@ZedDB what you've described regarding package management being separate from Blender is in fact already the case. The low-level package management commands don't depend on Blender at all.

Blender defines:

  • The repositories directories & their URL's which are stored in the user-preferences.
  • The location of blender's bundled scripts.
  • The python binary (which may be bundled with Blender), important if the operating system doesn't provide a recent Python version.

Reliably accessing this information for both portable & system installs on linux/macOS/windows isn't trivial. Nevertheless if this information was available somewhere, an external tool could perform an update.

This changes each blender release and may be confusing if you have multiple Blender's installed on a system, so it's simpler if Blender can provide this information... at that point you have to run blender, so triggering an update from blender means the sub-command can deal with the details of selecting the proper Python-binary, script-files and repositories from the preferences.

@ZedDB what you've described regarding package management being separate from Blender is in fact already the case. The low-level package management commands don't depend on Blender at all. Blender defines: - The repositories directories & their URL's which are stored in the user-preferences. - The location of blender's bundled scripts. - The python binary (which may be bundled with Blender), important if the operating system doesn't provide a recent Python version. Reliably accessing this information for both portable & system installs on linux/macOS/windows isn't trivial. Nevertheless if this information was available somewhere, an external tool could perform an update. This changes each blender release and may be confusing if you have multiple Blender's installed on a system, so it's simpler if Blender can provide this information... at that point you have to run blender, so triggering an update from blender means the sub-command can deal with the details of selecting the proper Python-binary, script-files and repositories from the preferences.
Author
Owner

Instead of blender --tool <tool-name>, I'd probably use something like blender run <tool-name>. It feels friendlier to me, but that's very subjective. It also makes it more obvious that you can't pass other arguments to Blender that you'd normally pass if you wanted to start a Blender session.

Regarding blender run, on it's own it seems OK however this can be used in combination with other options, where having run in the middle of multiple arguments seems strange and breaks the convention of arguments beginning with a -.

blender -d run {action} makes it seem run {action} are arguments for -d.

Whereas blender -d --run {action} makes it clearer that action is an argument to --run.

We could allow this exception but I think it may lead to ambiguous argument handling issues in the future, incidentally blender run will load the file called run if it exists.

Note that I'm not all that fussed about the exact name right now.

> Instead of `blender --tool <tool-name>`, I'd probably use something like `blender run <tool-name>`. It feels friendlier to me, but that's very subjective. It also makes it more obvious that you can't pass other arguments to Blender that you'd normally pass if you wanted to start a Blender session. Regarding `blender run`, on it's own it seems OK however this can be used in combination with other options, where having run in the middle of multiple arguments seems strange and breaks the convention of arguments beginning with a `-`. `blender -d run {action}` makes it seem `run {action}` are arguments for `-d`. Whereas `blender -d --run {action}` makes it clearer that `action` is an argument to `--run`. We could allow this exception but I think it may lead to ambiguous argument handling issues in the future, incidentally `blender run` will load the file called `run` if it exists. *Note that I'm not all that fussed about the exact name right now.*

@ideasman42 thanks for the explanation!

  • The repositories directories & their URL's which are stored in the user-preferences.
  • The location of blender's bundled scripts.
  • The python binary (which may be bundled with Blender), important if the operating system doesn't provide a recent Python version.

Wouldn't these be defined at compile time? So it shouldn't be too hard to derive where to look for these in a stand alone script that we ship with the release.
So we would have something similar to blender-thumbnailer. But it would of course be called something like blender-extensions as you suggested before.

@ideasman42 thanks for the explanation! > - The repositories directories & their URL's which are stored in the user-preferences. > - The location of blender's bundled scripts. > - The python binary (which may be bundled with Blender), important if the operating system doesn't provide a recent Python version. Wouldn't these be defined at compile time? So it shouldn't be too hard to derive where to look for these in a stand alone script that we ship with the release. So we would have something similar to `blender-thumbnailer`. But it would of course be called something like `blender-extensions` as you suggested before.
Author
Owner

@ZedDB these are known at compile time, yes, for portable builds they're relative to blender's base directory. So if we install a script with Blender, that can work.

User resource locations can be configured with BLENDER_USER_RESOURCES and some other environment variables, on windows - user-resources can be relative to the portable build too IIRC. My concern is that we would have to match this logic exactly with external scripts, any corner case where it differs is a bug but it can work and wouldn't be causing us problems for common cases.

There is still the case of having to either to extract the repository information from the preferences (or store it separately for more convenient access). If that's handled an external script that doesn't run Blender could be constructed & run as part of building Blender.

@ZedDB these are known at compile time, yes, for portable builds they're relative to blender's base directory. So if we install a script with Blender, that can work. User resource locations can be configured with `BLENDER_USER_RESOURCES` and some other environment variables, on windows - user-resources can be relative to the portable build too IIRC. My concern is that we would have to match this logic exactly with external scripts, any corner case where it differs is a bug but it can work and wouldn't be causing us problems for common cases. There is still the case of having to either to extract the repository information from the preferences (or store it separately for more convenient access). If that's handled an external script that doesn't run Blender could be constructed & run as part of building Blender.

There is still the case of having to either to extract the repository information from the preferences (or store it separately for more convenient access). If that's handled an external script that doesn't run Blender could be constructed & run as part of building Blender.

I would say that this information is package manager specific and should be stored separately. Especially if the package manager is already stand alone as you say.

This would also make it easier to have good test coverage of the functionality of the package manager without having to involve Blender in there as well.

> There is still the case of having to either to extract the repository information from the preferences (or store it separately for more convenient access). If that's handled an external script that doesn't run Blender could be constructed & run as part of building Blender. I would say that this information is package manager specific and should be stored separately. Especially if the package manager is already stand alone as you say. This would also make it easier to have good test coverage of the functionality of the package manager without having to involve Blender in there as well.

I don't have a strong opinion here, but proposed solution sounds fine to me.

Note that am not sure about debates regarding the 'ease of use' of each solutions - I have the feeling that for the vast majority of users nowadays, having to use the command line will never be considered as 'easy to use'.

I don't have a strong opinion here, but proposed solution sounds fine to me. Note that am not sure about debates regarding the 'ease of use' of each solutions - I have the feeling that for the vast majority of users nowadays, having to use the command line will never be considered as 'easy to use'.
Author
Owner

There is still the case of having to either to extract the repository information from the preferences (or store it separately for more convenient access). If that's handled an external script that doesn't run Blender could be constructed & run as part of building Blender.

I would say that this information is package manager specific and should be stored separately. Especially if the package manager is already stand alone as you say.

We could do this and I'm open to it, the main consideration is how we would store preference data outside of Blender & synchronize them properly.
If your proposing to remove extension-repositories from preferences and have Blender's preferences UI & manage external external data, that's a bigger change and I'd rather handle this as an stand-alone proposal.

This would also make it easier to have good test coverage of the functionality of the package manager without having to involve Blender in there as well.

It's already the case that most tests don't involve Blender, although this only the lower level actions are being tested (create repo, create an extension, install an extension over http/filesystem, updating etc.).

> > There is still the case of having to either to extract the repository information from the preferences (or store it separately for more convenient access). If that's handled an external script that doesn't run Blender could be constructed & run as part of building Blender. > > I would say that this information is package manager specific and should be stored separately. Especially if the package manager is already stand alone as you say. We could do this and I'm open to it, the main consideration is how we would store preference data outside of Blender & synchronize them properly. If your proposing to remove extension-repositories from preferences and have Blender's preferences UI & manage external external data, that's a bigger change and I'd rather handle this as an stand-alone proposal. > This would also make it easier to have good test coverage of the functionality of the package manager without having to involve Blender in there as well. It's already the case that most tests don't involve Blender, although this only the lower level actions are being tested (create repo, create an extension, install an extension over http/filesystem, updating etc.).
Author
Owner

Note that am not sure about debates regarding the 'ease of use' of each solutions - I have the feeling that for the vast majority of users nowadays, having to use the command line will never be considered as 'easy to use'.

Right, for something easy to use:

  • A technical user can run an update from a command-line (most likely other basic operations too).
  • Extension authors can create/validate/build (in the future upload) their extensions.

That's not a high bar, it's just at the moment we don't have a convenient way to do it.

> Note that am not sure about debates regarding the 'ease of use' of each solutions - I have the feeling that for the vast majority of users nowadays, having to use the command line will never be considered as 'easy to use'. Right, for something easy to use: - A technical user can run an update from a command-line (most likely other basic operations too). - Extension authors can create/validate/build (in the future upload) their extensions. That's not a high bar, it's just at the moment we don't have a convenient way to do it.

The description reads:

Scripts (including add-ons) may register their own tools.
[...]
See extensions task: #117286

That task description doesn't mention "tool" or "cli". I feel that this discussion gets a bit muddled (and still I'm joining below, yay) as it's not quite clear to me what concrete "tools" would be added by extensions.

The use case of having an extension package manager CLI interface is very concrete. I'm not convinced that such a meta-operation ("managing Blender", rather than "running Blender") should be handled the same way as "extensible CLI tools that are likely to depend on Blender itself".

Without having a clear definition of what these extension CLI tools can or cannot do, it's hard to come up with the best way to design them.

@ZedDB wrote:
To me this seems like we are trying to cram too much into one binary.

I agree with this sentiment.

@JacquesLucke wrote:
My initial feeling is that trying to separate it from the Blender binary might become more annoying later on. Might be true that in the beginning these tools don't actually need the Blender binary, but it feels likely that we want to reuse some parts of the binary at some point.

This I also agree with.

I think this PR, with the nice list of alternatives (thanks @ideasman42 for being so extensive!) illuminates the current limitations of Blender's architecture. Ideally it would be possible to simply create another binary that links against just a few of Blender's internal libraries. That way we can have a tool that's specific to managing Blender extensions, and that still shares certain logic with Blender. And yeah, I put the word "ideally" in there intentionally, as I know that this would require considerably more work than we have time for right now.

But still, maybe there are a few things that we can link directly, or with some slight refactoring. I don't think "it has to be able to find the bundled Python" is good enough reason to stop thinking about this direction. Usually moving towards separately-usable code also improves testability, documentability, and general code quality due to looser coupling.

Maybe a separate binary that always sits next to the main Blender executable is the way to go? That way that binary can run Blender when necessary, as it knows exactly which binary to use. It can provide a clean CLI that avoids mixing Blender arguments with extension-management arguments. For example, I wouldn't want to have to pass --noaudio and --background for all extension operations. This could work well for meta-operations, like package management.

Going back to what I wrote at the top: the PR is written in such a way that a bunch of the design ideas for these tools are actually mentioned in the implementation details. This makes talking about alternative implementations hard, because that would also have to change the design ideas (instead of the implementation following the design).

The description reads: > Scripts (including add-ons) may register their own tools. > [...] > See extensions task: #117286 That task description doesn't mention "tool" or "cli". I feel that this discussion gets a bit muddled (and still I'm joining below, yay) as it's not quite clear to me what concrete "tools" would be added by extensions. The use case of having an extension package manager CLI interface is very concrete. I'm not convinced that such a meta-operation ("managing Blender", rather than "running Blender") should be handled the same way as "extensible CLI tools that are likely to depend on Blender itself". Without having a clear definition of what these extension CLI tools can or cannot do, it's hard to come up with the best way to design them. > @ZedDB wrote: > To me this seems like we are trying to cram too much into one binary. I agree with this sentiment. > @JacquesLucke wrote: > My initial feeling is that trying to separate it from the Blender binary might become more annoying later on. Might be true that in the beginning these tools don't actually need the Blender binary, but it feels likely that we want to reuse some parts of the binary at some point. This I also agree with. I think this PR, with the nice list of alternatives (thanks @ideasman42 for being so extensive!) illuminates the current limitations of Blender's architecture. Ideally it would be possible to simply create another binary that links against just a few of Blender's internal libraries. That way we can have a tool that's specific to managing Blender extensions, and that still shares certain logic with Blender. And yeah, I put the word "ideally" in there intentionally, as I know that this would require considerably more work than we have time for right now. But still, maybe there are a few things that we can link directly, or with some slight refactoring. I don't think "it has to be able to find the bundled Python" is good enough reason to stop thinking about this direction. Usually moving towards separately-usable code also improves testability, documentability, and general code quality due to looser coupling. Maybe a separate binary that always sits next to the main Blender executable is the way to go? That way that binary can run Blender when necessary, as it knows exactly which binary to use. It can provide a clean CLI that avoids mixing Blender arguments with extension-management arguments. For example, I wouldn't want to have to pass `--noaudio` and `--background` for all extension operations. This could work well for meta-operations, like package management. Going back to what I wrote at the top: the PR is written in such a way that a bunch of the design ideas for these tools are actually mentioned in the implementation details. This makes talking about alternative implementations hard, because that would also have to change the design ideas (instead of the implementation following the design).
Author
Owner

In general using a separate binary is interesting but a bigger project, agree the kinds of refactoring encourage better design & decoupling libraries.

@dr.sybren wrote:
That task description doesn't mention "tool" or "cli".

The task mentioned adding --extensions command line argument, I wasn't so keen on this as it would add many command line arguments, thisj proposal is suggesting an alternative - although I should have made that clear.

@dr.sybren wrote:
I don't think "it has to be able to find the bundled Python" is good enough reason to stop thinking about this direction.

Agree, the current implementation of extensions depends on reading user preferences & paths from GHOST, we could refactor these areas so it's possible to access paths & preferences without pulling in other dependencies.

@dr.sybren wrote:
For example, I wouldn't want to have to pass --noaudio and --background for all extension operations.

The proposal states that --tool implies --background although it seems reasonable to disable audio as well.

In general using a separate binary is interesting but a bigger project, agree the kinds of refactoring encourage better design & decoupling libraries. > @dr.sybren wrote: > That task description doesn't mention "tool" or "cli". The task mentioned adding `--extensions` command line argument, I wasn't so keen on this as it would add many command line arguments, thisj proposal is suggesting an alternative - although I should have made that clear. > @dr.sybren wrote: > I don't think "it has to be able to find the bundled Python" is good enough reason to stop thinking about this direction. Agree, the current implementation of extensions depends on reading user preferences & paths from GHOST, we could refactor these areas so it's possible to access paths & preferences without pulling in other dependencies. > @dr.sybren wrote: > For example, I wouldn't want to have to pass `--noaudio` and `--background` for all extension operations. The proposal states that `--tool` implies `--background` although it seems reasonable to disable audio as well.

We could do this and I'm open to it, the main consideration is how we would store preference data outside of Blender & synchronize them properly.
If your proposing to remove extension-repositories from preferences and have Blender's preferences UI & manage external external data, that's a bigger change and I'd rather handle this as an stand-alone proposal.

What I had in mind was that the repositories and extension information would be stored outside of Blender, yes. This would also make it much easier for sys admins to maintain and setup computers with specific extension repos.

Blender would then "source" this data on startup and cache it internally. This could of course be re-sourced or saved back to disk while Blender is running. This is so we don't run into situations where everything is running fine from the standalone CLI, but it is broken when you try to do the same from within Blender. (or vise versa)

>We could do this and I'm open to it, the main consideration is how we would store preference data outside of Blender & synchronize them properly. If your proposing to remove extension-repositories from preferences and have Blender's preferences UI & manage external external data, that's a bigger change and I'd rather handle this as an stand-alone proposal. What I had in mind was that the repositories and extension information would be stored outside of Blender, yes. This would also make it much easier for sys admins to maintain and setup computers with specific extension repos. Blender would then "source" this data on startup and cache it internally. This could of course be re-sourced or saved back to disk while Blender is running. This is so we don't run into situations where everything is running fine from the standalone CLI, but it is broken when you try to do the same from within Blender. (or vise versa)
Author
Owner

@ZedDB wrote:

What I had in mind was that the repositories and extension information would be stored outside of Blender, yes. This would also make it much easier for sys admins to maintain and setup computers with specific extension repos.

Blender would then "source" this data on startup and cache it internally. This could of course be re-sourced or saved back to disk while Blender is running. This is so we don't run into situations where everything is running fine from the standalone CLI, but it is broken when you try to do the same from within Blender. (or vise versa)

Agree there are benefits to moving this outside of Blender it would help with testing not to depend on Blender at all, syncing the repository information between Blender's UI and an external repo could can get difficult but it's solvable.

Having said that, your comments make it seem as if using Blender necessarily results in a worse CLI - which I'd dispute, we should have a good CLI interface for can handle setting up repositories, updating etc. however the internals are implemented.

> @ZedDB wrote: > > What I had in mind was that the repositories and extension information would be stored outside of Blender, yes. This would also make it much easier for sys admins to maintain and setup computers with specific extension repos. > > Blender would then "source" this data on startup and cache it internally. This could of course be re-sourced or saved back to disk while Blender is running. This is so we don't run into situations where everything is running fine from the standalone CLI, but it is broken when you try to do the same from within Blender. (or vise versa) Agree there are benefits to moving this outside of Blender it would help with testing not to depend on Blender at all, syncing the repository information between Blender's UI and an external repo could can get difficult but it's solvable. Having said that, your comments make it seem as if using Blender necessarily results in a worse CLI - which I'd dispute, we should have a good CLI interface for can handle setting up repositories, updating etc. however the internals are implemented.

Having said that, your comments make it seem as if using Blender necessarily results in a worse CLI - which I'd dispute, we should have a good CLI interface for can handle setting up repositories, updating etc. however the internals are implemented.

My point is that if we have a lot of duplicate work/code then it is very likely that one of the ways to interact with our extension system will suffer. In this case I would bet that the CLI would be the one to get left behind as that is not what will be used by most users (as others have pointed out).

So what I am proposing is to have as little code or storage duplication as possible between Blender and the CLI tool. This is to make sure that Blender is simply using an external tool to achieve our goals instead of the reverse. (IE an external tool invoking Blender to manage a folder structure.)

Note that I don't think having a fully fledged GUI to use this stand alone system in Blender will lead to a worst CLI experience. On the contrary, I think that it will probably lead to a better user experience as users can write their own CLIs and improve official one if our APIs are easy to work with.

Of course the GUI is the place where we should put the most effort as that is what most people will use. But my point is that this should only be an interface/frontend and nothing else to ensure that the package manager and Blender itself don't get fused together later down the line.

> Having said that, your comments make it seem as if using Blender necessarily results in a worse CLI - which I'd dispute, we should have a good CLI interface for can handle setting up repositories, updating etc. however the internals are implemented. My point is that if we have a lot of duplicate work/code then it is very likely that one of the ways to interact with our extension system will suffer. In this case I would bet that the CLI would be the one to get left behind as that is not what will be used by most users (as others have pointed out). So what I am proposing is to have as little code or storage duplication as possible between Blender and the CLI tool. This is to make sure that Blender is simply using an external tool to achieve our goals instead of the reverse. (IE an external tool invoking Blender to manage a folder structure.) Note that I don't think having a fully fledged GUI to use this stand alone system in Blender will lead to a worst CLI experience. On the contrary, I think that it will probably lead to a better user experience as users can write their own CLIs and improve official one if our APIs are easy to work with. Of course the GUI is the place where we should put the most effort as that is what most people will use. But my point is that this should only be an interface/frontend and nothing else to ensure that the package manager and Blender itself don't get fused together later down the line.

Hi I believe this discussion is derailing. The extensions project is a motivation for this proposal, but it is beyond its scope discussing the project architecture.

That said, there are a few concrete questions which need answers (and for which feedback is welcome):

1.Do we want to make Blender command-line extendable?
2. If so, do we want to expose this to add-ons?
3. What should the API look like (the commands themselves)?
4. Do we need this now? (or we find a different solution for extensions which may or may not use this under the hood).

For the records, the use of the tools command-line would imply backgound mode.

Hi I believe this discussion is derailing. The extensions project is a motivation for this proposal, but it is beyond its scope discussing the project architecture. That said, there are a few concrete questions which need answers (and for which feedback is welcome): 1.Do we want to make Blender command-line extendable? 2. If so, do we want to expose this to add-ons? 3. What should the API look like (the commands themselves)? 4. Do we need this now? (or we find a different solution for extensions which may or may not use this under the hood). For the records, the use of the tools command-line would imply backgound mode.

There are a few different use cases:

  • Creating and validating extension packages.
  • Installing extensions and repositories, and more generally changing user preferences through the command line
  • Custom commands for add-ons, to some operation on actual .blend files

I think a generic --tool is doing too much, based on the .blend files loaded it seems specifically for editing preferences, but I would not guess that from the name. With the example commands in #117286, I also find it confusing that creating an extension somehow involves the user preferences.

I would rather have some distinction between these use cases, like:

# Factory startup, does not touch any user data.
blender --extension-package create
blender --extension-package validate

# userpref.blend, automatically saves changes at the end.
blender --preferences install-extension
blender --preferences add-repository
blender --preferences set <name> <value>

# userpref.blend and startup.blend or other .blend from command line
blender --command cycles-denoise-animation

For the first use case, I can see the argument that it should be its own CLI executable. I can't say I really mind being part of the regular blender command. Is it really much better to have a blender-extension-package rather than blender --extension-package from and end user point of view or implementation complexity in Blender?

The main downside I see is that if you want to package extensions as part of GitHub CI, or some extensions server, then having to run the entire Blender executable is not great. But on the other hand, if you don't have the Blender executable you also can't run integration or end-to-end tests, so it's sort of required.

I'm not against having that part in its own CLI command line tool as a design choice. But also not sure it's worth dealing with the work to build, install, code-sign that. And figure out how to make it available in Snap, Flatpak, Steam, Blender.app, etc.

For me options like --preferences and --command logically make sense as part of the Blender executable, and could be extensible by add-ons but preferably as two separate things instead of one --tool. I think it's fine for --extension-package to be built-in and not extensible by add-ons.

There are a few different use cases: * Creating and validating extension packages. * Installing extensions and repositories, and more generally changing user preferences through the command line * Custom commands for add-ons, to some operation on actual .blend files I think a generic `--tool` is doing too much, based on the .blend files loaded it seems specifically for editing preferences, but I would not guess that from the name. With the example commands in #117286, I also find it confusing that creating an extension somehow involves the user preferences. I would rather have some distinction between these use cases, like: ```sh # Factory startup, does not touch any user data. blender --extension-package create blender --extension-package validate # userpref.blend, automatically saves changes at the end. blender --preferences install-extension blender --preferences add-repository blender --preferences set <name> <value> # userpref.blend and startup.blend or other .blend from command line blender --command cycles-denoise-animation ``` For the first use case, I can see the argument that it should be its own CLI executable. I can't say I really mind being part of the regular blender command. Is it really much better to have a `blender-extension-package` rather than `blender --extension-package` from and end user point of view or implementation complexity in Blender? The main downside I see is that if you want to package extensions as part of GitHub CI, or some extensions server, then having to run the entire Blender executable is not great. But on the other hand, if you don't have the Blender executable you also can't run integration or end-to-end tests, so it's sort of required. I'm not against having that part in its own CLI command line tool as a design choice. But also not sure it's worth dealing with the work to build, install, code-sign that. And figure out how to make it available in Snap, Flatpak, Steam, Blender.app, etc. For me options like `--preferences` and `--command` logically make sense as part of the Blender executable, and could be extensible by add-ons but preferably as two separate things instead of one `--tool`. I think it's fine for `--extension-package` to be built-in and not extensible by add-ons.

@ideasman42 is this task still relevant, or can it be closed now?

@ideasman42 is this task still relevant, or can it be closed now?
Author
Owner

Yes, this can be closed, extension command line access is now possible see: blender -c extensions --help.

Yes, this can be closed, extension command line access is now possible see: `blender -c extensions --help`.
Blender Bot added the
Status
Archived
label 2024-05-24 15:12:10 +02:00
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser Project (Legacy)
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
7 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#117936
No description provided.