Proposal for extensible CLI sub-commands #117936

New Issue

Open

opened 2024-02-07 08:26:18 +01:00 by Campbell Barton · 17 comments

Campbell Barton commented 2024-02-07 08:26:18 +01:00

Owner

Copy Link

Motivation

For the Blender extensions project we would like to make command-line tools available for validating extensions, building extension archives and running automated updates.

To prevent adding too many top level command-line arguments this proposal suggests an extensible way to add sub-commands that can be used by built-in tools as well as add-ons.

See extensions task: #117286

Proposed Solution

Provide an interface for scripts to register CLI tools and a convenient way to access them.

Note that the exact details for how this is implemented is an open topic,
a suggested implementation is included here.

Usage:

• Add a new --tool command-line argument.
• All trailing arguments would be passed to the handler function, so: blender --tool extensions update
  would call the "extensions" handler passing in ["update"].
• By convention all tools are expected to have their own --help message, e.g. blender --tool extensions --help.
• Blender's --help message would list all registered --tool arguments.
  
• Scripts (including add-ons) may register their own tools.

Implementation:

• Two Python API functions would be added:
  bpy.utils.register_cli_tool("name", handler_fn) along with an unregister_cli_tool.
  These would wrap C++ functions, allowing native C++ sub-commands too.
• Using --tool implies --background.
• Using --tool loads the user preferences but not the users startup.blend.
  Avoid tools running slowly if the user happens to have a heavy startup.
• Blender's built-in argument parsing stops after --tool {ARGUMENT} (no need to use --).
• The handler takes a list of strings and returns an integer (used for the exit-code).

---

Pros:

• Convenient to access from the Blender command-line.
• Extensible (add-ons can easily add their own).
• Discoverable (shown in --help).

Cons:

• Calling blender --tool extensions is less convenient than a commend - blender-extensions for instance.
  For developers who might call these frequently we could recommend they setup aliases to tools they use frequently
  with the convention of blender --tool example -> blender-example.
• We can't take advantage of knowledge about what the tool requires in order avoid redundant work on startup.
  As add-ons need to be loaded, the user preferences also need to be loaded.
  In general it's impractical to declare that a tool doesn't require a Blender session temporary directory or any other details relating to Blender's environment which is typically initialized before loading add-ons.
  
  While this is a solvable problem, I don't see how we could solve this without involved auto-discovery of tools and their requirements which runs before low-level systems are initialized but after all enabled add-ons are known. This adds considerable overhead/complexity for something I believe we can manage without.
• Added overhead for running --help which needs to load all add-ons to display the tools.
  If loading Blender's internals to display help text is a problem: --tool help could be used to discover tools,
  matching --engine functionality.

Alternative Solutions

For completeness, I've listed some alternative solutions with their pros/cons.

The main advantage with them is we don't need to create & maintain an API for CLI tools in Blender
although I find the results are often less convenient.

Wrapper Scripts

It is already possible to create a wrapper script that runs a command-line tool that's bundled with an add-on.

If we wanted to use this solution, scripts for Linux/macOS/Windows would need to be included with Blender,
so Blender would come with additional tools which would be available along-side Blender.

This example shows how 'scripts/addons/my_addon/cli/my_custom_tool.py can be run from Blender on the command-line.

#!/bin/sh
BF_DIST_BIN=$(dirname "$0")
BF_PROGRAM="blender"

exec "$BF_DIST_BIN/$BF_PROGRAM" \
    --background \
    -noaudio \
    --python-expr \
    "import os, sys, subprocess, bpy; sys.exit(subprocess.call([sys.executable, os.path.join(bpy.utils.system_resource('SCRIPTS'), 'addons, 'my_addon, 'cli', 'my_custom_tool.py'), *sys.argv[sys.argv.index('--') + 1:]]))" -- "$@"

Pros:

• Doesn't require a specialized API (simply rely on wrapper script).
• Commands are discoverable and convenient to access.

Cons:

• 3rd party add-ons would not be able to register their own tools.
• With added visibility & inclusion in the PATH (for Linux at least) there is an incentive to avoid including obscure functionality in Blender's base directory. Using a sub-command has the advantage we can include more obscure actions without making a strong argument for adding additional top-level commands.
  Arguably a blender-extensions command could justify it's own top-level wrapper script, however we might take advantage of sub-commands in places where top-level wrapper scripts aren't as appropriate.
• Potential portability issues as we would have to support wrapper scripts on Linux/macOS/Windows, snap packages.

Support Modules as Scripts

Python supports -m module-name which terminates argument parsing and is used to implement sub-commands.

This example starts a local file server using http.server:

python3 -m http.server 3333

We could add an equivalent --python-module argument which allows for any module to be accessed from the command-line.

Pros:

• Doesn't require a specialized API (simply loads any module).
• It's making use of an existing convention - even allowing built-in Python modules to be used via Blender.

Cons:

• The full module path must be known, with newly added extensions,
  these are name-spaced so the module name depends on the extension repositories preferences.
• Not discoverable as any Python module might provide a command-line interface (solvable via documentation).
• Less convenient as it requires --background to be passed in for typical usage.

NOTE: we could support this whatever the outcome of this proposal, although it doesn't seem like a priority either.

Stand Alone Tools

We could release a command-line tool to deal with Blender extensions, this has a few down-sides though.

---

Pros:

• Doesn't require any integration with Blender.
• Once installed & in the users PATH, accessing is more convenient.
• May run faster (avoid the overhead of running Blender's binary to access a command-line tool).

Cons:

• A separate tool would need a way to reference the blender binary for Blender specific actions such as upgrading packages (take a --blender argument for instance).
• A separate tool could become outdated or incompatible with the version of Blender in use.
• The additional overhead for us to manage releases of a 3rd party tools as well as users to install and update a separate packaging tool.
• Adding additional tools requires a separate release process, users to install etc.
• Add-ons could not easily add their own tools.

Motivation ========== For the Blender extensions project we would like to make command-line tools available for validating extensions, building extension archives and running automated updates. To prevent adding too many top level command-line arguments this proposal suggests an extensible way to add sub-commands that can be used by built-in tools as well as add-ons. See extensions task: #117286 Proposed Solution ================= Provide an interface for scripts to register CLI tools and a convenient way to access them. Note that the exact details for how this is implemented is an open topic, a suggested implementation is included here. **Usage:** - Add a new `--tool` command-line argument. - All trailing arguments would be passed to the handler function, so: `blender --tool extensions update` would call the "extensions" handler passing in `["update"]`. - By convention all tools are expected to have their own `--help` message, e.g. `blender --tool extensions --help`. - Blender's `--help` message would list all registered `--tool` arguments.<br> - Scripts (including add-ons) may register their own tools. **Implementation:** - Two Python API functions would be added: `bpy.utils.register_cli_tool("name", handler_fn)` along with an `unregister_cli_tool`.<br>*These would wrap C++ functions, allowing native C++ sub-commands too.* - Using `--tool` implies `--background`. - Using `--tool` loads the user preferences but **not** the users `startup.blend`. *Avoid tools running slowly if the user happens to have a heavy startup.* - Blender's built-in argument parsing stops after `--tool {ARGUMENT}` (no need to use `--`). - The handler takes a list of strings and returns an integer (used for the exit-code). --- **Pros:** - Convenient to access from the Blender command-line. - Extensible (add-ons can easily add their own). - Discoverable (shown in `--help`). **Cons:** - Calling `blender --tool extensions` is less convenient than a commend - `blender-extensions` for instance. For developers who might call these frequently we could recommend they setup aliases to tools they use frequently with the convention of `blender --tool example` -> `blender-example`. - We can't take advantage of knowledge about what the tool requires in order avoid redundant work on startup. As add-ons need to be loaded, the user preferences also need to be loaded. In general it's impractical to declare that a tool doesn't require a Blender session temporary directory or any other details relating to Blender's environment which is typically initialized before loading add-ons.<br> *While this is a solvable problem, I don't see how we could solve this without involved auto-discovery of tools and their requirements which runs before low-level systems are initialized but after all enabled add-ons are known. This adds considerable overhead/complexity for something I believe we can manage without.* - Added overhead for running `--help` which needs to load all add-ons to display the tools. If loading Blender's internals to display help text is a problem: `--tool help` could be used to discover tools, matching `--engine` functionality. Alternative Solutions ===================== For completeness, I've listed some alternative solutions with their pros/cons. The main advantage with them is we don't need to create & maintain an API for CLI tools in Blender although I find the results are often less convenient. ### Wrapper Scripts It is already possible to create a wrapper script that runs a command-line tool that's bundled with an add-on. If we wanted to use this solution, scripts for Linux/macOS/Windows would need to be included with Blender, so Blender would come with additional tools which would be available along-side Blender. This example shows how `'scripts/addons/my_addon/cli/my_custom_tool.py` can be run from Blender on the command-line. ```.sh #!/bin/sh BF_DIST_BIN=$(dirname "$0") BF_PROGRAM="blender" exec "$BF_DIST_BIN/$BF_PROGRAM" \ --background \ -noaudio \ --python-expr \ "import os, sys, subprocess, bpy; sys.exit(subprocess.call([sys.executable, os.path.join(bpy.utils.system_resource('SCRIPTS'), 'addons, 'my_addon, 'cli', 'my_custom_tool.py'), *sys.argv[sys.argv.index('--') + 1:]]))" -- "$@" ``` **Pros:** - Doesn't require a specialized API (simply rely on wrapper script). - Commands are discoverable and convenient to access. **Cons:** - 3rd party add-ons would not be able to register their own tools. - With added visibility & inclusion in the PATH (for Linux at least) there is an incentive to avoid including obscure functionality in Blender's base directory. Using a sub-command has the advantage we can include more obscure actions without making a strong argument for adding additional top-level commands. *Arguably a `blender-extensions` command **could** justify it's own top-level wrapper script, however we might take advantage of sub-commands in places where top-level wrapper scripts aren't as appropriate.* - Potential portability issues as we would have to support wrapper scripts on Linux/macOS/Windows, snap packages. ### Support Modules as Scripts Python supports `-m module-name` which terminates argument parsing and is used to implement sub-commands. This example starts a local file server using `http.server`: ``` python3 -m http.server 3333 ``` We could add an equivalent `--python-module` argument which allows for any module to be accessed from the command-line. **Pros:** - Doesn't require a specialized API (simply loads any module). - It's making use of an existing convention - even allowing built-in Python modules to be used via Blender. **Cons:** - The full module path must be known, with newly added extensions, these are name-spaced so the module name depends on the extension repositories preferences. - Not discoverable as any Python module *might* provide a command-line interface (solvable via documentation). - Less convenient as it requires `--background` to be passed in for typical usage. *NOTE: we could support this whatever the outcome of this proposal, although it doesn't seem like a priority either.* ### Stand Alone Tools We could release a command-line tool to deal with Blender extensions, this has a few down-sides though. --- **Pros:** - Doesn't require any integration with Blender. - Once installed & in the users PATH, accessing is more convenient. - May run faster (avoid the overhead of running Blender's binary to access a command-line tool). **Cons:** - A separate tool would need a way to reference the `blender` binary for Blender specific actions such as upgrading packages (take a `--blender` argument for instance). - A separate tool could become outdated or incompatible with the version of Blender in use. - The additional overhead for us to manage releases of a 3rd party tools as well as users to install and update a separate packaging tool. - Adding additional tools requires a separate release process, users to install etc. - Add-ons could not easily add their own tools.

🚀 1

Campbell Barton added the

Interest

Python API

Type

Design

Module

Core

labels 2024-02-07 08:26:19 +01:00

Campbell Barton referenced this issue 2024-02-07 11:23:05 +01:00

Extensions Platform: Version 1 (alpha launch) #117286

Sebastian Parborg commented 2024-02-07 11:50:13 +01:00

Member

Copy Link

To me this seems like we are trying to cram too much into one binary. Especially since the functionality is kinda orthogonal to Blender (3D creation software != package manager). IE the extensions platform is essentially only managing a folder structure on the end users computer. This folder structure is then used by Blender to load the extensions. But there shouldn't be anything that strongly ties this together with the actual blender program/binary.

I personally think that this should be a stand alone tool. More akin to how the Vim and Emacs plugin system(s) are setup.

This means that fetching, updating and installing plugins is handled by third party plugins/tools that can be invoked from outside and inside of the program. (Of course additional functionality is there if used from inside of Blender)
For Blender, we would of course ship the plugin and the tools with our releases.

Pros:

• You will never need the blender binary to package, fetch, install, or update addons.
  (However if you execute any of these steps from within Blender the tools will notify Blender when to update/refresh the updated parts)
• More flexibility. If someone wants to change how the fetching or updating works or add new functionality, they don't need to recompile blender. It is also much easier to test this from the command line.
• By making this a standalone thing it is easier for us to focus on providing stable APIs that plugin manager systems can use.

Cons:

• Less tight integration with Blender (however I would argue that this is actually a pro)
• Could be cumbersome if our APIs on the Blender side are very unstable and changes a lot. (This should not be the case though as the APIs should be as simple and minimal as possible)

To me this seems like we are trying to cram too much into one binary. Especially since the functionality is kinda orthogonal to Blender (3D creation software != package manager). IE the extensions platform is essentially only managing a folder structure on the end users computer. This folder structure is then used by Blender to load the extensions. But there shouldn't be anything that strongly ties this together with the actual blender program/binary. I personally think that this should be a stand alone tool. More akin to how the Vim and Emacs plugin system(s) are setup. This means that fetching, updating and installing plugins is handled by third party plugins/tools that can be invoked from outside and inside of the program. (Of course additional functionality is there if used from inside of Blender) For Blender, we would of course ship the plugin and the tools with our releases. Pros: - You will never need the blender binary to package, fetch, install, or update addons. (However if you execute any of these steps from within Blender the tools will notify Blender when to update/refresh the updated parts) - More flexibility. If someone wants to change how the fetching or updating works or add new functionality, they don't need to recompile blender. It is also much easier to test this from the command line. - By making this a standalone thing it is easier for us to focus on providing stable APIs that plugin manager systems can use. Cons: - Less tight integration with Blender (however I would argue that this is actually a pro) - Could be cumbersome if our APIs on the Blender side are very unstable and changes a lot. (This should not be the case though as the APIs should be as simple and minimal as possible)

Jacques Lucke commented 2024-02-07 12:12:44 +01:00

Member

Copy Link

My initial feeling is that trying to separate it from the Blender binary might become more annoying later on. Might be true that in the beginning these tools don't actually need the Blender binary, but it feels likely that we want to reuse some parts of the binary at some point.

Instead of blender --tool <tool-name>, I'd probably use something like blender run <tool-name>. It feels friendlier to me, but that's very subjective. It also makes it more obvious that you can't pass other arguments to Blender that you'd normally pass if you wanted to start a Blender session.

My initial feeling is that trying to separate it from the Blender binary might become more annoying later on. Might be true that in the beginning these tools don't actually need the Blender binary, but it feels likely that we want to reuse some parts of the binary at some point. Instead of `blender --tool <tool-name>`, I'd probably use something like `blender run <tool-name>`. It feels friendlier to me, but that's very subjective. It also makes it more obvious that you can't pass other arguments to Blender that you'd normally pass if you wanted to start a Blender session.

Campbell Barton commented 2024-02-07 12:57:21 +01:00

Author

Owner

Copy Link

@ZedDB what you've described regarding package management being separate from Blender is in fact already the case. The low-level package management commands don't depend on Blender at all.

Blender defines:

• The repositories directories & their URL's which are stored in the user-preferences.
• The location of blender's bundled scripts.
• The python binary (which may be bundled with Blender), important if the operating system doesn't provide a recent Python version.

Reliably accessing this information for both portable & system installs on linux/macOS/windows isn't trivial. Nevertheless if this information was available somewhere, an external tool could perform an update.

This changes each blender release and may be confusing if you have multiple Blender's installed on a system, so it's simpler if Blender can provide this information... at that point you have to run blender, so triggering an update from blender means the sub-command can deal with the details of selecting the proper Python-binary, script-files and repositories from the preferences.

@ZedDB what you've described regarding package management being separate from Blender is in fact already the case. The low-level package management commands don't depend on Blender at all. Blender defines: - The repositories directories & their URL's which are stored in the user-preferences. - The location of blender's bundled scripts. - The python binary (which may be bundled with Blender), important if the operating system doesn't provide a recent Python version. Reliably accessing this information for both portable & system installs on linux/macOS/windows isn't trivial. Nevertheless if this information was available somewhere, an external tool could perform an update. This changes each blender release and may be confusing if you have multiple Blender's installed on a system, so it's simpler if Blender can provide this information... at that point you have to run blender, so triggering an update from blender means the sub-command can deal with the details of selecting the proper Python-binary, script-files and repositories from the preferences.

Campbell Barton commented 2024-02-07 13:10:03 +01:00

Author

Owner

Copy Link

Instead of blender --tool <tool-name>, I'd probably use something like blender run <tool-name>. It feels friendlier to me, but that's very subjective. It also makes it more obvious that you can't pass other arguments to Blender that you'd normally pass if you wanted to start a Blender session.

Regarding blender run, on it's own it seems OK however this can be used in combination with other options, where having run in the middle of multiple arguments seems strange and breaks the convention of arguments beginning with a -.

blender -d run {action} makes it seem run {action} are arguments for -d.

Whereas blender -d --run {action} makes it clearer that action is an argument to --run.

We could allow this exception but I think it may lead to ambiguous argument handling issues in the future, incidentally blender run will load the file called run if it exists.

Note that I'm not all that fussed about the exact name right now.

> Instead of `blender --tool <tool-name>`, I'd probably use something like `blender run <tool-name>`. It feels friendlier to me, but that's very subjective. It also makes it more obvious that you can't pass other arguments to Blender that you'd normally pass if you wanted to start a Blender session. Regarding `blender run`, on it's own it seems OK however this can be used in combination with other options, where having run in the middle of multiple arguments seems strange and breaks the convention of arguments beginning with a `-`. `blender -d run {action}` makes it seem `run {action}` are arguments for `-d`. Whereas `blender -d --run {action}` makes it clearer that `action` is an argument to `--run`. We could allow this exception but I think it may lead to ambiguous argument handling issues in the future, incidentally `blender run` will load the file called `run` if it exists. *Note that I'm not all that fussed about the exact name right now.*

Sebastian Parborg commented 2024-02-07 14:29:54 +01:00

Member

Copy Link

@ideasman42 thanks for the explanation!

• The repositories directories & their URL's which are stored in the user-preferences.
• The location of blender's bundled scripts.
• The python binary (which may be bundled with Blender), important if the operating system doesn't provide a recent Python version.

Wouldn't these be defined at compile time? So it shouldn't be too hard to derive where to look for these in a stand alone script that we ship with the release.
So we would have something similar to blender-thumbnailer. But it would of course be called something like blender-extensions as you suggested before.

@ideasman42 thanks for the explanation! > - The repositories directories & their URL's which are stored in the user-preferences. > - The location of blender's bundled scripts. > - The python binary (which may be bundled with Blender), important if the operating system doesn't provide a recent Python version. Wouldn't these be defined at compile time? So it shouldn't be too hard to derive where to look for these in a stand alone script that we ship with the release. So we would have something similar to `blender-thumbnailer`. But it would of course be called something like `blender-extensions` as you suggested before.

Campbell Barton commented 2024-02-07 14:42:30 +01:00

Author

Owner

Copy Link

@ZedDB these are known at compile time, yes, for portable builds they're relative to blender's base directory. So if we install a script with Blender, that can work.

User resource locations can be configured with BLENDER_USER_RESOURCES and some other environment variables, on windows - user-resources can be relative to the portable build too IIRC. My concern is that we would have to match this logic exactly with external scripts, any corner case where it differs is a bug but it can work and wouldn't be causing us problems for common cases.

There is still the case of having to either to extract the repository information from the preferences (or store it separately for more convenient access). If that's handled an external script that doesn't run Blender could be constructed & run as part of building Blender.

@ZedDB these are known at compile time, yes, for portable builds they're relative to blender's base directory. So if we install a script with Blender, that can work. User resource locations can be configured with `BLENDER_USER_RESOURCES` and some other environment variables, on windows - user-resources can be relative to the portable build too IIRC. My concern is that we would have to match this logic exactly with external scripts, any corner case where it differs is a bug but it can work and wouldn't be causing us problems for common cases. There is still the case of having to either to extract the repository information from the preferences (or store it separately for more convenient access). If that's handled an external script that doesn't run Blender could be constructed & run as part of building Blender.

Sebastian Parborg commented 2024-02-07 15:25:08 +01:00

Member

Copy Link

There is still the case of having to either to extract the repository information from the preferences (or store it separately for more convenient access). If that's handled an external script that doesn't run Blender could be constructed & run as part of building Blender.

I would say that this information is package manager specific and should be stored separately. Especially if the package manager is already stand alone as you say.

This would also make it easier to have good test coverage of the functionality of the package manager without having to involve Blender in there as well.

> There is still the case of having to either to extract the repository information from the preferences (or store it separately for more convenient access). If that's handled an external script that doesn't run Blender could be constructed & run as part of building Blender. I would say that this information is package manager specific and should be stored separately. Especially if the package manager is already stand alone as you say. This would also make it easier to have good test coverage of the functionality of the package manager without having to involve Blender in there as well.

Bastien Montagne commented 2024-02-07 19:16:55 +01:00

Owner

Copy Link

I don't have a strong opinion here, but proposed solution sounds fine to me.

Note that am not sure about debates regarding the 'ease of use' of each solutions - I have the feeling that for the vast majority of users nowadays, having to use the command line will never be considered as 'easy to use'.

I don't have a strong opinion here, but proposed solution sounds fine to me. Note that am not sure about debates regarding the 'ease of use' of each solutions - I have the feeling that for the vast majority of users nowadays, having to use the command line will never be considered as 'easy to use'.

Campbell Barton commented 2024-02-07 22:22:53 +01:00

Author

Owner

Copy Link

There is still the case of having to either to extract the repository information from the preferences (or store it separately for more convenient access). If that's handled an external script that doesn't run Blender could be constructed & run as part of building Blender.

I would say that this information is package manager specific and should be stored separately. Especially if the package manager is already stand alone as you say.

We could do this and I'm open to it, the main consideration is how we would store preference data outside of Blender & synchronize them properly.
If your proposing to remove extension-repositories from preferences and have Blender's preferences UI & manage external external data, that's a bigger change and I'd rather handle this as an stand-alone proposal.

This would also make it easier to have good test coverage of the functionality of the package manager without having to involve Blender in there as well.

It's already the case that most tests don't involve Blender, although this only the lower level actions are being tested (create repo, create an extension, install an extension over http/filesystem, updating etc.).

> > There is still the case of having to either to extract the repository information from the preferences (or store it separately for more convenient access). If that's handled an external script that doesn't run Blender could be constructed & run as part of building Blender. > > I would say that this information is package manager specific and should be stored separately. Especially if the package manager is already stand alone as you say. We could do this and I'm open to it, the main consideration is how we would store preference data outside of Blender & synchronize them properly. If your proposing to remove extension-repositories from preferences and have Blender's preferences UI & manage external external data, that's a bigger change and I'd rather handle this as an stand-alone proposal. > This would also make it easier to have good test coverage of the functionality of the package manager without having to involve Blender in there as well. It's already the case that most tests don't involve Blender, although this only the lower level actions are being tested (create repo, create an extension, install an extension over http/filesystem, updating etc.).

Campbell Barton commented 2024-02-07 22:38:54 +01:00

Author

Owner

Copy Link

Note that am not sure about debates regarding the 'ease of use' of each solutions - I have the feeling that for the vast majority of users nowadays, having to use the command line will never be considered as 'easy to use'.

Right, for something easy to use:

• A technical user can run an update from a command-line (most likely other basic operations too).
• Extension authors can create/validate/build (in the future upload) their extensions.

That's not a high bar, it's just at the moment we don't have a convenient way to do it.

> Note that am not sure about debates regarding the 'ease of use' of each solutions - I have the feeling that for the vast majority of users nowadays, having to use the command line will never be considered as 'easy to use'. Right, for something easy to use: - A technical user can run an update from a command-line (most likely other basic operations too). - Extension authors can create/validate/build (in the future upload) their extensions. That's not a high bar, it's just at the moment we don't have a convenient way to do it.

Sybren A. Stüvel commented 2024-02-08 12:12:16 +01:00

Member

Copy Link

The description reads:

Scripts (including add-ons) may register their own tools.
[...]
See extensions task: #117286

That task description doesn't mention "tool" or "cli". I feel that this discussion gets a bit muddled (and still I'm joining below, yay) as it's not quite clear to me what concrete "tools" would be added by extensions.

The use case of having an extension package manager CLI interface is very concrete. I'm not convinced that such a meta-operation ("managing Blender", rather than "running Blender") should be handled the same way as "extensible CLI tools that are likely to depend on Blender itself".

Without having a clear definition of what these extension CLI tools can or cannot do, it's hard to come up with the best way to design them.

@ZedDB wrote:
To me this seems like we are trying to cram too much into one binary.

I agree with this sentiment.

@JacquesLucke wrote:
My initial feeling is that trying to separate it from the Blender binary might become more annoying later on. Might be true that in the beginning these tools don't actually need the Blender binary, but it feels likely that we want to reuse some parts of the binary at some point.

This I also agree with.

I think this PR, with the nice list of alternatives (thanks @ideasman42 for being so extensive!) illuminates the current limitations of Blender's architecture. Ideally it would be possible to simply create another binary that links against just a few of Blender's internal libraries. That way we can have a tool that's specific to managing Blender extensions, and that still shares certain logic with Blender. And yeah, I put the word "ideally" in there intentionally, as I know that this would require considerably more work than we have time for right now.

But still, maybe there are a few things that we can link directly, or with some slight refactoring. I don't think "it has to be able to find the bundled Python" is good enough reason to stop thinking about this direction. Usually moving towards separately-usable code also improves testability, documentability, and general code quality due to looser coupling.

Maybe a separate binary that always sits next to the main Blender executable is the way to go? That way that binary can run Blender when necessary, as it knows exactly which binary to use. It can provide a clean CLI that avoids mixing Blender arguments with extension-management arguments. For example, I wouldn't want to have to pass --noaudio and --background for all extension operations. This could work well for meta-operations, like package management.

Going back to what I wrote at the top: the PR is written in such a way that a bunch of the design ideas for these tools are actually mentioned in the implementation details. This makes talking about alternative implementations hard, because that would also have to change the design ideas (instead of the implementation following the design).

The description reads: > Scripts (including add-ons) may register their own tools. > [...] > See extensions task: #117286 That task description doesn't mention "tool" or "cli". I feel that this discussion gets a bit muddled (and still I'm joining below, yay) as it's not quite clear to me what concrete "tools" would be added by extensions. The use case of having an extension package manager CLI interface is very concrete. I'm not convinced that such a meta-operation ("managing Blender", rather than "running Blender") should be handled the same way as "extensible CLI tools that are likely to depend on Blender itself". Without having a clear definition of what these extension CLI tools can or cannot do, it's hard to come up with the best way to design them. > @ZedDB wrote: > To me this seems like we are trying to cram too much into one binary. I agree with this sentiment. > @JacquesLucke wrote: > My initial feeling is that trying to separate it from the Blender binary might become more annoying later on. Might be true that in the beginning these tools don't actually need the Blender binary, but it feels likely that we want to reuse some parts of the binary at some point. This I also agree with. I think this PR, with the nice list of alternatives (thanks @ideasman42 for being so extensive!) illuminates the current limitations of Blender's architecture. Ideally it would be possible to simply create another binary that links against just a few of Blender's internal libraries. That way we can have a tool that's specific to managing Blender extensions, and that still shares certain logic with Blender. And yeah, I put the word "ideally" in there intentionally, as I know that this would require considerably more work than we have time for right now. But still, maybe there are a few things that we can link directly, or with some slight refactoring. I don't think "it has to be able to find the bundled Python" is good enough reason to stop thinking about this direction. Usually moving towards separately-usable code also improves testability, documentability, and general code quality due to looser coupling. Maybe a separate binary that always sits next to the main Blender executable is the way to go? That way that binary can run Blender when necessary, as it knows exactly which binary to use. It can provide a clean CLI that avoids mixing Blender arguments with extension-management arguments. For example, I wouldn't want to have to pass `--noaudio` and `--background` for all extension operations. This could work well for meta-operations, like package management. Going back to what I wrote at the top: the PR is written in such a way that a bunch of the design ideas for these tools are actually mentioned in the implementation details. This makes talking about alternative implementations hard, because that would also have to change the design ideas (instead of the implementation following the design).

Campbell Barton commented 2024-02-08 13:19:04 +01:00

Author

Owner

Copy Link

In general using a separate binary is interesting but a bigger project, agree the kinds of refactoring encourage better design & decoupling libraries.

@dr.sybren wrote:
That task description doesn't mention "tool" or "cli".

The task mentioned adding --extensions command line argument, I wasn't so keen on this as it would add many command line arguments, thisj proposal is suggesting an alternative - although I should have made that clear.

@dr.sybren wrote:
I don't think "it has to be able to find the bundled Python" is good enough reason to stop thinking about this direction.

Agree, the current implementation of extensions depends on reading user preferences & paths from GHOST, we could refactor these areas so it's possible to access paths & preferences without pulling in other dependencies.

@dr.sybren wrote:
For example, I wouldn't want to have to pass --noaudio and --background for all extension operations.

The proposal states that --tool implies --background although it seems reasonable to disable audio as well.

In general using a separate binary is interesting but a bigger project, agree the kinds of refactoring encourage better design & decoupling libraries. > @dr.sybren wrote: > That task description doesn't mention "tool" or "cli". The task mentioned adding `--extensions` command line argument, I wasn't so keen on this as it would add many command line arguments, thisj proposal is suggesting an alternative - although I should have made that clear. > @dr.sybren wrote: > I don't think "it has to be able to find the bundled Python" is good enough reason to stop thinking about this direction. Agree, the current implementation of extensions depends on reading user preferences & paths from GHOST, we could refactor these areas so it's possible to access paths & preferences without pulling in other dependencies. > @dr.sybren wrote: > For example, I wouldn't want to have to pass `--noaudio` and `--background` for all extension operations. The proposal states that `--tool` implies `--background` although it seems reasonable to disable audio as well.

Sebastian Parborg commented 2024-02-08 16:50:47 +01:00

Member

Copy Link

We could do this and I'm open to it, the main consideration is how we would store preference data outside of Blender & synchronize them properly.
If your proposing to remove extension-repositories from preferences and have Blender's preferences UI & manage external external data, that's a bigger change and I'd rather handle this as an stand-alone proposal.

What I had in mind was that the repositories and extension information would be stored outside of Blender, yes. This would also make it much easier for sys admins to maintain and setup computers with specific extension repos.

Blender would then "source" this data on startup and cache it internally. This could of course be re-sourced or saved back to disk while Blender is running. This is so we don't run into situations where everything is running fine from the standalone CLI, but it is broken when you try to do the same from within Blender. (or vise versa)

>We could do this and I'm open to it, the main consideration is how we would store preference data outside of Blender & synchronize them properly. If your proposing to remove extension-repositories from preferences and have Blender's preferences UI & manage external external data, that's a bigger change and I'd rather handle this as an stand-alone proposal. What I had in mind was that the repositories and extension information would be stored outside of Blender, yes. This would also make it much easier for sys admins to maintain and setup computers with specific extension repos. Blender would then "source" this data on startup and cache it internally. This could of course be re-sourced or saved back to disk while Blender is running. This is so we don't run into situations where everything is running fine from the standalone CLI, but it is broken when you try to do the same from within Blender. (or vise versa)

Campbell Barton commented 2024-02-09 06:13:55 +01:00

Author

Owner

Copy Link

@ZedDB wrote:

What I had in mind was that the repositories and extension information would be stored outside of Blender, yes. This would also make it much easier for sys admins to maintain and setup computers with specific extension repos.

Blender would then "source" this data on startup and cache it internally. This could of course be re-sourced or saved back to disk while Blender is running. This is so we don't run into situations where everything is running fine from the standalone CLI, but it is broken when you try to do the same from within Blender. (or vise versa)

Agree there are benefits to moving this outside of Blender it would help with testing not to depend on Blender at all, syncing the repository information between Blender's UI and an external repo could can get difficult but it's solvable.

Having said that, your comments make it seem as if using Blender necessarily results in a worse CLI - which I'd dispute, we should have a good CLI interface for can handle setting up repositories, updating etc. however the internals are implemented.

> @ZedDB wrote: > > What I had in mind was that the repositories and extension information would be stored outside of Blender, yes. This would also make it much easier for sys admins to maintain and setup computers with specific extension repos. > > Blender would then "source" this data on startup and cache it internally. This could of course be re-sourced or saved back to disk while Blender is running. This is so we don't run into situations where everything is running fine from the standalone CLI, but it is broken when you try to do the same from within Blender. (or vise versa) Agree there are benefits to moving this outside of Blender it would help with testing not to depend on Blender at all, syncing the repository information between Blender's UI and an external repo could can get difficult but it's solvable. Having said that, your comments make it seem as if using Blender necessarily results in a worse CLI - which I'd dispute, we should have a good CLI interface for can handle setting up repositories, updating etc. however the internals are implemented.

Sebastian Parborg commented 2024-02-09 11:53:01 +01:00

Member

Copy Link

Having said that, your comments make it seem as if using Blender necessarily results in a worse CLI - which I'd dispute, we should have a good CLI interface for can handle setting up repositories, updating etc. however the internals are implemented.

My point is that if we have a lot of duplicate work/code then it is very likely that one of the ways to interact with our extension system will suffer. In this case I would bet that the CLI would be the one to get left behind as that is not what will be used by most users (as others have pointed out).

So what I am proposing is to have as little code or storage duplication as possible between Blender and the CLI tool. This is to make sure that Blender is simply using an external tool to achieve our goals instead of the reverse. (IE an external tool invoking Blender to manage a folder structure.)

Note that I don't think having a fully fledged GUI to use this stand alone system in Blender will lead to a worst CLI experience. On the contrary, I think that it will probably lead to a better user experience as users can write their own CLIs and improve official one if our APIs are easy to work with.

Of course the GUI is the place where we should put the most effort as that is what most people will use. But my point is that this should only be an interface/frontend and nothing else to ensure that the package manager and Blender itself don't get fused together later down the line.

> Having said that, your comments make it seem as if using Blender necessarily results in a worse CLI - which I'd dispute, we should have a good CLI interface for can handle setting up repositories, updating etc. however the internals are implemented. My point is that if we have a lot of duplicate work/code then it is very likely that one of the ways to interact with our extension system will suffer. In this case I would bet that the CLI would be the one to get left behind as that is not what will be used by most users (as others have pointed out). So what I am proposing is to have as little code or storage duplication as possible between Blender and the CLI tool. This is to make sure that Blender is simply using an external tool to achieve our goals instead of the reverse. (IE an external tool invoking Blender to manage a folder structure.) Note that I don't think having a fully fledged GUI to use this stand alone system in Blender will lead to a worst CLI experience. On the contrary, I think that it will probably lead to a better user experience as users can write their own CLIs and improve official one if our APIs are easy to work with. Of course the GUI is the place where we should put the most effort as that is what most people will use. But my point is that this should only be an interface/frontend and nothing else to ensure that the package manager and Blender itself don't get fused together later down the line.

Dalai Felinto commented 2024-02-12 14:31:21 +01:00

Owner

Copy Link

Hi I believe this discussion is derailing. The extensions project is a motivation for this proposal, but it is beyond its scope discussing the project architecture.

That said, there are a few concrete questions which need answers (and for which feedback is welcome):

1.Do we want to make Blender command-line extendable?
2. If so, do we want to expose this to add-ons?
3. What should the API look like (the commands themselves)?
4. Do we need this now? (or we find a different solution for extensions which may or may not use this under the hood).

For the records, the use of the tools command-line would imply backgound mode.

Hi I believe this discussion is derailing. The extensions project is a motivation for this proposal, but it is beyond its scope discussing the project architecture. That said, there are a few concrete questions which need answers (and for which feedback is welcome): 1.Do we want to make Blender command-line extendable? 2. If so, do we want to expose this to add-ons? 3. What should the API look like (the commands themselves)? 4. Do we need this now? (or we find a different solution for extensions which may or may not use this under the hood). For the records, the use of the tools command-line would imply backgound mode.

👍 1

Brecht Van Lommel commented 2024-02-12 16:21:54 +01:00

Owner

Copy Link

There are a few different use cases:

• Creating and validating extension packages.
• Installing extensions and repositories, and more generally changing user preferences through the command line
• Custom commands for add-ons, to some operation on actual .blend files

I think a generic --tool is doing too much, based on the .blend files loaded it seems specifically for editing preferences, but I would not guess that from the name. With the example commands in #117286, I also find it confusing that creating an extension somehow involves the user preferences.

I would rather have some distinction between these use cases, like:

# Factory startup, does not touch any user data.
blender --extension-package create
blender --extension-package validate

# userpref.blend, automatically saves changes at the end.
blender --preferences install-extension
blender --preferences add-repository
blender --preferences set <name> <value>

# userpref.blend and startup.blend or other .blend from command line
blender --command cycles-denoise-animation

For the first use case, I can see the argument that it should be its own CLI executable. I can't say I really mind being part of the regular blender command. Is it really much better to have a blender-extension-package rather than blender --extension-package from and end user point of view or implementation complexity in Blender?

The main downside I see is that if you want to package extensions as part of GitHub CI, or some extensions server, then having to run the entire Blender executable is not great. But on the other hand, if you don't have the Blender executable you also can't run integration or end-to-end tests, so it's sort of required.

I'm not against having that part in its own CLI command line tool as a design choice. But also not sure it's worth dealing with the work to build, install, code-sign that. And figure out how to make it available in Snap, Flatpak, Steam, Blender.app, etc.

For me options like --preferences and --command logically make sense as part of the Blender executable, and could be extensible by add-ons but preferably as two separate things instead of one --tool. I think it's fine for --extension-package to be built-in and not extensible by add-ons.

There are a few different use cases: * Creating and validating extension packages. * Installing extensions and repositories, and more generally changing user preferences through the command line * Custom commands for add-ons, to some operation on actual .blend files I think a generic `--tool` is doing too much, based on the .blend files loaded it seems specifically for editing preferences, but I would not guess that from the name. With the example commands in #117286, I also find it confusing that creating an extension somehow involves the user preferences. I would rather have some distinction between these use cases, like: ```sh # Factory startup, does not touch any user data. blender --extension-package create blender --extension-package validate # userpref.blend, automatically saves changes at the end. blender --preferences install-extension blender --preferences add-repository blender --preferences set <name> <value> # userpref.blend and startup.blend or other .blend from command line blender --command cycles-denoise-animation ``` For the first use case, I can see the argument that it should be its own CLI executable. I can't say I really mind being part of the regular blender command. Is it really much better to have a `blender-extension-package` rather than `blender --extension-package` from and end user point of view or implementation complexity in Blender? The main downside I see is that if you want to package extensions as part of GitHub CI, or some extensions server, then having to run the entire Blender executable is not great. But on the other hand, if you don't have the Blender executable you also can't run integration or end-to-end tests, so it's sort of required. I'm not against having that part in its own CLI command line tool as a design choice. But also not sure it's worth dealing with the work to build, install, code-sign that. And figure out how to make it available in Snap, Flatpak, Steam, Blender.app, etc. For me options like `--preferences` and `--command` logically make sense as part of the Blender executable, and could be extensible by add-ons but preferably as two separate things instead of one `--tool`. I think it's fine for `--extension-package` to be built-in and not extensible by add-ons.

👍 2

Campbell Barton referenced this issue 2024-03-06 08:53:58 +01:00

CLI: support defining custom commands via C++ & Python API #119115

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

brush-assets-project

universal-scene-description

fix-121021

blender-v4.1-release

blender-v3.3-release

blender-v3.6-release

blender-v3.6-temp_wmoss_animrig_public

temp-sculpt-dyntopo

gpencil-next

anim/animation-id-113594

blender-v4.0-release

blender-projects-basics

bridge-curves

sculpt-blender

asset-browser-frontend-split

asset-shelf

tmp-usd-python-mtl

tmp-usd-3.6

blender-v3.5-release

blender-v2.93-release

realtime-clock

sculpt-dev

bevelv2

xr-dev

v4.1.1

v3.6.11

v3.3.18

v4.1.0

v3.3.17

v3.6.10

v3.6.9

v3.3.16

v3.6.8

v3.6.7

v3.3.14

v4.0.2

v4.0.1

v4.0.0

v3.6.5

v3.3.12

v3.6.4

v3.6.3

v3.3.11

v3.6.2

v3.3.10

v3.6.1

v3.3.9

v3.6.0

v3.3.8

v3.3.7

v2.93.18

v3.5.1

v3.3.6

v2.93.17

v3.5.0

v2.93.16

v3.3.5

v3.3.4

v2.93.15

v2.93.14

v3.3.3

v2.93.13

v2.93.12

v3.4.1

v3.3.2

v3.4.0

v3.3.1

v2.93.11

v3.3.0

v3.2.2

v2.93.10

v3.2.1

v3.2.0

v2.83.20

v2.93.9

v3.1.2

v3.1.1

v3.1.0

v2.83.19

v2.93.8

v3.0.1

v2.93.7

v3.0.0

v2.93.6

v2.93.5

v2.83.18

v2.93.4

v2.93.3

v2.83.17

v2.93.2

v2.93.1

v2.83.16

v2.93.0

v2.83.15

v2.83.14

v2.83.13

v2.92.0

v2.83.12

v2.91.2

v2.83.10

v2.91.0

v2.83.9

v2.83.8

v2.83.7

v2.90.1

v2.83.6.1

v2.83.6

v2.90.0

v2.83.5

v2.83.4

v2.83.3

v2.83.2

v2.83.1

v2.83

v2.82a

v2.82

v2.81a

v2.81

v2.80

v2.80-rc3

v2.80-rc2

v2.80-rc1

v2.79b

v2.79a

v2.79

v2.79-rc2

v2.79-rc1

v2.78c

v2.78b

v2.78a

v2.78

v2.78-rc2

v2.78-rc1

v2.77a

v2.77

v2.77-rc2

v2.77-rc1

v2.76b

v2.76a

v2.76

v2.76-rc3

v2.76-rc2

v2.76-rc1

v2.75a

v2.75

v2.75-rc2

v2.75-rc1

v2.74

v2.74-rc4

v2.74-rc3

v2.74-rc2

v2.74-rc1

v2.73a

v2.73

v2.73-rc1

v2.72b

2.72b

v2.72a

v2.72

v2.72-rc1

v2.71

v2.71-rc2

v2.71-rc1

v2.70a

v2.70

v2.70-rc2

v2.70-rc

v2.69

v2.68a

v2.68

v2.67b

v2.67a

v2.67

v2.66a

v2.66

v2.65a

v2.65

v2.64a

v2.64

v2.63a

v2.63

v2.61

v2.60a

v2.60

v2.59

v2.58a

v2.58

v2.57b

v2.57a

v2.57

v2.56a

v2.56

v2.55

v2.54

v2.53

v2.52

v2.51

v2.50

v2.49b

v2.49a

v2.49

v2.48a

v2.48

v2.47

v2.46

v2.45

v2.44

v2.43

v2.42a

v2.42

v2.41

v2.40

v2.37a

v2.37

v2.36

v2.35a

v2.35

v2.34

v2.33a

v2.33

v2.32

v2.31a

v2.31

v2.30

v2.28c

v2.28a

v2.28

v2.27

v2.26

v2.25

Labels

Clear labels   

Interest

Alembic

  

Interest

Animation & Rigging

  

Interest

Asset Browser

  

Interest

Asset Browser Project Overview

  

Interest

Audio

  

Interest

Automated Testing

  

Interest

Blender Asset Bundle

  

Interest

BlendFile

  

Interest

Collada

  

Interest

Compatibility

This issue affects/is about backward or forward compatibility

  

Interest

Compositing

  

Interest

Core

  

Interest

Cycles

  

Interest

Dependency Graph

  

Interest

Development Management

  

Interest

EEVEE

  

Interest

EEVEE & Viewport

  

Interest

Freestyle

  

Interest

Geometry Nodes

  

Interest

Grease Pencil

  

Interest

ID Management

  

Interest

Images & Movies

  

Interest

Import Export

  

Interest

Line Art

  

Interest

Masking

  

Interest

Metal

  

Interest

Modeling

  

Interest

Modifiers

  

Interest

Motion Tracking

  

Interest

Nodes & Physics

  

Interest

OpenGL

  

Interest

Overlay

  

Interest

Overrides

  

Interest

Performance

  

Interest

Physics

  

Interest

Pipeline, Assets & IO

  

Interest

Platforms, Builds & Tests

  

Interest

Python API

  

Interest

Render & Cycles

  

Interest

Render Pipeline

  

Interest

Sculpt, Paint & Texture

  

Interest

Text Editor

  

Interest

Translations

  

Interest

Triaging

  

Interest

Undo

  

Interest

USD

  

Interest

User Interface

  

Interest

UV Editing

  

Interest

VFX & Video

  

Interest

Video Sequencer

  

Interest

Virtual Reality

  

Interest

Vulkan

  

Interest

Wayland

Issues relating to Wayland windowing support.

  

Interest

Workbench

  

Interest: X11

Issues relating to Xorg/X11 support.

  

Legacy

Blender 2.8 Project

  

Legacy

Milestone 1: Basic, Local Asset Browser

  

Legacy

OpenGL Error

  

Meta

Good First Issue

  

Meta

Papercut

  

Meta

Retrospective

  

Meta

Security

Issues relating to security: https://wiki.blender.org/wiki/Process/Vulnerability_Reports

  

Module

Animation & Rigging

  

Module

Core

  

Module

Development Management

  

Module

EEVEE & Viewport

  

Module

Grease Pencil

  

Module

Modeling

  

Module

Nodes & Physics

  

Module

Pipeline, Assets & IO

  

Module

Platforms, Builds & Tests

  

Module

Python API

  

Module

Render & Cycles

  

Module

Sculpt, Paint & Texture

  

Module

Triaging

  

Module

User Interface

  

Module

VFX & Video

  

Platform

FreeBSD

  

Platform

Linux

  

Platform

macOS

  

Platform

Windows

  

Priority

High

  

Priority

Low

  

Priority

Normal

  

Priority

Unbreak Now!

  

Status

Archived

  

Status

Confirmed

  

Status

Duplicate

  

Status

Needs Info from Developers

  

Status

Needs Information from User

  

Status

Needs Triage

  

Status

Resolved

  

Type

Bug

  

Type

Design

  

Type

Known Issue

  

Type

Patch

  

Type

Report

  

Type

To Do

No Label

Interest

Alembic

Interest

Animation & Rigging

Interest

Asset Browser

Interest

Asset Browser Project Overview

Interest

Audio

Interest

Automated Testing

Interest

Blender Asset Bundle

Interest

BlendFile

Interest

Collada

Interest

Compatibility

Interest

Compositing

Interest

Core

Interest

Cycles

Interest

Dependency Graph

Interest

Development Management

Interest

EEVEE

Interest

EEVEE & Viewport

Interest

Freestyle

Interest

Geometry Nodes

Interest

Grease Pencil

Interest

ID Management

Interest

Images & Movies

Interest

Import Export

Interest

Line Art

Interest

Masking

Interest

Metal

Interest

Modeling

Interest

Modifiers

Interest

Motion Tracking

Interest

Nodes & Physics

Interest

OpenGL

Interest

Overlay

Interest

Overrides

Interest

Performance

Interest

Physics

Interest

Pipeline, Assets & IO

Interest

Platforms, Builds & Tests

Interest

Python API

Interest

Render & Cycles

Interest

Render Pipeline

Interest

Sculpt, Paint & Texture

Interest

Text Editor

Interest

Translations

Interest

Triaging

Interest

Undo

Interest

USD

Interest

User Interface

Interest

UV Editing

Interest

VFX & Video

Interest

Video Sequencer

Interest

Virtual Reality

Interest

Vulkan

Interest

Wayland

Interest

Workbench

Interest: X11

Legacy

Blender 2.8 Project

Legacy

Milestone 1: Basic, Local Asset Browser

Legacy

OpenGL Error

Meta

Good First Issue

Meta

Papercut

Meta

Retrospective

Meta

Security

Module

Animation & Rigging

Module

Core

Module

Development Management

Module

EEVEE & Viewport

Module

Grease Pencil

Module

Modeling

Module

Nodes & Physics

Module

Pipeline, Assets & IO

Module

Platforms, Builds & Tests

Module

Python API

Module

Render & Cycles

Module

Sculpt, Paint & Texture

Module

Triaging

Module

User Interface

Module

VFX & Video

Platform

FreeBSD

Platform

Linux

Platform

macOS

Platform

Windows

Priority

High

Priority

Low

Priority

Normal

Priority

Unbreak Now!

Status

Archived

Status

Confirmed

Status

Duplicate

Status

Needs Info from Developers

Status

Needs Information from User

Status

Needs Triage

Status

Resolved

Type

Bug

Type

Design

Type

Known Issue

Type

Patch

Type

Report

Type

To Do

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

7 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#117936

No description provided.