blender
/
blender-developer-docs
4
4
Fork 25
Code Issues 1 Pull Requests 7 Activity

New section: Add-on Dependencies #40

Open
Sybren A. Stüvel wants to merge 1 commits from dr.sybren/blender-developer-docs:pr/addon-dependencies into main
pull from: dr.sybren/blender-developer-docs:pr/addon-dependencies
merge into: blender:main

When changing the target branch, be careful to rebase the branch in your fork to match. See documentation.
Conversation 3 Commits 1 Files Changed 2 +217
Sybren A. Stüvel commented 2024-03-08 15:41:11 +01:00
Member
Copy Link

A new section on how to load add-on dependencies, in a way that doesn't interfere with other add-ons. It's basically explaining what I did for Flamenco.

A new section on how to load add-on dependencies, in a way that doesn't interfere with other add-ons. It's basically explaining what I did for [Flamenco](https://flamenco.blender.org/).
Sybren A. Stüvel added 1 commit 2024-03-08 15:41:13 +01:00
397b6ec7f6 New section: Add-on Dependencies 
A new section on how to load add-on dependencies, in a way that doesn't
interfere with other add-ons.
Sybren A. Stüvel requested review from Campbell Barton 2024-03-08 15:41:27 +01:00
Sybren A. Stüvel requested review from Dalai Felinto 2024-03-08 15:41:28 +01:00
Dalai Felinto commented 2024-03-11 15:20:30 +01:00
Owner
Copy Link

Thanks for this. As we discussed IRL it is better to have this as part of the Python API docs, not the developer docs.

Once we have this merged I will update the user manual to point to that.

Thanks for this. As we discussed IRL it is better to have this as part of the Python API docs, not the developer docs. Once we have this merged I will update the user manual to point to that.
Campbell Barton reviewed 2024-03-15 09:19:24 +01:00
Campbell Barton left a comment
Owner
Copy Link

Her are the result of some tests, I first wanted to get an idea how well this method works when add-on developers try this with various wheels.

The results of the experiment are in this repository: https://gitlab.com/ideasman42/temp-blender-wheel-test

Findings

  • Most wheels work.
  • Some wheel names don't match the module names (support added in the test repo).
  • Modules depending on other modules is something we might want to support..
  • Binary wheels often work (users may end up including them in extensions, we could support selecting the wheel based on the architecture).
  • Wheel's may fail to load sub-modules with cryptic errors: greenlet couldn't load it's own sub-module for some reason.
  • Some modules need to load data from the file-system and can't run from a wheel: xmlschema attempted to stat a file on the file-system.

Details

I picked a handful of modules from pypi (picked from lists of popular Python modules, excluding some that pull in many other dependencies).

From 10 modules:

https://pypi.org/project/appdirs
https://pypi.org/project/lxml
https://pypi.org/project/pillow
https://pypi.org/project/python-dateutil
https://pypi.org/project/scrat
https://pypi.org/project/shortuuid
https://pypi.org/project/simple-dotdict
https://pypi.org/project/tablib
https://pypi.org/project/tomlkit
https://pypi.org/project/xmlschema

Downloaded with pip wheel -r requirements.txt -w ./my_addon/wheels, pulled in an additional 7 dependencies.

From 17 wheels, 14 loaded, although 15 would have loaded with support for multiple wheels in the sys.path at once.

Output from test script:

Loading: ('appdirs', (), None) ...OK
Loading: ('click', (), None) ...OK
Loading: ('elementpath', (), None) ...OK
Loading: ('greenlet', (), None) ...fail Unable to load 'greenlet' from /src/blender_extension_wheel_example/my_addon/wheels/greenlet-3.0.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl: No module named 'greenlet._greenlet'
Loading: ('lxml', (), None) ...OK
Loading: ('PIL', (), 'pillow') ...OK
Loading: ('dateutil', (), 'python_dateutil') ...OK
Loading: ('yaml', (), 'PyYAML') ...OK
Loading: ('scrat', (), None) ...fail Unable to load 'scrat' from /src/blender_extension_wheel_example/my_addon/wheels/scrat-0.3.0-py3-none-any.whl: No module named 'sqlalchemy'
Loading: ('shortuuid', (), None) ...OK
Loading: ('simple_dotdict', (), None) ...OK
Loading: ('six', (), None) ...OK
Loading: ('sqlalchemy', (), 'SQLAlchemy') ...OK
Loading: ('tablib', (), None) ...OK
Loading: ('tomlkit', (), None) ...OK
Loading: ('typing_extensions', (), None) ...OK
Loading: ('xmlschema', (), None) ...fail Unable to load 'xmlschema' from /src/blender_extension_wheel_example/my_addon/wheels/xmlschema-3.1.0-py3-none-any.whl: No module named 'elementpath'
ok 14 no 3
Her are the result of some tests, I first wanted to get an idea how well this method works when add-on developers try this with various wheels. The results of the experiment are in this repository: https://gitlab.com/ideasman42/temp-blender-wheel-test ### Findings - Most wheels work. - Some wheel names don't match the module names (support added in the test repo). - Modules depending on other modules is something we might want to support.. - Binary wheels often work (users may end up including them in extensions, we could support selecting the wheel based on the architecture). - Wheel's may fail to load sub-modules with cryptic errors: `greenlet` couldn't load it's own sub-module for some reason. - Some modules need to load data from the file-system and can't run from a wheel: `xmlschema` attempted to stat a file on the file-system. ---- ### Details I picked a handful of modules from pypi (picked from lists of popular Python modules, excluding some that pull in many other dependencies). From 10 modules: https://pypi.org/project/appdirs https://pypi.org/project/lxml https://pypi.org/project/pillow https://pypi.org/project/python-dateutil https://pypi.org/project/scrat https://pypi.org/project/shortuuid https://pypi.org/project/simple-dotdict https://pypi.org/project/tablib https://pypi.org/project/tomlkit https://pypi.org/project/xmlschema Downloaded with `pip wheel -r requirements.txt -w ./my_addon/wheels`, pulled in an additional 7 dependencies. From 17 wheels, 14 loaded, although 15 would have loaded with support for multiple wheels in the `sys.path` at once. Output from test script: ``` Loading: ('appdirs', (), None) ...OK Loading: ('click', (), None) ...OK Loading: ('elementpath', (), None) ...OK Loading: ('greenlet', (), None) ...fail Unable to load 'greenlet' from /src/blender_extension_wheel_example/my_addon/wheels/greenlet-3.0.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl: No module named 'greenlet._greenlet' Loading: ('lxml', (), None) ...OK Loading: ('PIL', (), 'pillow') ...OK Loading: ('dateutil', (), 'python_dateutil') ...OK Loading: ('yaml', (), 'PyYAML') ...OK Loading: ('scrat', (), None) ...fail Unable to load 'scrat' from /src/blender_extension_wheel_example/my_addon/wheels/scrat-0.3.0-py3-none-any.whl: No module named 'sqlalchemy' Loading: ('shortuuid', (), None) ...OK Loading: ('simple_dotdict', (), None) ...OK Loading: ('six', (), None) ...OK Loading: ('sqlalchemy', (), 'SQLAlchemy') ...OK Loading: ('tablib', (), None) ...OK Loading: ('tomlkit', (), None) ...OK Loading: ('typing_extensions', (), None) ...OK Loading: ('xmlschema', (), None) ...fail Unable to load 'xmlschema' from /src/blender_extension_wheel_example/my_addon/wheels/xmlschema-3.1.0-py3-none-any.whl: No module named 'elementpath' ok 14 no 3 ```
❤️ 1
docs/handbook/addons/dependencies.md
@ -0,0 +88,4 @@
_log = logging.getLogger(__name__)
def load_wheel(module_name: str, submodules: Iterable[str]) -> list[ModuleType]:
Campbell Barton commented 2024-03-15 08:44:34 +01:00
Owner
Copy Link

The module_name and the wheel name don't always match, when testing wheels I found a few which didn't use the same name [PIL / pillow, sqlalchemy / SQLAlchemy, yaml / PyYAML] for e.g.

fname_prefix can be made an optional argument to support cases when the names don't match.

The `module_name` and the wheel name don't always match, when testing wheels I found a few which didn't use the same name [`PIL` / `pillow`, `sqlalchemy` / `SQLAlchemy`, `yaml` / `PyYAML`] for e.g. `fname_prefix` can be made an optional argument to support cases when the names don't match.
👍 1
docs/handbook/addons/dependencies.md
@ -0,0 +185,4 @@
old_sysmod = sys.modules.copy()
try:
sys.path.insert(0, str(wheel_file))
Campbell Barton commented 2024-03-15 08:47:46 +01:00
Owner
Copy Link

When testing wheels I ran into cases where one wheel depends on another, if the dependencies are added here, loading those wheels worked.

This API could be extended to support dependencies indirect.

When testing wheels I ran into cases where one wheel depends on another, if the dependencies are added here, loading those wheels worked. This API could be extended to support dependencies indirect.
Sybren A. Stüvel commented 2024-03-18 11:40:07 +01:00
Author
Member
Copy Link

We could also lift the context manager to the top level of the intended use, so that you get something like:

with wheels.loader() as load:
    load("some_dependency_1")
    load("some_dependency_2")
    _bat_modules = load("blender_asset_tracer", ("blendfile", "bpathlib"))
    toplevel, blendfile, bpathlib = _bat_modules

That way everything within the context can 'see' each other in sys.modules, which should help in resolving such dependencies.

Of course, how this is shaped exactly heavily depends on other factors, like the other changes you describe in your review.

We could also lift the context manager to the top level of the intended use, so that you get something like: ```python with wheels.loader() as load: load("some_dependency_1") load("some_dependency_2") _bat_modules = load("blender_asset_tracer", ("blendfile", "bpathlib")) toplevel, blendfile, bpathlib = _bat_modules ``` That way everything within the context can 'see' each other in `sys.modules`, which should help in resolving such dependencies. Of course, how this is shaped exactly heavily depends on other factors, like the other changes you describe in your review.
docs/handbook/addons/dependencies.md
@ -0,0 +192,4 @@
# held by other code will stay valid.
sys.path[:] = old_syspath
sys.modules.clear()
sys.modules.update(old_sysmod)
Campbell Barton commented 2024-03-15 08:41:17 +01:00
Owner
Copy Link

While in most cases this shouldn't cause issues there is a potential problem caused from having multiple versions of the same module.

When a wheel loads a built-in Python module that isn't part of the wheel which will be cleared from sys.modules but accessible from the wheels module.

The same module may be loaded again later on, any singletons will then exist twice in memory, so if the module defines an atexit function, that would run multiple times when exiting for e.g.

Instead of fully restoring modules, only modules which are part of this wheel file could be removed.
Checking if the __file__ starts with wheel_file seems to work.

While in most cases this shouldn't cause issues there is a potential problem caused from having multiple versions of the same module. When a wheel loads a built-in Python module that isn't part of the wheel which will be cleared from `sys.modules` but accessible from the wheels module. The same module may be loaded again later on, any singletons will then exist twice in memory, so if the module defines an `atexit` function, that would run multiple times when exiting for e.g. Instead of fully restoring modules, only modules which are part of this wheel file could be removed. Checking if the `__file__` starts with `wheel_file` seems to work.
👍 1
docs/handbook/addons/dependencies.md
@ -0,0 +196,4 @@
def _wheel_filename(fname_prefix: str) -> Path:
path_pattern = "%s*.whl" % fname_prefix
Campbell Barton commented 2024-03-15 07:32:46 +01:00
Owner
Copy Link

The naming convention uses dash separators: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#binary-distribution-format

So it looks like "%s*.whl" can be "%s-*.whl" to avoid an error if one wheel is the prefix of another.

The naming convention uses dash separators: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#binary-distribution-format So it looks like `"%s*.whl"` can be `"%s-*.whl"` to avoid an error if one wheel is the prefix of another.
👍 1
docs/handbook/addons/dependencies.md
@ -0,0 +201,4 @@
if not wheels:
raise RuntimeError("Unable to find wheel at %r" % path_pattern)
# If there are multiple wheels that match, load the last-modified one.
Campbell Barton commented 2024-03-15 07:35:20 +01:00
Owner
Copy Link

In the context of distributing extensions I don't see why there would ever be more than one wheel *. That could even be made into an error AFAICS.

If there is some reason to support this, the version number from each could be extracted and compared using setuptools.wheel.parse_version.

* except in the case of different architectures although in that case we would still want only one wheel per architecture.

In the context of distributing extensions I don't see why there would ever be more than one wheel \*. That could even be made into an error AFAICS. If there is some reason to support this, the version number from each could be extracted and compared using `setuptools.wheel.parse_version`. ---- \* except in the case of different architectures although in that case we would still want only one wheel per architecture.
Sybren A. Stüvel commented 2024-03-18 11:33:57 +01:00
Author
Member
Copy Link

With regular/traditional add-ons, this was necessary to allow upgrading an add-on by simply installing the newer version. Since that didn't first delete the old add-on, you could end up with wheel files from both the old and the new version.

A better approach would be to include the exact version number in the loading function, so that the correct wheel file can be loaded explicitly. That way it's also possible to downgrade a dependency, for example. That's also the reason why I used timestamps: the last-installed file is the most important, and not necessarily the one with the highest version number.

With regular/traditional add-ons, this was necessary to allow upgrading an add-on by simply installing the newer version. Since that didn't first delete the old add-on, you could end up with wheel files from both the old and the new version. A better approach would be to include the exact version number in the loading function, so that the correct wheel file can be loaded explicitly. That way it's also possible to downgrade a dependency, for example. That's also the reason why I used timestamps: the last-installed file is the most important, and not necessarily the one with the highest version number.
Campbell Barton commented 2024-03-15 11:40:28 +01:00
Owner
Copy Link

Firstly, having looked into this I think it may be worth prioritizing wheel support that doesn't depend on temporary sys.path hacks - since developers will loose time whenever they run into a case which doesn't work, and generally make the experience of trying to get their add-on ported to an extension worse.

When developers run into a problem the answer from Blender developers is likely something along the lines of:

"We don't know why this is broken and we're not going to investigate why individual wheels don't work with this method."

While some additional cases can be supported, wheel's that depend on other wheels for e.g. We could even extract the wheel to disk so files on the filesystem can be read, or support binary wheels by loading wheels with the appropriate architecture.

Even if we do all this there will still be enough cases that fail, time is probably better spent on having a way for extensions to install wheels to a shared location in Blender's Python installation.

Firstly, having looked into this I think it may be worth prioritizing wheel support that doesn't depend on temporary `sys.path` hacks - since developers will loose time whenever they run into a case which doesn't work, and generally make the experience of trying to get their add-on ported to an extension worse. When developers run into a problem the answer from Blender developers is likely something along the lines of: _"We don't know why this is broken and we're not going to investigate why individual wheels don't work with this method."_ While some additional cases _can_ be supported, wheel's that depend on other wheels for e.g. We could even extract the wheel to disk so files on the filesystem can be read, or support binary wheels by loading wheels with the appropriate architecture. Even if we do all this there will still be enough cases that fail, time is probably better spent on having a way for extensions to install wheels to a shared location in Blender's Python installation.
Sybren A. Stüvel commented 2024-03-18 11:43:48 +01:00
Author
Member
Copy Link

Thanks for reviewing this so thoroughly.

Firstly, having looked into this I think it may be worth prioritizing wheel support that doesn't depend on temporary sys.path hacks - since developers will loose time whenever they run into a case which doesn't work, and generally make the experience of trying to get their add-on ported to an extension worse.

I agree.

While some additional cases can be supported, wheel's that depend on other wheels for e.g. We could even extract the wheel to disk so files on the filesystem can be read, or support binary wheels by loading wheels with the appropriate architecture.

That's an interesting approach.

Even if we do all this there will still be enough cases that fail, time is probably better spent on having a way for extensions to install wheels to a shared location in Blender's Python installation.

That sounds pretty good to me. I'm not sure if I have the time to be the one to build this, though. Do you think this could be a target for the Extensions project?

Thanks for reviewing this so thoroughly. > Firstly, having looked into this I think it may be worth prioritizing wheel support that doesn't depend on temporary `sys.path` hacks - since developers will loose time whenever they run into a case which doesn't work, and generally make the experience of trying to get their add-on ported to an extension worse. I agree. > While some additional cases _can_ be supported, wheel's that depend on other wheels for e.g. We could even extract the wheel to disk so files on the filesystem can be read, or support binary wheels by loading wheels with the appropriate architecture. That's an interesting approach. > Even if we do all this there will still be enough cases that fail, time is probably better spent on having a way for extensions to install wheels to a shared location in Blender's Python installation. That sounds pretty good to me. I'm not sure if I have the time to be the one to build this, though. Do you think this could be a target for the Extensions project?
Merge conflict checking is in progress. Try again in few moments.
View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u pr/addon-dependencies:dr.sybren-pr/addon-dependencies
git checkout dr.sybren-pr/addon-dependencies

Merge

Merge the changes and update on Gitea.
git checkout main
git merge --no-ff dr.sybren-pr/addon-dependencies
git checkout main
git merge --ff-only dr.sybren-pr/addon-dependencies
git checkout dr.sybren-pr/addon-dependencies
git rebase main
git checkout main
git merge --no-ff dr.sybren-pr/addon-dependencies
git checkout main
git merge --squash dr.sybren-pr/addon-dependencies
git checkout main
git merge dr.sybren-pr/addon-dependencies
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Campbell Barton
Dalai Felinto
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender-developer-docs#40
No description provided.
Delete Branch "dr.sybren/blender-developer-docs:pr/addon-dependencies"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?