Platform specific (multi-OS) extensions and wheels #74

Open
opened 2024-04-09 12:22:27 +02:00 by Dalai Felinto · 5 comments

Multi-OS (add-ons) extensions

Guiding principles:

  • A single extension package (.zip) may be cross-platform (with or without wheels).
  • Users should be able to specify which platforms an extension package supports.

They are two main reasons an extension may be OS specific:

  • It requires an integration with software which is only available on Windows for example.
  • It requires wheels which are platform specific.

In both cases this is indicated by the new plaforms manifest field.

Manifest:

  • platforms: windows-amd64, windows-arm64, macos-x86_64, macos-arm64, linux-x86_64
  • wheels: list of bundled wheel files.
# Optional list of supported platforms. If ommitted, the extension will be available in all operating systems.
# platforms = ["windows-amd64", "macos-arm64", "linux-x86_64"]
# Other supported platforms: "windows-arm64", "macos-x86_64"

# Optional list of Python wheels bundled with the extension. The paths are relative to the manifest.
# wheels = [ 
#     "wheels/my-wheel-2.3.4-windows-arm64.whl",
#     "wheels/my-wheel-2.3.4-macos-x86_64.whl",
#     "wheels/my-wheel-2.3.4-linux-x86_64.whl",
# ]

Extensions package

To reduce the bandwith required to download extensions, the server can generate multiple platform-specific package from one uploaded package. The uploaded package is always a single .zip package containing the dependencies of all supported platforms. The server will then split the uploaded package into platform specific packages, with only the dependencies relevant to that platform.

Uploaded package:

my_extension-0.0.1.zip
└─ my_extension-0.0.1
    ├─ __init__.py
    ├─ blender_manifest.toml
    │   └─ platforms = ["windows-amd64", "macos-x86_64"]
    └─ ./wheels
         ├─ my-wheel-2.3.4-windows-amd64.whl
         ├─ my-wheel-2.3.4-windows-arm64.whl
         ├─ my-wheel-2.3.4-macos-x86_64.whl
         └─ my-wheel-2.3.4-linux-x86_64.whl

Server-generated windows-amd64 package:

my_extension-0.0.1-windows-amd64.zip
└─ my_extension-0.0.1
    ├─ __init__.py
    ├─ blender_manifest.toml
    │   └─ platforms = ["windows-amd64", "macos-x86_64"]
    └─ ./wheels
         └─ my-wheel-2.3.4-windows-amd64.whl

Server-generated macos-x86_64 package:

my_extension-0.0.1-macos-x86_64.zip
└─ my_extension-0.0.1
    ├─ __init__.py
    ├─ blender_manifest.toml
    │   └─ platforms = ["windows-amd64", "macos-x86_64"]
    └─ ./wheels
         └─ my-wheel-2.3.4-macos-x86_64.whl

Tasks:

  • Update the documentation for the manifest 1.0.0 schema [1] [2].
  • Blender: pass platform argument to query the API.
  • Backend: JSON include the new extension fields (platforms and wheels).
  • Backend: Implement multi-OS packages.
  • Backend: Handle OS-specific wheel package generation (can be done after everything else).
  • Frontend: Serve the correct package based on user-agent - list all available OSs for direct download.
# Multi-OS (add-ons) extensions Guiding principles: * A single extension package (.zip) may be cross-platform (with or without wheels). * Users should be able to specify which platforms an extension package supports. They are two main reasons an extension may be OS specific: * It requires an integration with software which is only available on Windows for example. * It requires [wheels](https://pythonwheels.com/) which are platform specific. In both cases this is indicated by the new `plaforms` manifest field. ### Manifest: * **platforms**: `windows-amd64`, `windows-arm64`, `macos-x86_64`, `macos-arm64`, `linux-x86_64` * **wheels**: list of bundled wheel files. ```toml # Optional list of supported platforms. If ommitted, the extension will be available in all operating systems. # platforms = ["windows-amd64", "macos-arm64", "linux-x86_64"] # Other supported platforms: "windows-arm64", "macos-x86_64" # Optional list of Python wheels bundled with the extension. The paths are relative to the manifest. # wheels = [ # "wheels/my-wheel-2.3.4-windows-arm64.whl", # "wheels/my-wheel-2.3.4-macos-x86_64.whl", # "wheels/my-wheel-2.3.4-linux-x86_64.whl", # ] ``` ## Extensions package To reduce the bandwith required to download extensions, the server can generate multiple platform-specific package from one uploaded package. The uploaded package is always a single .zip package containing the dependencies of all supported platforms. The server will then split the uploaded package into platform specific packages, with only the dependencies relevant to that platform. Uploaded package: ``` my_extension-0.0.1.zip └─ my_extension-0.0.1 ├─ __init__.py ├─ blender_manifest.toml │ └─ platforms = ["windows-amd64", "macos-x86_64"] └─ ./wheels ├─ my-wheel-2.3.4-windows-amd64.whl ├─ my-wheel-2.3.4-windows-arm64.whl ├─ my-wheel-2.3.4-macos-x86_64.whl └─ my-wheel-2.3.4-linux-x86_64.whl ``` Server-generated windows-amd64 package: ``` my_extension-0.0.1-windows-amd64.zip └─ my_extension-0.0.1 ├─ __init__.py ├─ blender_manifest.toml │ └─ platforms = ["windows-amd64", "macos-x86_64"] └─ ./wheels └─ my-wheel-2.3.4-windows-amd64.whl ``` Server-generated macos-x86_64 package: ``` my_extension-0.0.1-macos-x86_64.zip └─ my_extension-0.0.1 ├─ __init__.py ├─ blender_manifest.toml │ └─ platforms = ["windows-amd64", "macos-x86_64"] └─ ./wheels └─ my-wheel-2.3.4-macos-x86_64.whl ``` ## Tasks: * [ ] Update the documentation for the manifest 1.0.0 schema [[1](https://developer.blender.org/docs/features/extensions/schema/1.0.0/)] [[2](https://docs.blender.org/manual/en/dev/extensions/getting_started.html)]. * [ ] Blender: pass platform argument to query the API. * [x] Backend: JSON include the new extension fields (platforms and wheels). * [x] Backend: Implement multi-OS packages. * [ ] Backend: Handle OS-specific wheel package generation (can be done after everything else). * [ ] Frontend: Serve the correct package based on user-agent - list all available OSs for direct download.
Oleg-Komarov self-assigned this 2024-05-14 10:19:47 +02:00
Owner

I think it is beneficial to adopt the notation defined in https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#platform-tag (takes over from https://peps.python.org/pep-0425/#platform-tag and uses the same rules): using underscores in the platform tag.

Unfortunately the docs don't provide a comprehensive list of platform tags and the spec document is not comprehensive enough, but we can also look at the popular wheel distributions compliant with the spec, e.g. https://pypi.org/project/Cython/#files and https://pypi.org/project/numpy/#files

We can also consult with packaging tests: https://github.com/pypa/packaging/blob/main/tests/test_tags.py

The windows tags then become: win_amd64, win_arm64 (couldn't easily find those in the wild and don't see blender releases available for windows arm)
Linux: the spec differentiates between manylinux and musllinux, making linux_x86_64 insufficient for a general case, but later in the same doc uses linux_x86_64 in examples. Not sure if we need to differentiate here, this also depends on how blender will be selecting wheels during installation. See also https://peps.python.org/pep-0600/
MacOS: is also not uniform, you can find macosx_11_0_arm64 and macosx_10_9_x86_64 (and other ones, see the test)

I think it is beneficial to adopt the notation defined in https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#platform-tag (takes over from https://peps.python.org/pep-0425/#platform-tag and uses the same rules): using underscores in the platform tag. Unfortunately the docs don't provide a comprehensive list of platform tags and the spec document is not comprehensive enough, but we can also look at the popular wheel distributions compliant with the spec, e.g. https://pypi.org/project/Cython/#files and https://pypi.org/project/numpy/#files We can also consult with packaging tests: https://github.com/pypa/packaging/blob/main/tests/test_tags.py The windows tags then become: `win_amd64`, `win_arm64` (couldn't easily find those in the wild and don't see blender releases available for windows arm) Linux: the spec differentiates between manylinux and musllinux, making `linux_x86_64` insufficient for a general case, but later in the same doc uses `linux_x86_64` in examples. Not sure if we need to differentiate here, this also depends on how blender will be selecting wheels during installation. See also https://peps.python.org/pep-0600/ MacOS: is also not uniform, you can find `macosx_11_0_arm64` and `macosx_10_9_x86_64` (and other ones, see the test)
Owner

We have a choice to ignore the packaging spec for the platforms field and use whatever blender can produce as its platform tag, but I think that having consistency across our platform tags and wheel names is important - it leaves less room for surprises, when a wrong wheel may be picked during installation; and I would expect that wheels will be built by standard tooling, leading to the names complying with the spec.

We have a choice to ignore the packaging spec for the `platforms` field and use whatever blender can produce as its platform tag, but I think that having consistency across our platform tags and wheel names is important - it leaves less room for surprises, when a wrong wheel may be picked during installation; and I would expect that wheels will be built by standard tooling, leading to the names complying with the spec.
Author
Owner

I updated the schema on the documentation already to include the platforms field:

I updated the schema on the documentation already to include the `platforms` field: * [User Manual](https://docs.blender.org/manual/en/dev/extensions/getting_started.html#manifest) * [Technical Documentation](https://developer.blender.org/docs/features/extensions/schema/1.0.0/)
Owner

Summary of a chat with @ideasman42 and @dfelinto (please correct if I missed anything)

  • blender currently doesn't do a granular platform detection, so fully mirroring the python platform tags in platforms field won't be practical
  • authors can [and should?] package multiple wheels for the same OS when it is necessary, and blender can try to pick up the correct one, but at the moment this behavior is not specified yet
  • in general, we expect that there will be convergence around vfx reference platform (https://vfxplatform.com/) that standardizes the requirements for platforms, e.g. glibc version in linux, mac osx version

These points support the decision to use a simplified tag, such as linux-x86_64, macos-arm64, etc (see the docs)

Also it's important to note that the platforms field may be used to mark platform-specific code in the packaged code that doesn't ship any wheels at all, this is a reason to not rely on just the wheels field in the manifest.

Summary of a chat with @ideasman42 and @dfelinto (please correct if I missed anything) - blender currently doesn't do a granular platform detection, so fully mirroring the python platform tags in `platforms` field won't be practical - authors can [and should?] package multiple wheels for the same OS when it is necessary, and blender can try to pick up the correct one, but at the moment this behavior is not specified yet - in general, we expect that there will be convergence around vfx reference platform (https://vfxplatform.com/) that standardizes the requirements for platforms, e.g. glibc version in linux, mac osx version These points support the decision to use a simplified tag, such as `linux-x86_64`, `macos-arm64`, etc (see the docs) Also it's important to note that the `platforms` field may be used to mark platform-specific code in the packaged code that doesn't ship any wheels at all, this is a reason to not rely on just the `wheels` field in the manifest.
Owner

Deployed #131 in production.

Repackaging is not implemented for now.

Deployed #131 in production. Repackaging is not implemented for now.
Oleg-Komarov removed their assignment 2024-05-16 18:30:11 +02:00
Sign in to join this conversation.
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: infrastructure/extensions-website#74
No description provided.