Platform specific (multi-OS) extensions and wheels #74

New Issue

Dalai Felinto · 2024-04-09T12:22:27+02:00

Dalai Felinto commented

2024-04-09 12:22:27 +02:00

Multi-OS (add-ons) extensions

Guiding principles:

A single extension package (.zip) may be cross-platform (with or without wheels).
Users should be able to specify which platforms an extension package supports.

They are two main reasons an extension may be OS specific:

It requires an integration with software which is only available on Windows for example.
It requires wheels which are platform specific.

In both cases this is indicated by the new plaforms manifest field.

Manifest:

platforms: windows-amd64, windows-arm64, macos-x86_64, macos-arm64, linux-x86_64
wheels: list of bundled wheel files.

# Optional list of supported platforms. If ommitted, the extension will be available in all operating systems.
# platforms = ["windows-amd64", "macos-arm64", "linux-x86_64"]
# Other supported platforms: "windows-arm64", "macos-x86_64"

# Optional list of Python wheels bundled with the extension. The paths are relative to the manifest.
# wheels = [ 
#     "wheels/my-wheel-2.3.4-windows-arm64.whl",
#     "wheels/my-wheel-2.3.4-macos-x86_64.whl",
#     "wheels/my-wheel-2.3.4-linux-x86_64.whl",
# ]

Extensions package

To reduce the bandwith required to download extensions, the server can generate multiple platform-specific package from one uploaded package. The uploaded package is always a single .zip package containing the dependencies of all supported platforms. The server will then split the uploaded package into platform specific packages, with only the dependencies relevant to that platform.

Uploaded package:

my_extension-0.0.1.zip
└─ my_extension-0.0.1
    ├─ __init__.py
    ├─ blender_manifest.toml
    │   └─ platforms = ["windows-amd64", "macos-x86_64"]
    └─ ./wheels
         ├─ my-wheel-2.3.4-windows-amd64.whl
         ├─ my-wheel-2.3.4-windows-arm64.whl
         ├─ my-wheel-2.3.4-macos-x86_64.whl
         └─ my-wheel-2.3.4-linux-x86_64.whl

Server-generated windows-amd64 package:

my_extension-0.0.1-windows-amd64.zip
└─ my_extension-0.0.1
    ├─ __init__.py
    ├─ blender_manifest.toml
    │   └─ platforms = ["windows-amd64", "macos-x86_64"]
    └─ ./wheels
         └─ my-wheel-2.3.4-windows-amd64.whl

Server-generated macos-x86_64 package:

my_extension-0.0.1-macos-x86_64.zip
└─ my_extension-0.0.1
    ├─ __init__.py
    ├─ blender_manifest.toml
    │   └─ platforms = ["windows-amd64", "macos-x86_64"]
    └─ ./wheels
         └─ my-wheel-2.3.4-macos-x86_64.whl

Tasks:

Update the documentation for the manifest 1.0.0 schema [1] [2].
Blender: pass platform argument to query the API.
Backend: JSON include the new extension fields (platforms and wheels).
Backend: Implement multi-OS packages.
Backend: Handle OS-specific wheel package generation (can be done after everything else).
Frontend: Serve the correct package based on user-agent - list all available OSs for direct download.

# Multi-OS (add-ons) extensions Guiding principles: * A single extension package (.zip) may be cross-platform (with or without wheels). * Users should be able to specify which platforms an extension package supports. They are two main reasons an extension may be OS specific: * It requires an integration with software which is only available on Windows for example. * It requires [wheels](https://pythonwheels.com/) which are platform specific. In both cases this is indicated by the new `plaforms` manifest field. ### Manifest: * **platforms**: `windows-amd64`, `windows-arm64`, `macos-x86_64`, `macos-arm64`, `linux-x86_64` * **wheels**: list of bundled wheel files. ```toml # Optional list of supported platforms. If ommitted, the extension will be available in all operating systems. # platforms = ["windows-amd64", "macos-arm64", "linux-x86_64"] # Other supported platforms: "windows-arm64", "macos-x86_64" # Optional list of Python wheels bundled with the extension. The paths are relative to the manifest. # wheels = [ # "wheels/my-wheel-2.3.4-windows-arm64.whl", # "wheels/my-wheel-2.3.4-macos-x86_64.whl", # "wheels/my-wheel-2.3.4-linux-x86_64.whl", # ] ``` ## Extensions package To reduce the bandwith required to download extensions, the server can generate multiple platform-specific package from one uploaded package. The uploaded package is always a single .zip package containing the dependencies of all supported platforms. The server will then split the uploaded package into platform specific packages, with only the dependencies relevant to that platform. Uploaded package: ``` my_extension-0.0.1.zip └─ my_extension-0.0.1 ├─ __init__.py ├─ blender_manifest.toml │ └─ platforms = ["windows-amd64", "macos-x86_64"] └─ ./wheels ├─ my-wheel-2.3.4-windows-amd64.whl ├─ my-wheel-2.3.4-windows-arm64.whl ├─ my-wheel-2.3.4-macos-x86_64.whl └─ my-wheel-2.3.4-linux-x86_64.whl ``` Server-generated windows-amd64 package: ``` my_extension-0.0.1-windows-amd64.zip └─ my_extension-0.0.1 ├─ __init__.py ├─ blender_manifest.toml │ └─ platforms = ["windows-amd64", "macos-x86_64"] └─ ./wheels └─ my-wheel-2.3.4-windows-amd64.whl ``` Server-generated macos-x86_64 package: ``` my_extension-0.0.1-macos-x86_64.zip └─ my_extension-0.0.1 ├─ __init__.py ├─ blender_manifest.toml │ └─ platforms = ["windows-amd64", "macos-x86_64"] └─ ./wheels └─ my-wheel-2.3.4-macos-x86_64.whl ``` ## Tasks: * [ ] Update the documentation for the manifest 1.0.0 schema [[1](https://developer.blender.org/docs/features/extensions/schema/1.0.0/)] [[2](https://docs.blender.org/manual/en/dev/extensions/getting_started.html)]. * [ ] Blender: pass platform argument to query the API. * [ ] Backend: JSON include the new extension fields (platforms and wheels). * [ ] Backend: Implement multi-OS packages. * [ ] Backend: Handle OS-specific wheel package generation (can be done after everything else). * [ ] Frontend: Serve the correct package based on user-agent - list all available OSs for direct download.

Dalai Felinto referenced this issue

2024-04-09 12:25:57 +02:00

Extensions Website - Backend Road to Beta #70

Oleg-Komarov self-assigned this 2024-05-14 10:19:47 +02:00

Oleg-Komarov commented

2024-05-14 11:23:05 +02:00

I think it is beneficial to adopt the notation defined in https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#platform-tag (takes over from https://peps.python.org/pep-0425/#platform-tag and uses the same rules): using underscores in the platform tag.

Unfortunately the docs don't provide a comprehensive list of platform tags and the spec document is not comprehensive enough, but we can also look at the popular wheel distributions compliant with the spec, e.g. https://pypi.org/project/Cython/#files and https://pypi.org/project/numpy/#files

We can also consult with packaging tests: https://github.com/pypa/packaging/blob/main/tests/test_tags.py

The windows tags then become: win_amd64, win_arm64 (couldn't easily find those in the wild and don't see blender releases available for windows arm)
Linux: the spec differentiates between manylinux and musllinux, making linux_x86_64 insufficient for a general case, but later in the same doc uses linux_x86_64 in examples. Not sure if we need to differentiate here, this also depends on how blender will be selecting wheels during installation. See also https://peps.python.org/pep-0600/
MacOS: is also not uniform, you can find macosx_11_0_arm64 and macosx_10_9_x86_64 (and other ones, see the test)

I think it is beneficial to adopt the notation defined in https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#platform-tag (takes over from https://peps.python.org/pep-0425/#platform-tag and uses the same rules): using underscores in the platform tag. Unfortunately the docs don't provide a comprehensive list of platform tags and the spec document is not comprehensive enough, but we can also look at the popular wheel distributions compliant with the spec, e.g. https://pypi.org/project/Cython/#files and https://pypi.org/project/numpy/#files We can also consult with packaging tests: https://github.com/pypa/packaging/blob/main/tests/test_tags.py The windows tags then become: `win_amd64`, `win_arm64` (couldn't easily find those in the wild and don't see blender releases available for windows arm) Linux: the spec differentiates between manylinux and musllinux, making `linux_x86_64` insufficient for a general case, but later in the same doc uses `linux_x86_64` in examples. Not sure if we need to differentiate here, this also depends on how blender will be selecting wheels during installation. See also https://peps.python.org/pep-0600/ MacOS: is also not uniform, you can find `macosx_11_0_arm64` and `macosx_10_9_x86_64` (and other ones, see the test)

Oleg-Komarov commented

2024-05-14 11:32:01 +02:00

We have a choice to ignore the packaging spec for the platforms field and use whatever blender can produce as its platform tag, but I think that having consistency across our platform tags and wheel names is important - it leaves less room for surprises, when a wrong wheel may be picked during installation; and I would expect that wheels will be built by standard tooling, leading to the names complying with the spec.

We have a choice to ignore the packaging spec for the `platforms` field and use whatever blender can produce as its platform tag, but I think that having consistency across our platform tags and wheel names is important - it leaves less room for surprises, when a wrong wheel may be picked during installation; and I would expect that wheels will be built by standard tooling, leading to the names complying with the spec.

Dalai Felinto commented

2024-05-14 17:58:05 +02:00

I updated the schema on the documentation already to include the platforms field:

I updated the schema on the documentation already to include the `platforms` field: * [User Manual](https://docs.blender.org/manual/en/dev/extensions/getting_started.html#manifest) * [Technical Documentation](https://developer.blender.org/docs/features/extensions/schema/1.0.0/)

Sign in to join this conversation.