infrastructure/extensions-website
4
7
Fork 7
Code Issues 15 Pull Requests 2 Projects Releases Wiki Activity

Platform specific (multi-OS) extensions and wheels #74

New Issue
Closed
opened 2024-04-09 12:22:27 +02:00 by Dalai Felinto · 9 comments
Dalai Felinto commented 2024-04-09 12:22:27 +02:00
Owner
Copy Link

Multi-OS (add-ons) extensions

Guiding principles:

  • A single extension package (.zip) may be cross-platform (with or without wheels).
  • Users should be able to specify which platforms an extension package supports.

They are two main reasons an extension may be OS specific:

  • It requires an integration with software which is only available on Windows for example.
  • It requires wheels which are platform specific.

In both cases this is indicated by the new plaforms manifest field.

Manifest:

  • platforms: windows-amd64, windows-arm64, macos-x86_64, macos-arm64, linux-x86_64
  • wheels: list of bundled wheel files.
# Optional list of supported platforms. If ommitted, the extension will be available in all operating systems.
# platforms = ["windows-amd64", "macos-arm64", "linux-x86_64"]
# Other supported platforms: "windows-arm64", "macos-x86_64"

# Optional list of Python wheels bundled with the extension. The paths are relative to the manifest.
# wheels = [ 
#     "wheels/my-wheel-2.3.4-windows-arm64.whl",
#     "wheels/my-wheel-2.3.4-macos-x86_64.whl",
#     "wheels/my-wheel-2.3.4-linux-x86_64.whl",
# ]

Extensions package

To reduce the bandwith required to download extensions, the server can generate multiple platform-specific package from one uploaded package. The uploaded package is always a single .zip package containing the dependencies of all supported platforms. The server will then split the uploaded package into platform specific packages, with only the dependencies relevant to that platform.

Uploaded package:

my_extension-0.0.1.zip
└─ my_extension-0.0.1
    ├─ __init__.py
    ├─ blender_manifest.toml
    │   └─ platforms = ["windows-amd64", "macos-x86_64"]
    └─ ./wheels
         ├─ my-wheel-2.3.4-windows-amd64.whl
         ├─ my-wheel-2.3.4-windows-arm64.whl
         ├─ my-wheel-2.3.4-macos-x86_64.whl
         └─ my-wheel-2.3.4-linux-x86_64.whl

Server-generated windows-amd64 package:

my_extension-0.0.1-windows-amd64.zip
└─ my_extension-0.0.1
    ├─ __init__.py
    ├─ blender_manifest.toml
    │   └─ platforms = ["windows-amd64", "macos-x86_64"]
    └─ ./wheels
         └─ my-wheel-2.3.4-windows-amd64.whl

Server-generated macos-x86_64 package:

my_extension-0.0.1-macos-x86_64.zip
└─ my_extension-0.0.1
    ├─ __init__.py
    ├─ blender_manifest.toml
    │   └─ platforms = ["windows-amd64", "macos-x86_64"]
    └─ ./wheels
         └─ my-wheel-2.3.4-macos-x86_64.whl

Tasks:

  • Update the documentation for the manifest 1.0.0 schema [1] [2].
  • Blender: pass platform argument to query the API.
  • Backend: JSON include the new extension fields (platforms and wheels).
  • Backend: Implement multi-OS packages.
  • Backend: Handle OS-specific wheel package generation (can be done after everything else). Support multiple uploads per version
  • Frontend: Serve the correct package based on user-agent - list all available OSs for direct download.
# Multi-OS (add-ons) extensions Guiding principles: * A single extension package (.zip) may be cross-platform (with or without wheels). * Users should be able to specify which platforms an extension package supports. They are two main reasons an extension may be OS specific: * It requires an integration with software which is only available on Windows for example. * It requires [wheels](https://pythonwheels.com/) which are platform specific. In both cases this is indicated by the new `plaforms` manifest field. ### Manifest: * **platforms**: `windows-amd64`, `windows-arm64`, `macos-x86_64`, `macos-arm64`, `linux-x86_64` * **wheels**: list of bundled wheel files. ```toml # Optional list of supported platforms. If ommitted, the extension will be available in all operating systems. # platforms = ["windows-amd64", "macos-arm64", "linux-x86_64"] # Other supported platforms: "windows-arm64", "macos-x86_64" # Optional list of Python wheels bundled with the extension. The paths are relative to the manifest. # wheels = [ # "wheels/my-wheel-2.3.4-windows-arm64.whl", # "wheels/my-wheel-2.3.4-macos-x86_64.whl", # "wheels/my-wheel-2.3.4-linux-x86_64.whl", # ] ``` ## Extensions package To reduce the bandwith required to download extensions, the server can generate multiple platform-specific package from one uploaded package. The uploaded package is always a single .zip package containing the dependencies of all supported platforms. The server will then split the uploaded package into platform specific packages, with only the dependencies relevant to that platform. Uploaded package: ``` my_extension-0.0.1.zip └─ my_extension-0.0.1 ├─ __init__.py ├─ blender_manifest.toml │ └─ platforms = ["windows-amd64", "macos-x86_64"] └─ ./wheels ├─ my-wheel-2.3.4-windows-amd64.whl ├─ my-wheel-2.3.4-windows-arm64.whl ├─ my-wheel-2.3.4-macos-x86_64.whl └─ my-wheel-2.3.4-linux-x86_64.whl ``` Server-generated windows-amd64 package: ``` my_extension-0.0.1-windows-amd64.zip └─ my_extension-0.0.1 ├─ __init__.py ├─ blender_manifest.toml │ └─ platforms = ["windows-amd64", "macos-x86_64"] └─ ./wheels └─ my-wheel-2.3.4-windows-amd64.whl ``` Server-generated macos-x86_64 package: ``` my_extension-0.0.1-macos-x86_64.zip └─ my_extension-0.0.1 ├─ __init__.py ├─ blender_manifest.toml │ └─ platforms = ["windows-amd64", "macos-x86_64"] └─ ./wheels └─ my-wheel-2.3.4-macos-x86_64.whl ``` ## Tasks: * [x] Update the documentation for the manifest 1.0.0 schema [[1](https://developer.blender.org/docs/features/extensions/schema/1.0.0/)] [[2](https://docs.blender.org/manual/en/dev/extensions/getting_started.html)]. * [x] Blender: pass platform argument to query the API. * [x] Backend: JSON include the new extension fields (platforms and wheels). * [x] Backend: Implement multi-OS packages. * [ ] Backend: ~~Handle OS-specific wheel package generation (can be done after everything else).~~ Support multiple uploads per version * [ ] Frontend: Serve the correct package based on user-agent - list all available OSs for direct download.
Oleg-Komarov self-assigned this 2024-05-14 10:19:47 +02:00
Oleg-Komarov commented 2024-05-14 11:23:05 +02:00
Owner
Copy Link

I think it is beneficial to adopt the notation defined in https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#platform-tag (takes over from https://peps.python.org/pep-0425/#platform-tag and uses the same rules): using underscores in the platform tag.

Unfortunately the docs don't provide a comprehensive list of platform tags and the spec document is not comprehensive enough, but we can also look at the popular wheel distributions compliant with the spec, e.g. https://pypi.org/project/Cython/#files and https://pypi.org/project/numpy/#files

We can also consult with packaging tests: https://github.com/pypa/packaging/blob/main/tests/test_tags.py

The windows tags then become: win_amd64, win_arm64 (couldn't easily find those in the wild and don't see blender releases available for windows arm)
Linux: the spec differentiates between manylinux and musllinux, making linux_x86_64 insufficient for a general case, but later in the same doc uses linux_x86_64 in examples. Not sure if we need to differentiate here, this also depends on how blender will be selecting wheels during installation. See also https://peps.python.org/pep-0600/
MacOS: is also not uniform, you can find macosx_11_0_arm64 and macosx_10_9_x86_64 (and other ones, see the test)

I think it is beneficial to adopt the notation defined in https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#platform-tag (takes over from https://peps.python.org/pep-0425/#platform-tag and uses the same rules): using underscores in the platform tag. Unfortunately the docs don't provide a comprehensive list of platform tags and the spec document is not comprehensive enough, but we can also look at the popular wheel distributions compliant with the spec, e.g. https://pypi.org/project/Cython/#files and https://pypi.org/project/numpy/#files We can also consult with packaging tests: https://github.com/pypa/packaging/blob/main/tests/test_tags.py The windows tags then become: `win_amd64`, `win_arm64` (couldn't easily find those in the wild and don't see blender releases available for windows arm) Linux: the spec differentiates between manylinux and musllinux, making `linux_x86_64` insufficient for a general case, but later in the same doc uses `linux_x86_64` in examples. Not sure if we need to differentiate here, this also depends on how blender will be selecting wheels during installation. See also https://peps.python.org/pep-0600/ MacOS: is also not uniform, you can find `macosx_11_0_arm64` and `macosx_10_9_x86_64` (and other ones, see the test)
Oleg-Komarov commented 2024-05-14 11:32:01 +02:00
Owner
Copy Link

We have a choice to ignore the packaging spec for the platforms field and use whatever blender can produce as its platform tag, but I think that having consistency across our platform tags and wheel names is important - it leaves less room for surprises, when a wrong wheel may be picked during installation; and I would expect that wheels will be built by standard tooling, leading to the names complying with the spec.

We have a choice to ignore the packaging spec for the `platforms` field and use whatever blender can produce as its platform tag, but I think that having consistency across our platform tags and wheel names is important - it leaves less room for surprises, when a wrong wheel may be picked during installation; and I would expect that wheels will be built by standard tooling, leading to the names complying with the spec.
Dalai Felinto commented 2024-05-14 17:58:05 +02:00
Author
Owner
Copy Link

I updated the schema on the documentation already to include the platforms field:

I updated the schema on the documentation already to include the `platforms` field: * [User Manual](https://docs.blender.org/manual/en/dev/extensions/getting_started.html#manifest) * [Technical Documentation](https://developer.blender.org/docs/features/extensions/schema/1.0.0/)
Oleg-Komarov commented 2024-05-16 09:05:25 +02:00
Owner
Copy Link

Summary of a chat with @ideasman42 and @dfelinto (please correct if I missed anything)

  • blender currently doesn't do a granular platform detection, so fully mirroring the python platform tags in platforms field won't be practical
  • authors can [and should?] package multiple wheels for the same OS when it is necessary, and blender can try to pick up the correct one, but at the moment this behavior is not specified yet
  • in general, we expect that there will be convergence around vfx reference platform (https://vfxplatform.com/) that standardizes the requirements for platforms, e.g. glibc version in linux, mac osx version

These points support the decision to use a simplified tag, such as linux-x86_64, macos-arm64, etc (see the docs)

Also it's important to note that the platforms field may be used to mark platform-specific code in the packaged code that doesn't ship any wheels at all, this is a reason to not rely on just the wheels field in the manifest.

Summary of a chat with @ideasman42 and @dfelinto (please correct if I missed anything) - blender currently doesn't do a granular platform detection, so fully mirroring the python platform tags in `platforms` field won't be practical - authors can [and should?] package multiple wheels for the same OS when it is necessary, and blender can try to pick up the correct one, but at the moment this behavior is not specified yet - in general, we expect that there will be convergence around vfx reference platform (https://vfxplatform.com/) that standardizes the requirements for platforms, e.g. glibc version in linux, mac osx version These points support the decision to use a simplified tag, such as `linux-x86_64`, `macos-arm64`, etc (see the docs) Also it's important to note that the `platforms` field may be used to mark platform-specific code in the packaged code that doesn't ship any wheels at all, this is a reason to not rely on just the `wheels` field in the manifest.
Oleg-Komarov commented 2024-05-16 18:30:01 +02:00
Owner
Copy Link

Deployed #131 in production.

Repackaging is not implemented for now.

Deployed #131 in production. Repackaging is not implemented for now.
Oleg-Komarov removed their assignment 2024-05-16 18:30:11 +02:00
Oleg-Komarov commented 2024-06-17 09:21:22 +02:00
Owner
Copy Link

In addition to the initial support of multi-platform single-file uploads we want to start supporting multiple files per Version.
This allows to avoid unnecessary large file sizes for extensions that include their dependencies for all supported platforms.

Instead of implementing the logic for splitting uploaded files into multiple downloadable files ourselves, we will give control to maintainers, allowing them to prepare packages that will be shipped to end-users as is.

This approach has other benefits as well:

  • we don't have to temper with the downloadable files, all files on the platform come from their original authors, and we can potentially implement code signing in the future
  • this is easier to support - less brittle logic that might increase the attack surface of the platform

see also https://devtalk.blender.org/t/2024-06-14-extensions-platform/35136

In addition to the initial support of multi-platform single-file uploads we want to start supporting multiple files per Version. This allows to avoid unnecessary large file sizes for extensions that include their dependencies for all supported platforms. Instead of implementing the logic for splitting uploaded files into multiple downloadable files ourselves, we will give control to maintainers, allowing them to prepare packages that will be shipped to end-users as is. This approach has other benefits as well: - we don't have to temper with the downloadable files, all files on the platform come from their original authors, and we can potentially implement code signing in the future - this is easier to support - less brittle logic that might increase the attack surface of the platform see also https://devtalk.blender.org/t/2024-06-14-extensions-platform/35136
👍 1
Oleg-Komarov self-assigned this 2024-06-17 09:23:11 +02:00
Dalai Felinto commented 2024-06-18 09:36:16 +02:00
Author
Owner
Copy Link

@Oleg-Komarov for the records, Blender already has the -c extension build --split-platforms, which uses [build.generated]. So it is up to the server now to support [build.generated].

@Oleg-Komarov for the records, Blender already has the -c extension build --split-platforms, which uses `[build.generated]`. So it is up to the server now to support `[build.generated]`.
Oleg-Komarov commented 2024-06-21 13:11:57 +02:00
Owner
Copy Link

[build.generated] is now supported in production

`[build.generated]` is now supported in production
🎉 1
Dalai Felinto commented 2024-06-21 16:28:31 +02:00
Author
Owner
Copy Link

Closing this. The main parts are in already, and the remaining tasks will be implemented as part of: #194

Closing this. The main parts are in already, and the remaining tasks will be implemented as part of: https://projects.blender.org/infrastructure/extensions-website/issues/194
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Normal
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

  
Type
Breaking
Breaking change that won't be backward compatible

  
Type
Documentation
Documentation changes

  
Type
Enhancement
Improve existing functionality

  
Type
Feature
New functionality

  
Type
Report
Something is not working

  
Type
Security
This is security issue

  
Type
Suggestion
New ideas and proposals

  
Type
Testing
Issue or pull request related to testing

No Label
Priority
Critical
Priority
High
Priority
Low
Priority
Normal
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Type
Breaking
Type
Documentation
Type
Enhancement
Type
Feature
Type
Report
Type
Security
Type
Suggestion
Type
Testing
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
Oleg-Komarov
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: infrastructure/extensions-website#74
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?