WIP: PyAPI: Use py_capi_utils in bpy_prop_array and bpy_prop_collection foreach_get/set #114773

Draft
Thomas Barlow wants to merge 48 commits from Mysteryem/blender:py_buffer_all_use_capi_utils_pr into main

When changing the target branch, be careful to rebase the branch in your fork to match. See documentation.
Member

Part of #114429

Updates bpy_prop_array and bpy_prop_collection foreach_get/foreach_set
(py_prop_array_foreach_getset and foreach_getset in bpy_rna.cc) to use
the Py_Buffer format utilities in py_capi_utils.h, like the existing
buffer handling in idprop_py_api.cc#idp_from_PySequence.

Previously, foreach_get/set would check for specific first characters of
buffer format strings. This meant that only implicitly native byteorder
format strings could be used and meant that if the system running
Blender had multiple C types of the same kind of data (integer/floating-
point/bool) with the same size, only the types specifically checked for
would be supported.

This patch changes foreach_get/set to now only check that the buffer
format is the correct kind of data and that the buffer's itemsize
matches the property being foreach_get/set. Notably this patch now
allows:

  1. long format buffers to be used with bpy_prop_collection
    foreach_get/set and PROP_RAW_INT properties on systems where long
    and int are the same size. This was an issue when using NumPy arrays
    because NumPy's np.int32 dtype would default to long rather than
    int on such systems and bpy_prop_collection would only accept int
    buffers.
  2. Buffer formats with byteorder/size/alignment prefixes to be used.
    This was an issue with ctypes arrays because they use the < and >
    prefixes in their buffer formats.

To maintain foreach_get/set's existing behaviour of only allowing native
byteorder and specific signedness, a new PyC_StructFmt_is_native_order
function for checking if a format string is in native byteorder has been
added and PyC_StructFmt_type_is_int_any has been split into two
functions that check for either only signed integers or only unsigned
integers, with PyC_StructFmt_type_is_int_any's implementation being
replaced with calling both of the split functions.

Part of #114429 Updates bpy_prop_array and bpy_prop_collection foreach_get/foreach_set (py_prop_array_foreach_getset and foreach_getset in bpy_rna.cc) to use the Py_Buffer format utilities in py_capi_utils.h, like the existing buffer handling in idprop_py_api.cc#idp_from_PySequence. Previously, foreach_get/set would check for specific first characters of buffer format strings. This meant that only implicitly native byteorder format strings could be used and meant that if the system running Blender had multiple C types of the same kind of data (integer/floating- point/bool) with the same size, only the types specifically checked for would be supported. This patch changes foreach_get/set to now only check that the buffer format is the correct kind of data and that the buffer's itemsize matches the property being foreach_get/set. Notably this patch now allows: 1. `long` format buffers to be used with bpy_prop_collection foreach_get/set and `PROP_RAW_INT` properties on systems where `long` and `int` are the same size. This was an issue when using NumPy arrays because NumPy's `np.int32` dtype would default to `long` rather than `int` on such systems and bpy_prop_collection would only accept `int` buffers. 2. Buffer formats with byteorder/size/alignment prefixes to be used. This was an issue with `ctypes` arrays because they use the `<` and `>` prefixes in their buffer formats. To maintain foreach_get/set's existing behaviour of only allowing native byteorder and specific signedness, a new `PyC_StructFmt_is_native_order` function for checking if a format string is in native byteorder has been added and `PyC_StructFmt_type_is_int_any` has been split into two functions that check for either only signed integers or only unsigned integers, with `PyC_StructFmt_type_is_int_any`'s implementation being replaced with calling both of the split functions.
Thomas Barlow added 8 commits 2023-11-13 04:56:00 +01:00
Thomas Barlow added 40 commits 2023-12-16 18:49:37 +01:00
These could be considered safe casts if we were to support casting in
the future (as opposed to larger itemsizes which would be unsafe casts),
though since float16 support may be lacking, it may be best to only
consider the case of int16.
Systems where this is the case should be 32-bit Linux and Windows.

If we had tests for a double typed property, then we could also add a
similar test for where C double and C long double are both 64 bits.
This is much simpler.
bool tests are the exception, which expect to be a compatible buffer.
Move all sequence tests together
Ensure bool tests will pass if the bool property has raw access
Remove extra handling of ctypes.c_void_p arrays because the buffer always including an explicit byteorder/size/alignment prefix is annoying to deal with
Update sequence generation to use the default value of the property the sequences are being generated for
Update sequence generation to use the default value of the property the sequences are being generated for
This will allow for testing non-runtime properties where we have less control over the property values.
This is done by creating a Collection and linking Objects into that Collection so that foreach_get/foreach_set can be used on those Objects.
This pull request has changes conflicting with the target branch.
  • source/blender/python/intern/bpy_rna.cc
  • tests/python/bl_pyapi_prop_array.py

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u py_buffer_all_use_capi_utils_pr:Mysteryem-py_buffer_all_use_capi_utils_pr
git checkout Mysteryem-py_buffer_all_use_capi_utils_pr
Sign in to join this conversation.
No reviewers
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Asset Browser Project
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#114773
No description provided.