WIP: PyAPI: Use py_capi_utils in bpy_prop_array and bpy_prop_collection foreach_get/set #114773

Thomas Barlow · 2023-11-13T04:55:43+01:00

Thomas Barlow commented

2023-11-13 04:55:43 +01:00

Updates bpy_prop_array and bpy_prop_collection foreach_get/foreach_set
(py_prop_array_foreach_getset and foreach_getset in bpy_rna.cc) to use
the Py_Buffer format utilities in py_capi_utils.h, like the existing
buffer handling in idprop_py_api.cc#idp_from_PySequence.

Previously, foreach_get/set would check for specific first characters of
buffer format strings. This meant that only implicitly native byteorder
format strings could be used and meant that if the system running
Blender had multiple C types of the same kind of data (integer/floating-
point/bool) with the same size, only the types specifically checked for
would be supported.

This patch changes foreach_get/set to now only check that the buffer
format is the correct kind of data and that the buffer's itemsize
matches the property being foreach_get/set. Notably this patch now
allows:

long format buffers to be used with bpy_prop_collection
foreach_get/set and PROP_RAW_INT properties on systems where long
and int are the same size. This was an issue when using NumPy arrays
because NumPy's np.int32 dtype would default to long rather than
int on such systems and bpy_prop_collection would only accept int
buffers.
Buffer formats with byteorder/size/alignment prefixes to be used.
This was an issue with ctypes arrays because they use the < and >
prefixes in their buffer formats.

To maintain foreach_get/set's existing behaviour of only allowing native
byteorder and specific signedness, a new PyC_StructFmt_is_native_order
function for checking if a format string is in native byteorder has been
added and PyC_StructFmt_type_is_int_any has been split into two
functions that check for either only signed integers or only unsigned
integers, with PyC_StructFmt_type_is_int_any's implementation being
replaced with calling both of the split functions.

Part of #114429 Updates bpy_prop_array and bpy_prop_collection foreach_get/foreach_set (py_prop_array_foreach_getset and foreach_getset in bpy_rna.cc) to use the Py_Buffer format utilities in py_capi_utils.h, like the existing buffer handling in idprop_py_api.cc#idp_from_PySequence. Previously, foreach_get/set would check for specific first characters of buffer format strings. This meant that only implicitly native byteorder format strings could be used and meant that if the system running Blender had multiple C types of the same kind of data (integer/floating- point/bool) with the same size, only the types specifically checked for would be supported. This patch changes foreach_get/set to now only check that the buffer format is the correct kind of data and that the buffer's itemsize matches the property being foreach_get/set. Notably this patch now allows: 1. `long` format buffers to be used with bpy_prop_collection foreach_get/set and `PROP_RAW_INT` properties on systems where `long` and `int` are the same size. This was an issue when using NumPy arrays because NumPy's `np.int32` dtype would default to `long` rather than `int` on such systems and bpy_prop_collection would only accept `int` buffers. 2. Buffer formats with byteorder/size/alignment prefixes to be used. This was an issue with `ctypes` arrays because they use the `<` and `>` prefixes in their buffer formats. To maintain foreach_get/set's existing behaviour of only allowing native byteorder and specific signedness, a new `PyC_StructFmt_is_native_order` function for checking if a format string is in native byteorder has been added and `PyC_StructFmt_type_is_int_any` has been split into two functions that check for either only signed integers or only unsigned integers, with `PyC_StructFmt_type_is_int_any`'s implementation being replaced with calling both of the split functions.

Thomas Barlow added 8 commits 2023-11-13 04:56:00 +01:00

87e0fabd3c Split PyC_StructFmt_type_is_int_any into signed/unsigned

dd75403bbb Add comments explaining each buffer format type

d202237505 Add function to check if a buffer format is in native byte order

foreach_getset and pyprop_array_foreach_getset would only accept buffer
formats with an implicit '@' prefix because they would only check the
first character of the buffer's format.

This new function will also accept explicit prefixes that match the
current system's native byte order.

d5e57b3443 Update foreach_getset to use the format string utils in py_capi_utils.h

a385c10774 Update pyprop_array_foreach_getset to use the format string utils in py_capi_utils.h

8a48d516ab make format

57bbe936ea Move an in-line comment above the line for better formatting

7e07bc1628 Add comment explaining which formats PyC_StructFmt_type_from_str supports

Thomas Barlow added 40 commits 2023-12-16 18:49:37 +01:00

789bd5b7cd Fix existing tests using the wrong variables

b12d52d7d0 TestPropArray use `id_inst`

18f1e8ca20 Add tests for buffers with "@", "<" and ">" prefixes

af219fc2be Separate combined test methods into individual test methods

529bee8fcd Add tests for buffers with too small itemsize

These could be considered safe casts if we were to support casting in
the future (as opposed to larger itemsizes which would be unsafe casts),
though since float16 support may be lacking, it may be best to only
consider the case of int16.

3032143475 Add test for systems where both C int and C long are 32 bits

Systems where this is the case should be 32-bit Linux and Windows.

If we had tests for a double typed property, then we could also add a
similar test for where C double and C long double are both 64 bits.

c10ff91945 Add bpy_prop_collection tests

edab4f2a0e Deduplicate bpy_prop_collection test code using helper functions

6cdd724e36 Replace DummySequenceBuffer with an ndarray subclass that tracks whether it has been accessed as a sequence

This is much simpler.

efd1d4db8b Only allow np.ndarray buffers to ensure that no sequence tests are mistakenly testing buffer code

16802c4bf3 Remove hardcoded buffer/sequence lengths from prop_collection tests

19aa7d0c6e Add prop_collection tests for when C int and C long are 32 bits

189ab74bd6 Tidy up and comment helper functions

a562e353a4 Fix prop_array tests failing due to np.ndarray comparisons

270e2d1961 Rewrite most prop_collection tests to not care about which dtype results in a compatible buffer, only that at least one exists

bool tests are the exception, which expect to be a compatible buffer.

88745f682e formatting and comments

df5317ecdf Reorder tests

Move all sequence tests together

fceed82116 Deduplicate int and enum buffer tests

7f37a53f5f Big tests update

2ee7c6a8b0 Merge remote-tracking branch 'origin/main' into py_buffer_all_use_capi_utils_pr

841e96c298 Add tests for other byteorders

Ensure bool tests will pass if the bool property has raw access

a579d0b3c5 Begin breaking up test helper functions into smaller parts for simplicity

148d3455b8 Further split apart functions

Remove extra handling of ctypes.c_void_p arrays because the buffer always including an explicit byteorder/size/alignment prefix is annoying to deal with

3297fda9c4 Add attributes to tests

b8f75f02a2 Comment updates

9cdb66ab02 Separate idprop and attribute tests into subclasses

Update sequence generation to use the default value of the property the sequences are being generated for

6bb090df1a Separate idprop and attribute tests into subclasses

Update sequence generation to use the default value of the property the sequences are being generated for

23ae1daee0 Fix tests enum default value lookup from identifier

a01c6dbc96 Replace tests default value lookup with initial value lookup

This will allow for testing non-runtime properties where we have less control over the property values.

794578028d Fix tests enum default value lookup from identifier in prop_as_sequence

bf230cb8da Add test support for multi-dimensional array properties

8d00162cbd Add tests explicitly for properties without raw array access

This is done by creating a Collection and linking Objects into that Collection so that foreach_get/foreach_set can be used on those Objects.

8e399b82de Rename test classes and add docstrings

b8f60286f1 Add tests for collections with no item property pointer

c876643379 Add tests for collections with no item property pointer with mixed items

a410bd1abd Tidy up tests

d5677d3080 Tidy up tests

465a0c00c4 Tidy up tests

1379723407 Tidy up and document tests

2d8dfbfdb8 Merge remote-tracking branch 'origin/main' into py_buffer_all_use_capi_utils_pr

This pull request has changes conflicting with the target branch.

source/blender/python/intern/bpy_rna.cc
tests/python/bl_pyapi_prop_array.py

View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u py_buffer_all_use_capi_utils_pr:Mysteryem-py_buffer_all_use_capi_utils_pr

git checkout Mysteryem-py_buffer_all_use_capi_utils_pr

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

WIP: PyAPI: Use py_capi_utils in bpy_prop_array and bpy_prop_collection foreach_get/set #114773

Checkout