Including native Python header files within the python directory bundled with Blender #128242

New Issue

swiss_knight · 2024-09-27T12:29:57+02:00

swiss_knight commented

2024-09-27 12:29:57 +02:00

System Information
Operating system: Ubuntu 22.04.5 LTS
Graphics card: Nvidia Quadro T500 Mobile

Blender Version
Broken: 4.2.2
Worked: N/A (didn't tested much older versions about that issue)

Short description of error
Hello,

the python folder packaged with Blender is sorely lacking the common Python header files:

BLENDER_HOME="/home/username/blender" # Blender install dir
cd ${BLENDER_HOME}/4.2/python
$ tree -L 2
.
├── bin
│   └── python3.11
└── lib
    ├── libpython3.11.a
    └── python3.11/

My suggestion is that it should also include, well, an include/ folder such as:

$ tree -L 2
.
├── bin
│   └── python3.11
├── include
│    └── python3.11/
└── lib
    ├── libpython3.11.a
    └── python3.11/

This ìnclude/python3.11/ folder should at least contain the same as the header provider by the apt package libpython3.11-dev in the unix directory /usr/include/python3.11/:

$ cd /usr/include/python3.11
$ tree -L 1
.
├── abstract.h
├── bltinmodule.h
├── boolobject.h
├── bytearrayobject.h
├── bytesobject.h
├── ceval.h
├── codecs.h
├── compile.h
├── complexobject.h
├── cpython
├── datetime.h
├── descrobject.h
├── dictobject.h
├── dynamic_annotations.h
├── enumobject.h
├── errcode.h
├── exports.h
├── fileobject.h
├── fileutils.h
├── floatobject.h
├── frameobject.h
├── genericaliasobject.h
├── graminit.h
├── import.h
├── internal
├── intrcheck.h
├── iterobject.h
├── listobject.h
├── longobject.h
├── marshal.h
├── memoryobject.h
├── methodobject.h
├── modsupport.h
├── moduleobject.h
├── object.h
├── objimpl.h
├── opcode.h
├── osdefs.h
├── osmodule.h
├── patchlevel.h
├── pybuffer.h
├── pycapsule.h
├── pyconfig.h
├── py_curses.h
├── pydtrace.h
├── pyerrors.h
├── pyexpat.h
├── pyframe.h
├── pyhash.h
├── pylifecycle.h
├── pymacconfig.h
├── pymacro.h
├── pymath.h
├── pymem.h
├── pyport.h
├── pystate.h
├── pystrcmp.h
├── pystrtod.h
├── Python.h
├── pythonrun.h
├── pythread.h
├── pytypedefs.h
├── rangeobject.h
├── setobject.h
├── sliceobject.h
├── structmember.h
├── structseq.h
├── sysmodule.h
├── token.h
├── traceback.h
├── tracemalloc.h
├── tupleobject.h
├── typeslots.h
├── unicodeobject.h
├── warnings.h
└── weakrefobject.h

It's currently quite light to include these files: 192 items, totalling 847.2 kB

This is especially useful when one want to install some custom Python packages which needs some C++ libs such as gdal for example:

$ cd ${BLENDER_HOME}/4.2/python/bin
$ ./python3.11 -m pip install gdal==3.4.1
Collecting gdal==3.4.1
  Using cached GDAL-3.4.1.tar.gz (755 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: gdal
  Building wheel for gdal (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for gdal (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [143 lines of output]
...
g++ -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/home/<username_placeholder>/Downloads/blender-4.2.2-linux-x64/4.2/python/include/python3.11 -I. -I/usr/include -c extensions/ogr_wrap.cpp -o build/temp.linux-x86_64-cpython-311/extensions/ogr_wrap.o -I/usr/include/gdal
      building 'osgeo._gnm' extension
...
extensions/gnm_wrap.cpp:175:11: fatal error: Python.h: No such file or directory
        175 | # include <Python.h>
            |           ^~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/g++' failed with exit code 1
      [end of output]
...

And indeed the folder /home/<username_placeholder>/Downloads/blender-4.2.2-linux-x64/4.2/python/include/python3.11 does not exist after a fresh install of Blender!

To this end, the wheel package can also be useful, as can the pip executable in the bin/ folder, which could be a great help because the pip installed at the OS level is not necessarily plugged with the same Python version as the one embedded in Blender so chance are you will face a bad interpreter: No such file or directory error (plus it will not install pacakges in the Blender site-packages directory) and also Python path in Blender is not aware of Python packages installed at the OS or USER level (which is a good thing).

See also: https://github.com/OSGeo/gdal/issues/10877#issuecomment-2377140231

Thanks!

Warm Regards.

Exact steps for others to reproduce the error
Vanilla install from the archive file on the official Blender download page: https://www.blender.org/download/

**System Information** Operating system: Ubuntu 22.04.5 LTS Graphics card: Nvidia Quadro T500 Mobile **Blender Version** Broken: 4.2.2 Worked: N/A (didn't tested much older versions about that issue) **Short description of error** Hello, the python folder packaged with Blender is sorely lacking the common Python header files: ``` BLENDER_HOME="/home/username/blender" # Blender install dir cd ${BLENDER_HOME}/4.2/python $ tree -L 2 . ├── bin │ └── python3.11 └── lib ├── libpython3.11.a └── python3.11/ ``` My suggestion is that it should also include, well, an `include/` folder such as: ```sh $ tree -L 2 . ├── bin │ └── python3.11 ├── include │ └── python3.11/ └── lib ├── libpython3.11.a └── python3.11/ ``` This `ìnclude/python3.11/` folder should at least contain the same as the header provider by the apt package `libpython3.11-dev` in the unix directory `/usr/include/python3.11/`: ```sh $ cd /usr/include/python3.11 $ tree -L 1 . ├── abstract.h ├── bltinmodule.h ├── boolobject.h ├── bytearrayobject.h ├── bytesobject.h ├── ceval.h ├── codecs.h ├── compile.h ├── complexobject.h ├── cpython ├── datetime.h ├── descrobject.h ├── dictobject.h ├── dynamic_annotations.h ├── enumobject.h ├── errcode.h ├── exports.h ├── fileobject.h ├── fileutils.h ├── floatobject.h ├── frameobject.h ├── genericaliasobject.h ├── graminit.h ├── import.h ├── internal ├── intrcheck.h ├── iterobject.h ├── listobject.h ├── longobject.h ├── marshal.h ├── memoryobject.h ├── methodobject.h ├── modsupport.h ├── moduleobject.h ├── object.h ├── objimpl.h ├── opcode.h ├── osdefs.h ├── osmodule.h ├── patchlevel.h ├── pybuffer.h ├── pycapsule.h ├── pyconfig.h ├── py_curses.h ├── pydtrace.h ├── pyerrors.h ├── pyexpat.h ├── pyframe.h ├── pyhash.h ├── pylifecycle.h ├── pymacconfig.h ├── pymacro.h ├── pymath.h ├── pymem.h ├── pyport.h ├── pystate.h ├── pystrcmp.h ├── pystrtod.h ├── Python.h ├── pythonrun.h ├── pythread.h ├── pytypedefs.h ├── rangeobject.h ├── setobject.h ├── sliceobject.h ├── structmember.h ├── structseq.h ├── sysmodule.h ├── token.h ├── traceback.h ├── tracemalloc.h ├── tupleobject.h ├── typeslots.h ├── unicodeobject.h ├── warnings.h └── weakrefobject.h ``` It's currently quite light to include these files: `192 items, totalling 847.2 kB` This is especially useful when one want to install some custom Python packages which needs some C++ libs such as `gdal` for example: ```sh $ cd ${BLENDER_HOME}/4.2/python/bin $ ./python3.11 -m pip install gdal==3.4.1 Collecting gdal==3.4.1 Using cached GDAL-3.4.1.tar.gz (755 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Building wheels for collected packages: gdal Building wheel for gdal (pyproject.toml) ... error error: subprocess-exited-with-error × Building wheel for gdal (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [143 lines of output] ... g++ -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/home/<username_placeholder>/Downloads/blender-4.2.2-linux-x64/4.2/python/include/python3.11 -I. -I/usr/include -c extensions/ogr_wrap.cpp -o build/temp.linux-x86_64-cpython-311/extensions/ogr_wrap.o -I/usr/include/gdal building 'osgeo._gnm' extension ... extensions/gnm_wrap.cpp:175:11: fatal error: Python.h: No such file or directory 175 | # include <Python.h> | ^~~~~~~~~~ compilation terminated. error: command '/usr/bin/g++' failed with exit code 1 [end of output] ... ``` And indeed the folder `/home/<username_placeholder>/Downloads/blender-4.2.2-linux-x64/4.2/python/include/python3.11` does not exist after a fresh install of Blender! To this end, the `wheel` package can also be useful, as can the `pip` executable in the `bin/` folder, which could be a great help because the `pip` installed at the OS level is not necessarily plugged with the same Python version as the one embedded in Blender so chance are you will face a `bad interpreter: No such file or directory` error (plus it will not install pacakges in the Blender site-packages directory) and also Python path in Blender is not aware of Python packages installed at the OS or USER level (which is a good thing). See also: https://github.com/OSGeo/gdal/issues/10877#issuecomment-2377140231 Thanks! Warm Regards. **Exact steps for others to reproduce the error** Vanilla install from the archive file on the official Blender download page: https://www.blender.org/download/

swiss_knight added the

labels 2024-09-27 12:29:58 +02:00

YimingWu commented

2024-09-27 16:12:40 +02:00

I don't personally see a point against including those headers. Though on ubuntu they come with two packages libpython3.11 and libpython3.11-dev means that distributed tools probably already should have compiled binaries that you can use out of box instead having everything re-built from scratch.

I don't think we previously had reports on blender plugins who install stuff via pip that's encountering this issue. Maybe a good thing too is to have gdal mention the requirement on their page.

let me poke some python people here.

I don't personally see a point against including those headers. Though on ubuntu they come with two packages `libpython3.11` and `libpython3.11-dev` means that distributed tools probably already should have compiled binaries that you can use out of box instead having everything re-built from scratch. I don't think we previously had reports on blender plugins who install stuff via pip that's encountering this issue. Maybe a good thing too is to have `gdal` mention the requirement on their page. let me poke some python people here.

YimingWu added

Module

Python API

Status

Needs Info from Developers

and removed

Status

Needs Triage

labels 2024-09-27 16:12:52 +02:00

Campbell Barton added

Type

Design