VFX Reference Platform 2020 Compatibility #68774

New Issue

Brecht Van Lommel · 2019-08-17T15:35:40+02:00

Brecht Van Lommel commented

2019-08-17 15:35:40 +02:00

Starting with CY2020, the VFX Reference Platform uses Python 3.7. This means we can now align with it.

The decision was made to align all releases in 2020 to the CY2020 platform, which starts with Blender 2.82.

The VFX reference platform is currently for Linux only. It is planned to be extended to Windows and macOS in CY2021. Since we use the same libraries on all platforms and there is no specified compiled ABI for Windows and macOS, there is nothing specific that we need to do for those platforms.

Being compatible with the reference platform does not mean we have to use all the exact compiler or library versions listed. What matters is effective API/ABI compatibility, and file format compatibility. There are a few different considerations listed below.

GCC and glibc

We are already compatible with the required glibc version 2.17.

We can use newer GCC or Clang versions that produce more optimized code, as long as they are ABI compatible with the stated GCC version.

C/C++ Compatibility

We currently do not have a public C/C++ API, and are also not exposing any of the external libraries as an API.

A change we could make is to hide symbols by default. We currently hide some symbols that are likely to conflict, but not all. This is already automatically done on Windows, but not on Linux or macOS. Plug-ins can generally handle this on their side too, but we may as well make it easier to not conflict.

Python Compatibility

To guarantee full compatibility, we have to use Python 3.7 and Numpy 1.17. Python scripts written for 3.7 should generally work fine in 3.8, but not the other way around. For extensions there is a stable ABI , but it's not clear this is widely used and so extensions would be incompatible.

The downside is that we'd have to wait a year use Python 3.8 after its release. As the Python ecosystem moves to 3.8, we then deviate from the rest of the Python community that are not following the VFX reference platform.

For example on Windows the default version to download from the Python website in 2020 would be 3.8. But this means it would not be compatible with Blender.

File Format Compatibility

Libraries like OpenEXR, OpenVDB and OpenColorIO read and write files. We want those versions to be new enough that we can read files written by other software. Further, using a newer version can give files that other applications can't read.

Note for example that the OpenVDB file format remained compatible across some major versions. In such cases we can use a newer library version, if it contains functionality that we need.

On the other hand OpenColorIO 2 is breaking compatibility. It contains improvements that are quite useful to us, but is only planned for CY2021.

Starting with CY2020, the [VFX Reference Platform ](https://vfxplatform.com/) uses Python 3.7. This means we can now align with it. The decision was made to align all releases in 2020 to the CY2020 platform, which starts with Blender 2.82. The VFX reference platform is currently for Linux only. It is planned to be extended to Windows and macOS in CY2021. Since we use the same libraries on all platforms and there is no specified compiled ABI for Windows and macOS, there is nothing specific that we need to do for those platforms. Being compatible with the reference platform does not mean we have to use all the exact compiler or library versions listed. What matters is effective API/ABI compatibility, and file format compatibility. There are a few different considerations listed below. ### GCC and glibc We are already compatible with the required glibc version 2.17. We can use newer GCC or Clang versions that produce more optimized code, as long as they are ABI compatible with the stated GCC version. ### C/C++ Compatibility We currently do not have a public C/C++ API, and are also not exposing any of the external libraries as an API. A change we could make is to hide symbols by default. We currently hide some symbols that are likely to conflict, but not all. This is already automatically done on Windows, but not on Linux or macOS. Plug-ins can generally handle this on their side too, but we may as well make it easier to not conflict. ### Python Compatibility To guarantee full compatibility, we have to use Python 3.7 and Numpy 1.17. Python scripts written for 3.7 should generally work fine in 3.8, but not the other way around. For extensions there is a [stable ABI ](https://docs.python.org/3/c-api/stable.html), but it's not clear this is widely used and so extensions would be incompatible. The downside is that we'd have to wait a year use Python 3.8 after its release. As the Python ecosystem moves to 3.8, we then deviate from the rest of the Python community that are not following the VFX reference platform. For example on Windows the default version to download from the Python website in 2020 would be 3.8. But this means it would not be compatible with Blender. ### File Format Compatibility Libraries like OpenEXR, OpenVDB and OpenColorIO read and write files. We want those versions to be new enough that we can read files written by other software. Further, using a newer version can give files that other applications can't read. Note for example that the OpenVDB file format remained compatible across some major versions. In such cases we can use a newer library version, if it contains functionality that we need. On the other hand OpenColorIO 2 is breaking compatibility. It contains improvements that are quite useful to us, but is only planned for CY2021.

Brecht Van Lommel commented

2019-08-17 15:35:40 +02:00

Added subscriber: @brecht

Maciej Jutrzenka commented

2019-08-17 15:56:57 +02:00

Added subscriber: @MaciejJutrzenka

Maciej Jutrzenka commented

2019-08-17 15:56:57 +02:00

i never was a major fan of VFX reference platform as it chooked stuff for many many years. so i would say to follow it on API side but not on internal one.

also correct me if i am wrong but.. if u write plugin in python A version... it will not work with other apps... even if they support python A? right? so what is the point of beying on same python.. if it is not that u write 1 plugin and it works for all others DCC...

file formats okey fine. that is very logical. but other stuff i think should be newest?

i never was a major fan of VFX reference platform as it chooked stuff for many many years. so i would say to follow it on API side but not on internal one. also correct me if i am wrong but.. if u write plugin in python A version... it will not work with other apps... even if they support python A? right? so what is the point of beying on same python.. if it is not that u write 1 plugin and it works for all others DCC... file formats okey fine. that is very logical. but other stuff i think should be newest?

Brecht Van Lommel commented

2019-08-17 16:14:02 +02:00

There are two reason for using the same Python version:

For add-on developers creating extensions specifically for Blender, it would be most convenient if that could use the same Python version as other software. This is not a major issues in my opinion, installing multiple Python versions side by side is not that difficult and the number of developers doing this is not that high.
For a user of multiple softwares, it helps to have a single Python version for which they can install Python packages and use them across all software. This is the more important consideration, for technical users and studios that use Python scripts for their pipeline. Dealing with multiple Python versions here is problematic.

However, Python 3.8 will the default Python version to download from python.org in 2020, and so for Windows it's not going to be convenient if Blender uses a different version.

There are two reason for using the same Python version: * For add-on developers creating extensions specifically for Blender, it would be most convenient if that could use the same Python version as other software. This is not a major issues in my opinion, installing multiple Python versions side by side is not that difficult and the number of developers doing this is not that high. * For a user of multiple softwares, it helps to have a single Python version for which they can install Python packages and use them across all software. This is the more important consideration, for technical users and studios that use Python scripts for their pipeline. Dealing with multiple Python versions here is problematic. However, Python 3.8 will the default Python version to download from python.org in 2020, and so for Windows it's not going to be convenient if Blender uses a different version.

Ray molenkamp commented

2019-08-17 17:19:44 +02:00

Added subscriber: @LazyDodo

Ray molenkamp commented

2019-08-17 17:19:44 +02:00

The few times i have interacted with people asking about the VFX Platform, it was always these things

The python version

But when you asked a little further, they only reason they cared about the python version because they really wanted to use

QT/PyQt/PySide

which required py 2.7

the glibc version, we jumped too soon to a too new glibc, locking out our binary for the redhat/centos guys, naturally these people were not super thrilled. Not sure how many actually cared about the VFX platform but if it's your only piece of ammunition, you're gonna use it.

3 is already addressed, aligning our selves with the python requirement of 1 shouldn't be too hard, but 2... yikes.... that's gonna balloon our distro size significantly for a thing has come up only sporadically sofar

The few times i have interacted with people asking about the VFX Platform, it was always these things 1) The python version But when you asked a little further, they only reason they cared about the python version because they really wanted to use 2) QT/PyQt/PySide which required py 2.7 3) the glibc version, we jumped too soon to a too new glibc, locking out our binary for the redhat/centos guys, naturally these people were not super thrilled. Not sure how many actually cared about the VFX platform but if it's your only piece of ammunition, you're gonna use it. 3 is already addressed, aligning our selves with the python requirement of 1 shouldn't be too hard, but 2... yikes.... that's gonna balloon our distro size significantly for a thing has come up only sporadically sofar

Maciej Jutrzenka commented

2019-08-17 18:33:00 +02:00

About Glibc... aren't newest redhat 8.0 comes with glibc 2.28...

Brecht Van Lommel commented

2019-08-18 02:28:48 +02:00

@LazyDodo, we are not going to include Qt as part of trying to match the VFX reference platform, I have only libraries we already ship in mind here.

Campbell Barton commented

2019-11-19 03:32:21 +01:00

Added subscriber: @ideasman42

Campbell Barton commented

2019-11-19 03:32:21 +01:00

VFX Platform

Regarding the VFX platform it's self.

Who is currently following this?
Of the projects that follow this, do they comply with it entirely, or pick and choose which parts of it to follow? (ignoring libraries they don't use)
Do they publish a rationale behind their decision making process?
Is this something Blender could be involved with?

Python

My main concerns are:

They depend on QT libraries (holding back Python years?)

because of a library we don't use and aren't likely to take advantage of.

While currently they propose to use a recent Python version

there reasons for updating Python aren't aligned with ours.

Blender depends heavily on Python,

Other applications have C/C++ API's... meaning, having an older Python
platform doesn't negatively impact them the same way it does for us.

In short, we would be adhering to a set of versions we have no/to little control over,
defined by a group who don't depend on Python as their main method extension.

In practice writing Python that's compatible between two minor releases isn't so hard,
so I think the advantages with locking ourselves to an older Python
don't outweigh the benefits with using the latest.

Other Libs

Further, minimum requirements for glibc and other libraries,
I have no strong opinions on this.

Measuring Success

I assume the intention is that this would help in VFX studios adoption. However it would be good to know if this is really working as intended.

How would we measure if this is a success or not?

**VFX Platform** Regarding the VFX platform it's self. - Who is currently following this? - Of the projects that follow this, do they comply with it entirely, or pick and choose which parts of it to follow? *(ignoring libraries they don't use)* - Do they publish a rationale behind their decision making process? - Is this something Blender could be involved with? ---- **Python** My main concerns are: - They depend on QT libraries (holding back Python years?) ``` because of a library we don't use and aren't likely to take advantage of. ``` - While currently they propose to use a recent Python version ``` there reasons for updating Python aren't aligned with ours. ``` - Blender depends heavily on Python, ``` Other applications have C/C++ API's... meaning, having an older Python platform doesn't negatively impact them the same way it does for us. ``` In short, we would be adhering to a set of versions we have no/to little control over, defined by a group who don't depend on Python as their main method extension. In practice writing Python that's compatible between two minor releases isn't so hard, so I think the advantages with locking ourselves to an older Python don't outweigh the benefits with using the latest. ---- **Other Libs** Further, minimum requirements for glibc and other libraries, I have no strong opinions on this. ---- **Measuring Success** I assume the intention is that this would help in VFX studios adoption. However it would be good to know if this is really working as intended. How would we measure if this is a success or not?

lvxejay commented

2019-11-20 00:39:22 +01:00

Added subscriber: @lvxejay

lvxejay commented

2019-11-20 02:08:03 +01:00

Who is currently following this?
Of the projects that follow this, do they comply with it entirely, or pick and choose which parts of it to follow? (ignoring libraries they don't use)

I think it's difficult to quantify who/how many people are using the VFX Reference Platform. Additionally, I do not think there is strict compliance with the platform among end-users. It seems that generally, software vendors are trying, but the Python 2 -> 3 move has thrown a massive monkey wrench in the process.

@ideasman42 Regarding your other points, they are all valid. As you mentioned, Blender is extremely dependent on Python. Due to this fact though, I think we need a nuanced approach to this situation. In other words, we should wait until 2021 to make the jump to Python 3.8.

Yes, pure python code is mostly portable between Python 3.7 and Python 3.8. However, that is not the case for python extensions and modules which is an important use case for addon/plugin developers and studio adoption prospects. Major software vendors have not even begun shipping Python 3.7 builds of their software. In fact, some of the larger vendors won't have builds until 2021. Personally, I do not understand the rationale behind jumping into Python 3.8 so quickly. There is nothing wrong with waiting for a year. At the very least, we could wait until the 2021 platform specification is released, but I think it would be more responsible to just wait for most of the industry to transition to Python 3.7, first. It's also still pretty early into the Blender 2.8X release cycle. I'm sure many end-users would appreciate the added benefit of more bug fixes and features that will be added in the next few Blender releases. The fact that their copy of Blender will remain mostly ABI compatible with the rest of the industry for an extended period of time should not be ignored or understated.

Lagging behind the official Python releases is not inherently bad, but I do understand why you would want to avoid this. However, I think we should just wait for the industry to "catch up" first. As it pertains to Python on the VFX Reference Platform, once the rest of the industry catches up, I think we can go back to tracking the Python releases.

> Who is currently following this? > Of the projects that follow this, do they comply with it entirely, or pick and choose which parts of it to follow? (ignoring libraries they don't use) I think it's difficult to quantify who/how many people are using the VFX Reference Platform. Additionally, I do not think there is strict compliance with the platform among end-users. It seems that generally, software vendors are *trying*, but the Python 2 -> 3 move has thrown a massive monkey wrench in the process. @ideasman42 Regarding your other points, they are all valid. As you mentioned, Blender is extremely dependent on Python. Due to this fact though, I think we need a *nuanced* approach to this situation. In other words, we should wait until 2021 to make the jump to Python 3.8. Yes, pure python code is mostly portable between Python 3.7 and Python 3.8. However, that is not the case for python extensions and modules which is an important use case for addon/plugin developers and studio adoption prospects. Major software vendors have not even begun shipping Python 3.7 builds of their software. In fact, some of the larger vendors won't have builds until 2021. Personally, I do not understand the rationale behind jumping into Python 3.8 so quickly. There is nothing wrong with waiting for a year. At the very least, we could wait until the 2021 platform specification is released, but I think it would be more responsible to just wait for most of the industry to transition to Python 3.7, first. It's also still pretty early into the Blender 2.8X release cycle. I'm sure many end-users would appreciate the added benefit of more bug fixes and features that will be added in the next few Blender releases. The fact that their copy of Blender will remain mostly ABI compatible with the rest of the industry for an extended period of time should not be ignored or understated. Lagging behind the official Python releases is not inherently bad, but I do understand why you would want to avoid this. However, I think we should just wait for the industry to "catch up" first. As it pertains to Python on the VFX Reference Platform, once the rest of the industry catches up, I think we can go back to tracking the Python releases.

Campbell Barton commented

2019-11-20 07:04:22 +01:00

The fact that their copy of Blender will remain mostly ABI compatible with the rest of the industry for an extended period of time should not be ignored or understated.

Blender doesn't export symbols on all platforms, in general add-ons don't depend on ABI stability.

I can see you're making the case for this, I remain unconvinced it's a net gain. Just as users probably prefer not to use 1-2 years old Blender, Python developers can take advantage of recent improvements to Python. It's no rush - we bundle Python therefor we don't have to be stuck to 1-2 years old Python as you're suggesting.

> The fact that their copy of Blender will remain mostly ABI compatible with the rest of the industry for an extended period of time should not be ignored or understated. Blender doesn't export symbols on all platforms, in general add-ons don't depend on ABI stability. I can see you're making the case for this, I remain unconvinced it's a net gain. Just as users probably prefer not to use 1-2 years old Blender, Python developers can take advantage of recent improvements to Python. It's no rush - we bundle Python therefor we don't have to be stuck to 1-2 years old Python as you're suggesting.

Roger B commented

2019-11-20 08:29:25 +01:00

Added subscriber: @rboxman

Roger B commented

2019-11-20 08:29:25 +01:00

This goes beyond Blender. Blender does not exist in a vacuum. The vfx platform exists not to limit a piece of software's dependencies because they feel like it that year. It's there so "studios" can piece together the myriad of software required for their pipeline to support dozens of artists. A studio of any significance today is probably using Maya, Zbrush, Houdini, Nuke, etc. and yes, for the most part, they are all pretty well standardized to the platform's requirements. Even UnrealEngine is moving towards Python3 to align for instance [1].

That said, there are plenty of studios that chose to stay on particular versions for a variety of reasons too; sometimes because they don't all cooperate together (something the vfx platform is attempting to solve as best it can obviously).

An example studio that's doing this today is Goodbye Kansas. An artists from there even gave a talk at the Blender conference this year. One of their pipeline tools is integrated into Maya, Houdini, and Blender using a single PyQT codebase. Now their artists get a consistent experience in each software and their tech department is probably happier too.[2]

The questions that should be answered are:
- Does Blender want to be friendly to such heterogeneous environments? Or does Blender want to pretend they don't exist?
- At minimum, using the concrete example of PyQT, would Blender using Python3.8 break Goodby Kansas? Would they be required to stick with Blender 2.81 until the reference platform increases the version and Maya and Houdini update? That might be ok. Just get everyone onboard so it's a known thing.
- I know it would suck, but would a special, yearly version of Blender which aligns be possible? The normal version is free to explore new versions but the yearly version would be "studio friendly". Noone will like this, but make the call here and then make it official.

https://docs.unrealengine.com/en-US/Engine/Editor/ScriptingAndAutomation/Python/index.html (The Tip section under Python 2.7)
https://twitter.com/3DBystedt/status/1191527352012070912

This goes beyond Blender. Blender does not exist in a vacuum. The vfx platform exists not to limit a piece of software's dependencies because they feel like it that year. It's there so "studios" can piece together the myriad of software required for their pipeline to support dozens of artists. A studio of any significance today is probably using Maya, Zbrush, Houdini, Nuke, etc. and yes, for the most part, they are all pretty well standardized to the platform's requirements. Even UnrealEngine is moving towards Python3 to align for instance [1]. That said, there are plenty of studios that chose to stay on particular versions for a variety of reasons too; sometimes because they don't all cooperate together (something the vfx platform is attempting to solve as best it can obviously). An example studio that's doing this today is Goodbye Kansas. An artists from there even gave a talk at the Blender conference this year. One of their pipeline tools is integrated into Maya, Houdini, and Blender using a single PyQT codebase. Now their artists get a consistent experience in each software and their tech department is probably happier too.[2] The questions that should be answered are: - Does Blender want to be friendly to such heterogeneous environments? Or does Blender want to pretend they don't exist? - At minimum, using the concrete example of PyQT, would Blender using Python3.8 break Goodby Kansas? Would they be required to stick with Blender 2.81 until the reference platform increases the version and Maya and Houdini update? That might be ok. Just get everyone onboard so it's a known thing. - I know it would suck, but would a special, yearly version of Blender which aligns be possible? The normal version is free to explore new versions but the yearly version would be "studio friendly". Noone will like this, but make the call here and then make it official. - [x] https://docs.unrealengine.com/en-US/Engine/Editor/ScriptingAndAutomation/Python/index.html (The Tip section under Python 2.7) - [x] https://twitter.com/3DBystedt/status/1191527352012070912

Campbell Barton commented

2019-11-20 10:48:55 +01:00

An issue with the replies so far is they don't give any concrete insights, it would be useful if people who work in studios would comment on topics such as.

What C-extensions are studios using which would be important to run inside Blender?
Do these extensions use the stable Python C-API? (which allows 3.7x / 3.8x interoperability).
PyQT is being mentioned (by @rboxman), did anyone successfully use this inside Blender? Wherever I looked into this GUI toolkits tend not to work well with Blender as they need their own main loop.

I don't dispute that someone somewhere would benefit from this proposal, it would just be good to hear from them directly, to get a sense of whats important, what isn't.

Edit removed thumbs-down since there are too many open topics still.

An issue with the replies so far is they don't give any concrete insights, it would be useful if people who work in studios would comment on topics such as. - What C-extensions are studios using which would be important to run inside Blender? - Do these extensions use the stable Python C-API? (which allows 3.7x / 3.8x interoperability). - PyQT is being mentioned (by @rboxman), did anyone successfully use this inside Blender? Wherever I looked into this GUI toolkits tend not to work well with Blender as they need their own main loop. I don't dispute that someone somewhere would benefit from this proposal, it would just be good to hear from them directly, to get a sense of whats important, what isn't. ----- *Edit* removed *thumbs-down* since there are too many open topics still.

Brecht Van Lommel commented

2019-11-20 16:03:16 +01:00

In #68774#812767, @ideasman42 wrote:

Who is currently following this, and of the projects that follow this, do they comply with it entirely, or pick and choose which parts of it to follow? (ignoring libraries they don't use)

As far as I know Autodesk, The Foundry and Side Effects follow it. And not always exactly, but quite close.

Do they publish a rationale behind their decision making process?

There is no summary of that, but non-obvious changes generally have a note on the webpage or discussion on the mailing list I think.

Is this something Blender could be involved with?

I'm pretty sure we could if we wanted.

They depend on QT libraries (holding back Python years?)
because of a library we don't use and aren't likely to take advantage of.

While currently they propose to use a recent Python version
there reasons for updating Python aren't aligned with ours.

I think this was more of a one time thing related to the Python 2/3 switch. I'm not sure it is representative of what will happen going forward, it's hard to predict.

Blender depends heavily on Python,
Other applications have C/C++ API's... meaning, having an older Python
platform doesn't negatively impact them the same way it does for us.

In short, we would be adhering to a set of versions we have no/to little control over,
defined by a group who don't depend on Python as their main method extension.

In practice writing Python that's compatible between two minor releases isn't so hard,
so I think the advantages with locking ourselves to an older Python
don't outweigh the benefits with using the latest.

It's more about the users than Python vs C/C++ APIs I think, other apps also heavily rely on Python (for example the Houdini API for external renderers is Python only).

The main issue is Python packages and extensions, it's easier if you can have a single Python environment shared between 3D software. On the other hand, sticking to Python 3.7 for many users makes it more difficult to have a Python environment shared between Blender and other software which is likely using the latest version. The stable API makes that possible in principle, but as far as I know it's not widely used.

So either way, it's not an ideal solution and we have to make a choice. As I said on bf-committers, I think it's acceptable to use Python 3.8 given the trade-offs. This does not make anything impossible for studios, it's just more work to deal with multiple Python environments. The percentage of developers that need to deal with that is smaller, they may need to do that already for other reasons, so to me it seems ok on balance.

Going forward it may be best to get involved in the VFX reference platform and argue for tracking Python versions more closely.

> In #68774#812767, @ideasman42 wrote: > - Who is currently following this, and of the projects that follow this, do they comply with it entirely, or pick and choose which parts of it to follow? *(ignoring libraries they don't use)* As far as I know Autodesk, The Foundry and Side Effects follow it. And not always exactly, but quite close. > - Do they publish a rationale behind their decision making process? There is no summary of that, but non-obvious changes generally have a note on the webpage or discussion on the mailing list I think. > - Is this something Blender could be involved with? I'm pretty sure we could if we wanted. > - They depend on QT libraries (holding back Python years?) > because of a library we don't use and aren't likely to take advantage of. > - While currently they propose to use a recent Python version > there reasons for updating Python aren't aligned with ours. I think this was more of a one time thing related to the Python 2/3 switch. I'm not sure it is representative of what will happen going forward, it's hard to predict. > - Blender depends heavily on Python, > Other applications have C/C++ API's... meaning, having an older Python > platform doesn't negatively impact them the same way it does for us. > > In short, we would be adhering to a set of versions we have no/to little control over, > defined by a group who don't depend on Python as their main method extension. > > In practice writing Python that's compatible between two minor releases isn't so hard, > so I think the advantages with locking ourselves to an older Python > don't outweigh the benefits with using the latest. It's more about the users than Python vs C/C++ APIs I think, other apps also heavily rely on Python (for example the Houdini API for external renderers is Python only). The main issue is Python packages and extensions, it's easier if you can have a single Python environment shared between 3D software. On the other hand, sticking to Python 3.7 for many users makes it more difficult to have a Python environment shared between Blender and other software which is likely using the latest version. The stable API makes that possible in principle, but as far as I know it's not widely used. So either way, it's not an ideal solution and we have to make a choice. As I said on bf-committers, I think it's acceptable to use Python 3.8 given the trade-offs. This does not make anything impossible for studios, it's just more work to deal with multiple Python environments. The percentage of developers that need to deal with that is smaller, they may need to do that already for other reasons, so to me it seems ok on balance. Going forward it may be best to get involved in the VFX reference platform and argue for tracking Python versions more closely.

Brecht Van Lommel commented

2019-11-20 16:18:20 +01:00

In #68774#813389, @lvxejay wrote:
However, I think we should just wait for the industry to "catch up" first. As it pertains to Python on the VFX Reference Platform, once the rest of the industry catches up, I think we can go back to tracking the Python releases.

This makes no sense to me, seems like the worst of both worlds to use the specified Python 3 version when the rest of the industry doesn't, and then to use a different one when it does.

In #68774#813426, @rboxman wrote:
- At minimum, using the concrete example of PyQT, would Blender using Python3.8 break Goodby Kansas?

If they already have it working now with Python 2 in other apps and Python 3 in Blender, this should not be a big deal. It doesn't make anything impossible, it's only about potential extra work for studio TDs and developers to maintain multiple Python environments.

- I know it would suck, but would a special, yearly version of Blender which aligns be possible?  The normal version is free to explore new versions but the yearly version would be "studio friendly".  Noone will like this, but make the call here and then make it official.

That would add way too much complexity to the release process and add-on ecosystem, to solve a smaller problem.

> In #68774#813389, @lvxejay wrote: > However, I think we should just wait for the industry to "catch up" first. As it pertains to Python on the VFX Reference Platform, once the rest of the industry catches up, I think we can go back to tracking the Python releases. This makes no sense to me, seems like the worst of both worlds to use the specified Python 3 version when the rest of the industry doesn't, and then to use a different one when it does. > In #68774#813426, @rboxman wrote: > - At minimum, using the concrete example of PyQT, would Blender using Python3.8 break Goodby Kansas? If they already have it working now with Python 2 in other apps and Python 3 in Blender, this should not be a big deal. It doesn't make anything impossible, it's only about potential extra work for studio TDs and developers to maintain multiple Python environments. > - I know it would suck, but would a special, yearly version of Blender which aligns be possible? The normal version is free to explore new versions but the yearly version would be "studio friendly". Noone will like this, but make the call here and then make it official. That would add way too much complexity to the release process and add-on ecosystem, to solve a smaller problem.

Campbell Barton commented

2019-11-20 16:31:36 +01:00

The main issue is Python packages and extensions, it's easier if you can have a single Python environment shared between 3D software. On the other hand, sticking to Python 3.7 for many users makes it more difficult to have a Python environment shared between Blender and other software which is likely using the latest version. The stable API makes that possible in principle, but as far as I know it's not widely used.

Right, having multiple Python distributions to manage isn't so nice, although it's more on the inconvenience level then being a road-block (with some exceptions of course).

More general questions:

Are there any other implications for upgrades/downgrades of any other libraries besides Python?
It would be good to have feedback from other developers.
How to come to final decision on this task?

If we're not going with VFX platform for 2.82, I'd like to upgrade to Python 3.8, this should be done earlier in the release cycle (soon). So I'd rather not leave this task in limbo too long.

> The main issue is Python packages and extensions, it's easier if you can have a single Python environment shared between 3D software. On the other hand, sticking to Python 3.7 for many users makes it more difficult to have a Python environment shared between Blender and other software which is likely using the latest version. The stable API makes that possible in principle, but as far as I know it's not widely used. Right, having multiple Python distributions to manage isn't so nice, although it's more on the inconvenience level then being a road-block *(with some exceptions of course)*. ---- More general questions: - Are there any other implications for upgrades/downgrades of any other libraries besides Python? - It would be good to have feedback from other developers. - How to come to final decision on this task? If we're not going with VFX platform for 2.82, I'd like to upgrade to Python 3.8, this should be done earlier in the release cycle (soon). So I'd rather not leave this task in limbo too long.

Maciej Jutrzenka commented

2019-11-20 16:35:17 +01:00

For me the case is really simple.. it is not about what version of Python or some other thing we will support but for how long. Doing changes rewrites that is what is annoying... for anybody... that is why i believe we should choose the newest Python that is currently shipped as stable etc And just stick with it for 2 years for example.

But also more importantly. Blender is not part of industry VFX pipeline We don't support well alembic, we don't support openVDB at all u can't import or export volumes. we don't support USD yet. So.. Considering I/O of blender is not VFX ready. Why we would limit our technology by supporting python of VFX. When we don't support basic file formats in the first place.

i believe that Blender should come with newest avable technology that is currently possible.

For me the case is really simple.. it is not about what version of Python or some other thing we will support but for how long. Doing changes rewrites that is what is annoying... for anybody... that is why i believe we should choose the newest Python that is currently shipped as stable etc And just stick with it for 2 years for example. But also more importantly. Blender is not part of industry VFX pipeline We don't support well alembic, we don't support openVDB at all u can't import or export volumes. we don't support USD yet. So.. Considering I/O of blender is not VFX ready. Why we would limit our technology by supporting python of VFX. When we don't support basic file formats in the first place. i believe that Blender should come with newest avable technology that is currently possible.

Brecht Van Lommel commented

2019-11-20 17:21:29 +01:00

In #68774#813749, @ideasman42 wrote:

Are there any other implications for upgrades/downgrades of any other libraries besides Python?

There are other libraries we should upgrade for 2.82, but nothing that would cause problems as far as I can tell.

It would be good to have feedback from other developers.

How to come to final decision on this task?
If we're not going with VFX platform for 2.82, I'd like to upgrade to Python 3.8, this should be done earlier in the release cycle (soon). So I'd rather not leave this task in limbo too long.

For Python 3.8 the decision is with the Python module owners and the Blender admins. I'd say, communicate as Python module owners that you agree to upgrade to 3.8, and unless any Blender admins object you go ahead.

To complete this task as a whole, I would like to do do following:

Agree to follow VFX reference platform in general, but deviate where we think it causes bigger problems than it solves for our users.
Make a list of libraries to upgrade for 2.82, compatible with the reference platform where possible (probably everything except Python, and maybe some newer versions where we remain effectively compatible).
Make a task to do more symbol hiding on Linux and macOS

> In #68774#813749, @ideasman42 wrote: > - Are there any other implications for upgrades/downgrades of any other libraries besides Python? There are other libraries we should upgrade for 2.82, but nothing that would cause problems as far as I can tell. > - It would be good to have feedback from other developers. > - How to come to final decision on this task? > If we're not going with VFX platform for 2.82, I'd like to upgrade to Python 3.8, this should be done earlier in the release cycle (soon). So I'd rather not leave this task in limbo too long. For Python 3.8 the decision is with the Python module owners and the Blender admins. I'd say, communicate as Python module owners that you agree to upgrade to 3.8, and unless any Blender admins object you go ahead. To complete this task as a whole, I would like to do do following: * Agree to follow VFX reference platform in general, but deviate where we think it causes bigger problems than it solves for our users. * Make a list of libraries to upgrade for 2.82, compatible with the reference platform where possible (probably everything except Python, and maybe some newer versions where we remain effectively compatible). * Make a task to do more symbol hiding on Linux and macOS

Brian Savery (AMD) commented

2019-11-20 18:37:20 +01:00

Added subscriber: @BrianSavery

Brian Savery (AMD) commented

2019-11-20 18:37:20 +01:00

Strictly speaking about the Python 3.8 decision, I think this comment hit it on the head.

In #68774#813533, @ideasman42 wrote:
An issue with the replies so far is they don't give any concrete insights, it would be useful if people who work in studios would comment on topics such as.

What C-extensions are studios using which would be important to run inside Blender?

Do these extensions use the stable Python C-API? (which allows 3.7x / 3.8x interoperability).

PyQT is being mentioned (by @rboxman), did anyone successfully use this inside Blender? Wherever I looked into this GUI toolkits tend not to work well with Blender as they need their own main loop.

I don't dispute that someone somewhere would benefit from this proposal, it would just be good to hear from them directly, to get a sense of whats important, what isn't.

The Blender addons that are using c-extensions that I've seen generally fall into one of a few categories:

1 . Do some complex computation that is faster in C
2. Interop with another library
2.b Some library/software that is shared across multiple apps (maya, blender, houdini etc).

I'm pointing most directly at rendering addons here where Arnold, RenderMan, others have a common C-extension that is used in various plugins. They are sticking to the VFX platform spec for that compatibility across apps reason. I added Python 3.X compatibility to the renderman c-extension using the abi-stable method, but not sure that is what they currently do.

Furthermore I would point out that an addon maker who wants to make one with a c-extension currently needs a 3.6 (or is it 3.5) extension for Blender 2.79 and a 3.7 python extension for blender 2.80. This is ok as it's a major version.

So I guess the open question here is can addons that use c-extensions use the abi-stable method and is that acceptable? Like brecht said I'm not sure how many are doing that now.

Strictly speaking about the Python 3.8 decision, I think this comment hit it on the head. > In #68774#813533, @ideasman42 wrote: > An issue with the replies so far is they don't give any concrete insights, it would be useful if people who work in studios would comment on topics such as. > > - What C-extensions are studios using which would be important to run inside Blender? > - Do these extensions use the stable Python C-API? (which allows 3.7x / 3.8x interoperability). > - PyQT is being mentioned (by @rboxman), did anyone successfully use this inside Blender? Wherever I looked into this GUI toolkits tend not to work well with Blender as they need their own main loop. > > I don't dispute that someone somewhere would benefit from this proposal, it would just be good to hear from them directly, to get a sense of whats important, what isn't. > The Blender addons that are using c-extensions that I've seen generally fall into one of a few categories: 1 . Do some complex computation that is faster in C 2. Interop with another library 2.b Some library/software that is shared across multiple apps (maya, blender, houdini etc). I'm pointing most directly at rendering addons here where Arnold, RenderMan, others have a common C-extension that is used in various plugins. They are sticking to the VFX platform spec for that compatibility across apps reason. I added Python 3.X compatibility to the renderman c-extension using the abi-stable method, but not sure that is what they currently do. Furthermore I would point out that an addon maker who wants to make one with a c-extension currently needs a 3.6 (or is it 3.5) extension for Blender 2.79 and a 3.7 python extension for blender 2.80. This is ok as it's a major version. So I guess the open question here is can addons that use c-extensions use the abi-stable method and is that acceptable? Like brecht said I'm not sure how many are doing that now.

Tommi Hyppänen commented

2019-11-25 07:17:29 +01:00

Added subscriber: @TommiHyppanen

Tommi Hyppänen commented

2019-11-25 07:17:29 +01:00

My only issue is really with binary compatibility. Writing and using complex, high performance addons for Blender that require a complete rebuild and testing process for each minor Python version upgrade is a major hassle, and a lot of what feels: pointless work. It's not just the actual compilation, it's also you have to support multiple different binaries on multiple versions that is like pointlessly juggling 10 different balls in the air. An experience which can on the long run stop people trying to do that and lower the quality and capability of the addons available for Blender.

A lot of these issues can be circumvented by using a C API DLL with an excellent library like cffi (not a part of Blender btw). But for very useful C++/Python interop libraries like pybind11 is creating a stable Python ABI possible? Using C++ code as imported Python modules would after all be optimal. Pybind11 makes handling of Numpy arrays from Python and Eigen in C++ really comfortable.

I estimate that the core question here is the following: the convenience of seamless programming experience interfacing Blender Python API with C++, or a version bump in the Blender Python version? I personally don't understand the concrete benefits of a Python version bump from 3.7 to 3.8.

edit: It seems one of the most critical and useful components of transferring data between Blender Python API and C++ is disabled in the stable ABI:
https://www.python.org/dev/peps/pep-0384/#the-buffer-interface
https://pybind11.readthedocs.io/en/stable/advanced/pycpp/numpy.html#buffer-protocol

My only issue is really with binary compatibility. Writing and using complex, high performance addons for Blender that require a complete rebuild and testing process for each minor Python version upgrade is a major hassle, and a lot of what feels: pointless work. It's not just the actual compilation, it's also you have to support multiple different binaries on multiple versions that is like pointlessly juggling 10 different balls in the air. An experience which can on the long run stop people trying to do that and lower the quality and capability of the addons available for Blender. A lot of these issues can be circumvented by using a C API DLL with an excellent library like cffi (not a part of Blender btw). But for very useful C++/Python interop libraries like pybind11 is creating a stable Python ABI possible? Using C++ code as imported Python modules would after all be optimal. Pybind11 makes handling of Numpy arrays from Python and Eigen in C++ really comfortable. I estimate that the core question here is the following: the convenience of seamless programming experience interfacing Blender Python API with C++, or a version bump in the Blender Python version? I personally don't understand the concrete benefits of a Python version bump from 3.7 to 3.8. edit: It seems one of the most critical and useful components of transferring data between Blender Python API and C++ is disabled in the stable ABI: https://www.python.org/dev/peps/pep-0384/#the-buffer-interface https://pybind11.readthedocs.io/en/stable/advanced/pycpp/numpy.html#buffer-protocol

Bastien Montagne commented

2019-11-25 14:45:12 +01:00

Added subscriber: @mont29

Bastien Montagne commented

2019-11-25 14:45:12 +01:00

I’m all for switching to python 3.8 too.

I’m not fully against the VFX initiative, but for me it still has to prove it is a serious thing (and sorry, but sticking to python 2.7 for years and years after py3 had become a much better option is by no mean showing seriousness - laziness of people not wanting to port their code to py3 is a very bad reason for a “standard” choice).

So am all for following the VFX specs - as long as it does not keep us back from better solutions. And as a reminder, we do not create Blender only for studios who are stuck in some heavy non-flexible environments. "Agile" users wanting to use recent techs are also a valid target. Otherwise we could also do what 'professional' software do regarding hardware requirements, e.g. only officially supporting a few multi-€K GPUs and one or two driver versions… ;)

I’m all for switching to python 3.8 too. I’m not fully against the VFX initiative, but for me it still has to prove it is a serious thing (and sorry, but sticking to python 2.7 for years and years after py3 had become a much better option is by no mean showing seriousness - laziness of people not wanting to port their code to py3 is a **very** bad reason for a “standard” choice). So am all for following the VFX specs - as long as it does not keep us back from better solutions. And as a reminder, we do not create Blender **only** for studios who are stuck in some heavy non-flexible environments. "Agile" users wanting to use recent techs are also a valid target. Otherwise we could also do what 'professional' software do regarding hardware requirements, e.g. only officially supporting a few multi-€K GPUs and one or two driver versions… ;)

Andrea Monzini commented

2019-11-25 17:28:06 +01:00

Added subscriber: @AndreaMonzini

Stefan Werner commented

2019-11-25 20:36:49 +01:00

Added subscriber: @Stefan_Werner

Pipeliner commented

2019-11-25 20:41:33 +01:00

Added subscriber: @Pipeliner

lvxejay commented

2019-11-26 00:57:54 +01:00

I’m not fully against the VFX initiative, but for me it still has to prove it is a serious thing (and sorry, but sticking to python 2.7 for years and years after py3 had become a much better option is by no mean showing seriousness - laziness of people not wanting to port their code to py3 is a very bad reason for a “standard” choice).

I think there is a common misconception that sticking to Python 2.7 for so long was due to "laziness". This is simply not the case. Even mid-size studio pipelines are millions of lines of Python code. It's common to see codebases with few tests and very little documentation at studios where the original author's no longer work there. Upgrading monolithic pipelines with these characteristics is not easy, nor is it quick work. Also, studios are beholden to the software vendors, and as such, without Python 3.7 APIs there could be no upgrade. Thankfully, the industry is trying to uphold some standards and move forward with the Python migration, and I'd be willing to bet this was one of the motivating factors behind the VFX Reference Platform's creation. Regardless, while both studios and vendors can share the blame, the sluggishness in getting to this point should be correctly attributed to real fiscal, and technical, business costs.

As @TommiHyppanen mentioned, there is no buffer protocol support in the Py_LIMITED_API. This is a deal-breaker for plugins that are leveraging the buffer protocol (it's a pretty common use case). So that's most likely a good reason why there isn't much evidence of people using the stable ABI.

> I’m not fully against the VFX initiative, but for me it still has to prove it is a serious thing (and sorry, but sticking to python 2.7 for years and years after py3 had become a much better option is by no mean showing seriousness - laziness of people not wanting to port their code to py3 is a very bad reason for a “standard” choice). I think there is a common misconception that sticking to Python 2.7 for so long was due to "laziness". This is simply not the case. Even mid-size studio pipelines are millions of lines of Python code. It's common to see codebases with few tests and very little documentation at studios where the original author's no longer work there. Upgrading monolithic pipelines with these characteristics is not easy, nor is it quick work. Also, studios are beholden to the software vendors, and as such, without Python 3.7 APIs there could be no upgrade. Thankfully, the industry is trying to uphold some standards and move forward with the Python migration, and I'd be willing to bet this was one of the motivating factors behind the VFX Reference Platform's creation. Regardless, while both studios and vendors can share the blame, the sluggishness in getting to this point should be correctly attributed to *real fiscal, and technical, business costs*. As @TommiHyppanen mentioned, there is no buffer protocol support in the Py_LIMITED_API. This is a deal-breaker for plugins that are leveraging the buffer protocol (it's a pretty common use case). So that's most likely a good reason why there isn't much evidence of people using the stable ABI.

Aditia A. Pratama commented

2019-11-26 01:00:18 +01:00

Added subscriber: @AditiaA.Pratama

Iemand Iets commented

2019-11-26 06:03:47 +01:00

Added subscriber: @NumesSanguis-3

Iemand Iets commented

2019-11-26 06:03:47 +01:00

It's common to see codebases with few tests and very little documentation at studios where the original author's no longer work there. Upgrading monolithic pipelines with these characteristics is not easy, nor is it quick work.

Monolithic pipelines are indeed a struggle to update. Currently, the disadvantages of that are widely known and the modern practice is therefore to build modular / microservices pipelines. Now companies are upgrading to Python 3.7, if they still hold-on to the monolithic model approach instead of modernizing, that's just bad business practice that introduces technical depth right from the start. Blender should not support such short-sighted decision processes.
Also, with various industries moving to Docker, Kubernetes and the likes, creating environments with different version of libraries is easier than ever.

Blender has created so much passion in programmers exactly because it was interesting for the smaller hobbyist / semi-professional to use the latest technology, those people who have produced a lot of open source tools out of passion. Programmers reading up on the latest improvements, consider how those could be used to do some improvement, and directly go for it. That fire does not last a 1-year wait.

It would also be good to consider what Blender is already going to miss by not upgrading to 3.8 (Advanced JSON parsing, faster CSV parsing, multi-core pickle, simplified asyncio, continue working in finally again, multiprocessing.shared_memory for faster data sharing across processes (which would maybe help pipeline building), etc): https://docs.python.org/3/whatsnew/3.8.html

Even more relating to performance, sticking to outdated version will already run into holding back Blender with version 3.9. In that version Vectorcall will be stable (https://www.python.org/dev/peps/pep-0590/), a fast calling protocol for CPython. Since Blender relies a lot on these calls (I assume), waiting for this release will block Blender from enjoying faster processing.

> It's common to see codebases with few tests and very little documentation at studios where the original author's no longer work there. Upgrading monolithic pipelines with these characteristics is not easy, nor is it quick work. Monolithic pipelines are indeed a struggle to update. Currently, the disadvantages of that are widely known and the modern practice is therefore to build modular / microservices pipelines. Now companies are upgrading to Python 3.7, if they still hold-on to the monolithic model approach instead of modernizing, that's just bad business practice that introduces technical depth right from the start. Blender should not support such short-sighted decision processes. Also, with various industries moving to Docker, Kubernetes and the likes, creating environments with different version of libraries is easier than ever. Blender has created so much passion in programmers exactly because it was interesting for the smaller hobbyist / semi-professional to use the latest technology, those people who have produced a lot of open source tools out of passion. Programmers reading up on the latest improvements, consider how those could be used to do some improvement, and directly go for it. That fire does not last a 1-year wait. It would also be good to consider what Blender is already going to miss by not upgrading to 3.8 (Advanced JSON parsing, faster CSV parsing, multi-core pickle, simplified asyncio, `continue` working in `finally` again, `multiprocessing.shared_memory` for faster data sharing across processes (which would maybe help pipeline building), etc): https://docs.python.org/3/whatsnew/3.8.html Even more relating to performance, sticking to outdated version will already run into holding back Blender with version 3.9. In that version `Vectorcall` will be stable (https://www.python.org/dev/peps/pep-0590/), a fast calling protocol for CPython. Since Blender relies a lot on these calls (I assume), waiting for this release will block Blender from enjoying faster processing.

Laurent Noel commented

2019-11-26 10:46:16 +01:00

Added subscriber: @c2ba

Nick Cannon commented

2019-11-26 18:21:12 +01:00

Added subscriber: @ncannon

Nick Cannon commented

2019-11-26 18:21:12 +01:00

As a member of the VFX Reference Platform Working Group I can provide some perspective on the intent of the Platform and how it is organized. Brecht and Jared are already doing a good job here of understanding and accurately articulating some of the trade-offs involved.

There is more information at https:*vfxplatform.com/about and I'll repeat the goals of the Platform here:

Align software providers to eliminate incompatibilities between common dependencies of DCC software.
Reduce complexity and effort required to support Linux in VFX and Animation.
Lower the barrier to adopting Linux as a supported platform both for software providers and end-user studios.
Create a transparent process, fairly representing the interests of both software vendors and end-user studios.

This is very much a community-driven process, feedback is solicited from the community at each stage of developing and deciding on the Platform each year.

The Platform provides a common target for alignment, it's not a standard and nothing is mandated. The common target enables better compatibility between digital content creation tools which is particularly important for studios to create tightly integrated pipelines.

The Platform makes it easier for studios to adopt interoperable toolsets from different providers and its focus is more to help push providers forward in a consistent manner than to hold them back.

Ultimately the selection of versions is a balance between providing the latest features and providing a window of consistency to ease interoperability and adoption by studios.

For anyone wanting to stay in touch with developments, VFX Reference Platform news and the opportunity to provide feedback is available through the [vfx-platform-discuss Google Group ]] or [ https:*twitter.com/vfxplatform | @vfxplatform .

As a member of the VFX Reference Platform Working Group I can provide some perspective on the intent of the Platform and how it is organized. Brecht and Jared are already doing a good job here of understanding and accurately articulating some of the trade-offs involved. There is more information at [https:*vfxplatform.com/about](https:*vfxplatform.com/about) and I'll repeat the goals of the Platform here: - Align software providers to eliminate incompatibilities between common dependencies of DCC software. - Reduce complexity and effort required to support Linux in VFX and Animation. - Lower the barrier to adopting Linux as a supported platform both for software providers and end-user studios. - Create a transparent process, fairly representing the interests of both software vendors and end-user studios. This is very much a community-driven process, feedback is solicited from the community at each stage of developing and deciding on the Platform each year. The Platform provides a common target for alignment, it's not a standard and nothing is mandated. The common target enables better compatibility between digital content creation tools which is particularly important for studios to create tightly integrated pipelines. The Platform makes it easier for studios to adopt interoperable toolsets from different providers and its focus is more to help push providers forward in a consistent manner than to hold them back. Ultimately the selection of versions is a balance between providing the latest features and providing a window of consistency to ease interoperability and adoption by studios. For anyone wanting to stay in touch with developments, VFX Reference Platform news and the opportunity to provide feedback is available through the [vfx-platform-discuss Google Group ]] or [[ https:*twitter.com/vfxplatform | @vfxplatform ](https:*groups.google.com/forum/#!forum/vfx-platform-discuss).

Paul Szajner commented

2019-11-26 22:08:28 +01:00

Added subscriber: @sozap

Paul Szajner commented

2019-11-26 22:08:28 +01:00

One issue I had in a company was that they were working with shotgun and it had a python 2.7 API where blender would be 3.* .
I wanted to integrate shotgun into blender so they can communicate to get assets , or to set status on some tasks. I end up with doing something really limited by running python 2.7 in background to talk to shotgun with os module , then import these data into blender. I guess if 3.7 is compatible with 3.8 there won't be such issue.
It's also totally arguable that Shotgun could come with an API compatible with python 2.7 and 3.*. I guess as all the other software stayed with 2.7 , they haven't see the need to do so until 2.7 gets abandoned.

What studios are doing also is that they build an interface to their asset management system that they can integrate seamlessly into Maya , Nuke , Houdini... with an embedded QT interface.
So they need as much compatibility as possible and they try to do most of the work one time. One example of that : https://www.kabaretstudio.com/gallery/gallery_embedding
Blender isn't really designed with that in mind, and I guess it's an issue that face big studios that try to implement blender into their pipeline, I guess it's a long road to take to make blender work like other DCC in that regard.

Basically they do the same with C/C++ applications, they do some plugins that can run in different applications. But I guess this won't be an issue with blender as you can't build some C/C++ plugins for it, at least in the same way you can do for others DCC.
All that said, I'm not very used to these workflows so I can't elaborate on the details.

One issue I had in a company was that they were working with shotgun and it had a python 2.7 API where blender would be 3.* . I wanted to integrate shotgun into blender so they can communicate to get assets , or to set status on some tasks. I end up with doing something really limited by running python 2.7 in background to talk to shotgun with os module , then import these data into blender. I guess if 3.7 is compatible with 3.8 there won't be such issue. It's also totally arguable that Shotgun could come with an API compatible with python 2.7 and 3.*. I guess as all the other software stayed with 2.7 , they haven't see the need to do so until 2.7 gets abandoned. What studios are doing also is that they build an interface to their asset management system that they can integrate seamlessly into Maya , Nuke , Houdini... with an embedded QT interface. So they need as much compatibility as possible and they try to do most of the work one time. One example of that : https://www.kabaretstudio.com/gallery/gallery_embedding Blender isn't really designed with that in mind, and I guess it's an issue that face big studios that try to implement blender into their pipeline, I guess it's a long road to take to make blender work like other DCC in that regard. Basically they do the same with C/C++ applications, they do some plugins that can run in different applications. But I guess this won't be an issue with blender as you can't build some C/C++ plugins for it, at least in the same way you can do for others DCC. All that said, I'm not very used to these workflows so I can't elaborate on the details.

JT Nelson commented

2019-11-26 23:23:50 +01:00

Added subscriber: @jta

Evan Wilson commented

2019-11-27 05:55:15 +01:00

Added subscriber: @EAW

Campbell Barton commented

2019-11-27 12:53:30 +01:00

Mailed the vfx-platform-discuss regarding Python versions in the future:

It looks like the VFX platform will remain 1-2 years behind the the latest stable Python.
As the VFX platform uses Python/QT library (PySide2 ), updated versions of this library need to be available before the VFX platform would consider updating.

See thread .

Mailed the vfx-platform-discuss regarding Python versions in the future: - It looks like the VFX platform will remain 1-2 years behind the the latest stable Python. - As the VFX platform uses Python/QT library ([PySide2 ](https://pypi.org/project/PySide2/)), updated versions of this library need to be available before the VFX platform would consider updating. [See thread ](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/vfx-platform-discuss/8qRBxE_iC6A).

Jasper van Nieuwenhuizen commented

2019-11-27 22:17:27 +01:00

Added subscriber: @jasperge-2

Janne Aliu commented

2019-11-27 23:32:29 +01:00

Added subscriber: @Jaydead

Andrew Silke commented

2019-11-28 00:49:57 +01:00

Added subscriber: @andrewsilke

Andrew Silke commented

2019-11-28 00:49:57 +01:00

Agreed with Qt/PySide. We are a small company heavily invested in PySide UIs and 3d Tools. We'd love to be able to port tools to Blender. Seems like we can patch PySide, but a more thorough integration would be fantastic and a huge benefit to the industry.

Frieder Erdmann commented

2019-11-29 09:04:42 +01:00

Added subscriber: @Frieder

Brecht Van Lommel commented

2020-01-15 13:51:10 +01:00

The decision was made to stick to Python 3.7 and the VFX platform in general in 2020, see mail by Ton here:
https://lists.blender.org/pipermail/bf-committers/2020-January/050410.html

To give studios enough time and confidence to check out on Blender, I propose to respect the VFX Platform versions for the entire year of 2020. That implies we will be very conservative with upgrading libraries, for example Python will stick to 3.7 this year for official releases.

I've checked it with the core team and administrators, and they're OK - provided this won't hold back essential improvements for our users.

The decision was made to stick to Python 3.7 and the VFX platform in general in 2020, see mail by Ton here: https://lists.blender.org/pipermail/bf-committers/2020-January/050410.html >To give studios enough time and confidence to check out on Blender, I propose to respect the VFX Platform versions for the entire year of 2020. That implies we will be very conservative with upgrading libraries, for example Python will stick to 3.7 this year for official releases. > >I've checked it with the core team and administrators, and they're OK - provided this won't hold back essential improvements for our users.

Brecht Van Lommel changed title from ~~Define VFX Reference Platform Policy~~ to VFX Reference Platform 2020 Compatibility

2020-01-15 15:42:36 +01:00

Brecht Van Lommel commented

2020-01-15 18:39:41 +01:00

For 2.82 I propose to do the following, to conform to the VFX platform:

Boost: upgrade to 1.70
OpenVDB: upgrade to 7.0.0
OpenEXR: upgrade to 2.4.0
TBB: stick to 2019_U9 (it is ABI compatible with U6)
OpenSubdiv: stick to 3.4.0 RC2 (it is identical to 3.4.0)
Python: stick to 3.7.4

Work in progress patch in D6593: Build: upgrade to OpenEXR 2.4.0, OpenVDB 7.0.0 and Boost 1.70.0.

There are many other libraries we could upgrade as well that are not part of the VFX platform, but easiest to leave that as a second step. That could be before or after 2.82 depending if there is time.

For 2.82 I propose to do the following, to conform to the VFX platform: * Boost: upgrade to 1.70 * OpenVDB: upgrade to 7.0.0 * OpenEXR: upgrade to 2.4.0 * TBB: stick to 2019_U9 (it is ABI compatible with U6) * OpenSubdiv: stick to 3.4.0 RC2 (it is identical to 3.4.0) * Python: stick to 3.7.4 Work in progress patch in [D6593: Build: upgrade to OpenEXR 2.4.0, OpenVDB 7.0.0 and Boost 1.70.0](https://archive.blender.org/developer/D6593). There are many other libraries we could upgrade as well that are not part of the VFX platform, but easiest to leave that as a second step. That could be before or after 2.82 depending if there is time.

Francesco Sorrentino commented

2020-01-16 17:54:18 +01:00

Added subscriber: @Zeirus

Brecht Van Lommel self-assigned this 2020-01-18 14:28:26 +01:00

Charles Wardlaw commented

2020-01-21 14:30:16 +01:00

Added subscriber: @CharlesWardlaw

Charles Wardlaw commented

2020-01-21 14:30:16 +01:00

PyQT is being mentioned (by @rboxman), did anyone successfully use this inside Blender? Wherever I looked into this GUI toolkits tend not to work well with Blender as they need their own main loop.

PyQt has been used for many tools within Tangent since Next Gen. The Blender UI toolkit is too limiting for certain kinds of interfaces that PyQt makes trivial.

The main loop of PyQt was fixed specifically to allow for this kind of foreign embedding many, many versions ago and requires very little code to get going. The main issues tends to be around uncaught exceptions and some threading shenanigans but it’s very doable.

Speaking personally and NOT for the company, I’d prefer to move to 3.8 because of the new features but there are other considerations. If Blender’s python version matches the VFX platform, then third party creators can do a single build of python bindings. That by itself is huge— no need to call a company and request a custom build to integrate tools even if the workflow is unsupported.

> - PyQT is being mentioned (by @rboxman), did anyone successfully use this inside Blender? Wherever I looked into this GUI toolkits tend not to work well with Blender as they need their own main loop. PyQt has been used for many tools within Tangent since Next Gen. The Blender UI toolkit is too limiting for certain kinds of interfaces that PyQt makes trivial. The main loop of PyQt was fixed specifically to allow for this kind of foreign embedding many, many versions ago and requires very little code to get going. The main issues tends to be around uncaught exceptions and some threading shenanigans but it’s very doable. Speaking personally and NOT for the company, I’d prefer to move to 3.8 because of the new features but there are other considerations. If Blender’s python version matches the VFX platform, then third party creators can do a single build of python bindings. That by itself is huge— no need to call a company and request a custom build to integrate tools even if the workflow is unsupported.

Brian Savery (AMD) commented

2020-01-21 17:02:40 +01:00

PyQt has been used for many tools within Tangent since Next Gen. The Blender UI toolkit is too limiting for certain kinds of interfaces that PyQt makes trivial.
The main loop of PyQt was fixed specifically to allow for this kind of foreign embedding many, many versions ago and requires very little code to get going. The main issues tends to be around uncaught exceptions and some threading shenanigans but it’s very doable.

@CharlesWardlaw Charles: I'd like to start a side thread on pyqt running in Blender. I know of a few people interesting in doing something similar and struggling. We're not currently using that but am interested in seeing how this works. Do you mind starting a thread on https://devtalk.blender.org/c/other-topics/python/6 on how this basically works? Sounds cool.

> PyQt has been used for many tools within Tangent since Next Gen. The Blender UI toolkit is too limiting for certain kinds of interfaces that PyQt makes trivial. > The main loop of PyQt was fixed specifically to allow for this kind of foreign embedding many, many versions ago and requires very little code to get going. The main issues tends to be around uncaught exceptions and some threading shenanigans but it’s very doable. @CharlesWardlaw Charles: I'd like to start a side thread on pyqt running in Blender. I know of a few people interesting in doing something similar and struggling. We're not currently using that but am interested in seeing how this works. Do you mind starting a thread on https://devtalk.blender.org/c/other-topics/python/6 on how this basically works? Sounds cool.

Bob White commented

2020-01-21 17:53:39 +01:00

Added subscriber: @bob.w

Bob White commented

2020-01-21 17:53:39 +01:00

In #68774#854878, @CharlesWardlaw wrote:

PyQT is being mentioned (by @rboxman), did anyone successfully use this inside Blender? Wherever I looked into this GUI toolkits tend not to work well with Blender as they need their own main loop.

The main loop of PyQt was fixed specifically to allow for this kind of foreign embedding many, many versions ago and requires very little code to get going. The main issues tends to be around uncaught exceptions and some threading shenanigans but it’s very doable.

https://devtalk.blender.org/t/a-pyside2-wrapper-for-blender-2-8x/11405

Just want to share this link.
Basically bqt.py is a simple addon that integrates the QApplication loop with blender. Though we're currently targeting PySide2, the goal is to abstract that so that users can pick between PySide2 and PyQt5.

> In #68774#854878, @CharlesWardlaw wrote: >> - PyQT is being mentioned (by @rboxman), did anyone successfully use this inside Blender? Wherever I looked into this GUI toolkits tend not to work well with Blender as they need their own main loop. > The main loop of PyQt was fixed specifically to allow for this kind of foreign embedding many, many versions ago and requires very little code to get going. The main issues tends to be around uncaught exceptions and some threading shenanigans but it’s very doable. > https://devtalk.blender.org/t/a-pyside2-wrapper-for-blender-2-8x/11405 Just want to share this link. Basically `bqt.py` is a simple addon that integrates the `QApplication` loop with blender. Though we're currently targeting PySide2, the goal is to abstract that so that users can pick between PySide2 and PyQt5.

Brecht Van Lommel commented

2020-01-27 09:59:33 +01:00

Changed status from 'Confirmed' to: 'Resolved'

Brecht Van Lommel closed this issue

2020-01-27 09:59:33 +01:00

Brecht Van Lommel commented

2020-01-27 09:59:33 +01:00

Precompiled libraries for all platforms have now been updated, for the 2.82 release.

Thomas Dinges added this to the 2.82 milestone 2023-02-08 16:43:10 +01:00

Sign in to join this conversation.