C++ Version Upgrade #76783

New Issue

Jacques Lucke · 2020-05-15T14:12:38+02:00

Jacques Lucke commented

2020-05-15 14:12:38 +02:00

Blender uses C++11 currently. As we start using more C++ in Blender, we might want to move to a newer C++ standard eventually. I often find myself needing functionality from newer versions of C++. For example: std::make_unique, std::optional, structured bindings and support for statements in constexpr functions.

Background information:

We can upgrade to C++14 right away. For C++17 we will also need to upgrade compilers and fix some build errors.

Steps to upgrade:

- Change build flags to -std=c++14 and /std:c++14 throughout Blender code and fix any build errors (optional)
- Upgrade Linux CentOS buildbot environment to devtoolset-9 (and update buildbot_utils.py)
- Upgrade Windows buildbot VS 2019 to latest version
- Change build flags to -std=c++17 and /std:c++17 throughout Blender code and fix any build errors

Blender uses C++11 currently. As we start using more C++ in Blender, we might want to move to a newer C++ standard eventually. I often find myself needing functionality from newer versions of C++. For example: `std::make_unique`, `std::optional`, structured bindings and support for statements in `constexpr` functions. Background information: * [C++ compiler support ](https://en.cppreference.com/w/cpp/compiler_support) * [Usage examples for C++ features ](https://github.com/AnthonyCalandra/modern-cpp-features) * [Changes between C++11 and C++14 ](http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1319r0.html) * [Changes between C++14 and C++17 ](https://isocpp.org/files/papers/p0636r0.html) * [Changes between C++17 and C++20 ](http://open-std.org/JTC1/SC22/WG21/docs/papers/2020/p2131r0.html) * [VFX Reference Platform ](https://vfxplatform.com/) We can upgrade to C++14 right away. For C++17 we will also need to upgrade compilers and fix some build errors. Steps to upgrade: * - [x] Change build flags to `-std=c++14` and `/std:c++14` throughout Blender code and fix any build errors (optional) * - [x] Upgrade Linux CentOS buildbot environment to devtoolset-9 (and update `buildbot_utils.py`) * - [x] Upgrade Windows buildbot VS 2019 to latest version * - [x] Change build flags to `-std=c++17` and `/std:c++17` throughout Blender code and fix any build errors

Jacques Lucke commented

2020-05-15 14:12:38 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Jacques Lucke commented

2020-05-15 14:12:38 +02:00

Added subscribers: @JacquesLucke, @brecht, @LazyDodo, @Sergey

Sergey Sharybin commented

2020-05-15 14:37:35 +02:00

Personally not so much finding anything useful in C++14. C++17 or C++20 will be sooooooooooo much more better ;)

The VFX platforms technically specifies C++14. Going higher for Blender shouldn't be a problem since we don't have binary plugins, and dependencies we can compile with C++11/C++14.

For C++17 the current GCC version 6 is problematic: it has experimental C++17 support, and this is something I wouldn't personally rely my life on. Would prefer to go C++17 when we can use GCC which has it as official feature.

For Windows and macOS I don't see problem with current compilers. MSVC 2017 seems to have C++14 support, same for Clang on mac. (We should update to latest MSVC and XCode which will open doors for C++17, but this is an intrusive change, there is never good time for doing such upgrade).

All in all I do not see any stoppers for us to go C++14. For C++17 and higher I kind of would prefer to have VFX platform to advertise GCC which has it as an official feature set.

Personally not so much finding anything useful in C++14. C++17 or C++20 will be sooooooooooo much more better ;) The VFX platforms technically specifies C++14. Going higher for Blender shouldn't be a problem since we don't have binary plugins, and dependencies we can compile with C++11/C++14. For C++17 the current GCC version 6 is problematic: it has experimental C++17 support, and this is something I wouldn't personally rely my life on. Would prefer to go C++17 when we can use GCC which has it as official feature. For Windows and macOS I don't see problem with current compilers. MSVC 2017 seems to have C++14 support, same for Clang on mac. (We should update to latest MSVC and XCode which will open doors for C++17, but this is an intrusive change, there is never good time for doing such upgrade). All in all I do not see any stoppers for us to go C++14. For C++17 and higher I kind of would prefer to have VFX platform to advertise GCC which has it as an official feature set.

Julian Eisel commented

2020-05-15 14:56:13 +02:00

Added subscriber: @JulianEisel

Julian Eisel commented

2020-05-15 14:56:13 +02:00

Edit: Of course Sergey was faster :)

The OpenXR SDK team is considering to switch to C++14, even the public API, which made me investigate this topic for Blender too. I've done some experiments with different Linux VMs, and from what I can tell C++14 should be fine for the current supported platforms. CentOS seems to be the one limiting us the most. But even there, GCC 6.3.1 is available, which should be fine for 14. The VFX Platform is on C++14 since CY2018, which confirms that.

CentOS also seems to be what prevents us from going to C++17, which is a real shame. Many 17 features are only supported since GCC 7, which doesn't seem to be available easily in CentOS.
IIRC for MSVC, C++17 would be fine too if we bump the minimum version. Didn't check Apple Clang yet.

Edit: Of course Sergey was faster :) The OpenXR SDK team is considering to switch to C++14, even the public API, which made me investigate this topic for Blender too. I've done some experiments with different Linux VMs, and from what I can tell C++14 should be fine for the current supported platforms. CentOS seems to be the one limiting us the most. But even there, GCC 6.3.1 is available, which should be fine for 14. The [VFX Platform ](https://vfxplatform.com/) is on C++14 since CY2018, which confirms that. CentOS also seems to be what prevents us from going to C++17, which is a real shame. Many 17 features are only supported since GCC 7, which doesn't seem to be available easily in CentOS. IIRC for MSVC, C++17 would be fine too if we bump the minimum version. Didn't check Apple Clang yet.

Brecht Van Lommel commented

2020-05-15 15:54:48 +02:00

My reading of the VFX platform is that using newer GCC versions and upgrading to devtoolset-9 is fine for us. We are providing the libraries / program (including statically linking libstdc++). So GCC 6.3 is the minimum version required, not the maximum allowed.

GCC guarantees ABI compatibility between the different supported C++ versions, assuming we continue to use _GLIBCXX_USE_CXX11_ABI=0.
https://stackoverflow.com/a/49119902

C++20 support is still experimental which means there will be no ABI backwards compatibility. But I see no compatibility problem in building Blender itself with C++20, if that's what we want to do.

As far as I know GCC 9 is easily available in CentOS through devtoolset-9. In fact as the VFX platform page says, devtoolset-6 and so GCC 6 is no longer actively supported on CentOS.

My reading of the VFX platform is that using newer GCC versions and upgrading to devtoolset-9 is fine for us. We are providing the libraries / program (including statically linking libstdc++). So GCC 6.3 is the minimum version required, not the maximum allowed. GCC guarantees ABI compatibility between the different supported C++ versions, assuming we continue to use `_GLIBCXX_USE_CXX11_ABI=0`. https://stackoverflow.com/a/49119902 C++20 support is still experimental which means there will be no ABI backwards compatibility. But I see no compatibility problem in building Blender itself with C++20, if that's what we want to do. As far as I know GCC 9 is easily available in CentOS through devtoolset-9. In fact as the VFX platform page says, devtoolset-6 and so GCC 6 is no longer actively supported on CentOS.

Jacques Lucke commented

2020-05-17 15:42:38 +02:00

Thanks for your input. It seems we could make the upgrade to C++17 soonish. I'm still not sure what has to be done on the Blender side.
My guess is that it is better to switch to C++17 first, since it should be a "stable" platform. Afterwards, we can check if C++20 is still too experimental or not.

Thanks for your input. It seems we could make the upgrade to C++17 soonish. I'm still not sure what has to be done on the Blender side. My guess is that it is better to switch to C++17 first, since it should be a "stable" platform. Afterwards, we can check if C++20 is still too experimental or not.

Ray molenkamp commented

2020-05-17 18:39:02 +02:00

From a quick look with msvc with C++17, it's a few compiler switches that need to change.

Other issues

the use of binary_function in source\blender\freestyle\intern\view_map\ViewMapBuilder.cpp needs to be dealt with since it was removed in C++17
random_shuffle in extern_ceres : fixed upstream #373
unary_function in bf_intern_quadriflow

From a quick look with msvc with C++17, it's a few compiler switches that need to change. Other issues - the use of `binary_function` in `source\blender\freestyle\intern\view_map\ViewMapBuilder.cpp` needs to be dealt with since it was removed in C++17 - `random_shuffle` in extern_ceres : fixed upstream [#373 ](https://github.com/ceres-solver/ceres-solver/issues/373) - `unary_function` in bf_intern_quadriflow

Brecht Van Lommel commented

2020-05-22 22:54:54 +02:00

I've updated the description with more concrete steps.

@Sergey would need to install devtoolset-9 on the Centos 7 buildbot machine (which I think can be done side-by-side with devtoolset-6, in buildbot_utils.py the version is specified). And then any developer could make a branch and test that it builds on all platforms with the buildbot.

I don't see any other roadblock.

I've updated the description with more concrete steps. @Sergey would need to install devtoolset-9 on the Centos 7 buildbot machine (which I think can be done side-by-side with devtoolset-6, in `buildbot_utils.py` the version is specified). And then any developer could make a branch and test that it builds on all platforms with the buildbot. I don't see any other roadblock.

Ray molenkamp commented

2020-05-23 00:50:42 +02:00

I wouldn't mind a bump to 2019 (atleast 16.4) on the windows bot as well, since there are a few c++17 defects not fixed until that version.

I wouldn't mind a bump to 2019 (atleast 16.4) on the windows bot as well, since there are [a few c++17 defects ](https://docs.microsoft.com/en-us/cpp/overview/visual-cpp-language-conformance?view=vs-2019) not fixed until that version.

Sergey Sharybin commented

2020-05-26 07:56:10 +02:00

@brecht shouldn't be too hard, but what is the time frame for updates? It doesn't feel right to start poking release environment a week prior to 2.83 release. Would either wait a week, or I suggest we create a new VM for 2.90.

Edit: Creating new VM might actually be good idea. It would be inevitable anyway due to LTS.

@brecht shouldn't be too hard, but what is the time frame for updates? It doesn't feel right to start poking release environment a week prior to 2.83 release. Would either wait a week, or I suggest we create a new VM for 2.90. Edit: Creating new VM might actually be good idea. It would be inevitable anyway due to LTS.

Brecht Van Lommel commented

2020-05-27 16:27:05 +02:00

This can be done after the 2.83 release.

About a separate VM, as far as I know there is no problem installing multiple devtoolsets side-by-side, and the same is true for multiple buildbot builders. So I'm not sure why there is a need for a separate VM and one seems easier to maintain than multiple. But either is fine and you know more about this than me.

This can be done after the 2.83 release. About a separate VM, as far as I know there is no problem installing multiple devtoolsets side-by-side, and the same is true for multiple buildbot builders. So I'm not sure why there is a need for a separate VM and one seems easier to maintain than multiple. But either is fine and you know more about this than me.

blender-admin commented

2020-06-12 16:05:36 +02:00

This issue was referenced by 9f5cc128d9

This issue was referenced by 9f5cc128d9551a84bcceebdf005988f9eb095a36

Sergey Sharybin commented

2020-06-12 16:46:03 +02:00

Installed devtoolset-9 and set master branch to use it.
There seems to be a codegen bug in gcc-9.1.1 used by devtoolset-9: cycles_util_avxf_avx_test and cycles_util_avxf_avx2_test are failing. Can not reproduce locally using gcc-9.3.0.

Installed `devtoolset-9` and set `master` branch to use it. There seems to be a codegen bug in `gcc-9.1.1` used by `devtoolset-9`: `cycles_util_avxf_avx_test` and `cycles_util_avxf_avx2_test` are failing. Can not reproduce locally using `gcc-9.3.0`.

Steffen Dünner commented

2020-06-17 18:19:43 +02:00

Added subscriber: @SteffenD

T.R.O. Nunes commented

2020-06-17 22:33:37 +02:00

Added subscriber: @T.R.O.Nunes

Sergey Sharybin commented

2020-06-18 09:43:53 +02:00

All the builders have been updated to the latest toolchains (Windows is MSVC 2019, CentOS is devtoolset-9 9.3.1, macOS is apple clang 11.0.3). The devtoolset-9 is actually from testing repository: had to use it because of codegen bug in gcc 8.2-9.1. This fixed the avx unit tests failures mentioned above.

blender-admin commented

2020-06-18 11:01:52 +02:00

This issue was referenced by e9a7f4d389

This issue was referenced by e9a7f4d389047cfecc0b3150fbf406f65cf5117d

Ray molenkamp commented

2020-06-19 22:39:43 +02:00

The upgrade caused a small cycles regression on windows, logged #78047 for tracking purposes.

Jagannadhan Ravi commented

2020-06-20 02:26:05 +02:00

Added subscriber: @easythrees

Jagannadhan Ravi commented

2020-06-20 02:26:05 +02:00

Hi there, I'm guessing I need to re-run make to re-generate all the VS project files for this change to take effect (after I sync I mean)?

Ray molenkamp commented

2020-06-20 03:08:42 +02:00

The regeneration should happen automatically nothing special needed.

Brecht Van Lommel commented

2020-07-02 12:28:17 +02:00

Changed status from 'Confirmed' to: 'Resolved'

Brecht Van Lommel closed this issue

2020-07-02 12:28:17 +02:00

Brecht Van Lommel self-assigned this 2020-07-02 12:28:17 +02:00

Brecht Van Lommel commented

2020-07-02 12:28:17 +02:00

After discussion with Sergey, we decide not to upgrade the external libraries to C++14, and instead go directly to C++17 in about 6 months from now when we upgrade to VFX reference platform 2021.

The Windows libraries have already been committed, and I can't think of any real benefit to upgrading precompiled libraries from C++11 to C++14.

So, considering this task as resolved.

After discussion with Sergey, we decide not to upgrade the external libraries to C++14, and instead go directly to C++17 in about 6 months from now when we upgrade to VFX reference platform 2021. The Windows libraries have already been committed, and I can't think of any real benefit to upgrading precompiled libraries from C++11 to C++14. So, considering this task as resolved.

blender-admin commented

2020-07-02 12:28:57 +02:00

This issue was referenced by ff444da7c4

This issue was referenced by ff444da7c45403ac0d112042c9d1d0a1992168d4

Julian Eisel commented

2020-07-02 13:05:35 +02:00

There's an issue on macOS that multiple devs stumbled over now, bcf49d13e5. To address that we'd have to drop support for macOS 10.12 and 10.13 from what I can tell.

I think the only difference between std::optional<>::operator* and std::option<>::value() is that the latter throws an exception when no value is set. For the former the behavior is undefined then.
So working around this issue is trivial. But I don't see a way to poison .value() usages on non-macOS systems. So if we don't change to macOS 10.14 min requirement, we just have to avoid the function and fix build errors if somebody still uses it.

There may be other C++14/C++17 features unavailable for 10.12. According to the compiler support overview the filesystem library is available in XCode 11.0 only (macOS 10.14.4+).

(I can open a new task for this if needed)

There's an issue on macOS that multiple devs stumbled over now, bcf49d13e534. To address that we'd have to drop support for macOS 10.12 and 10.13 from what I can tell. I think the only difference between `std::optional<>::operator*` and `std::option<>::value()` is that the latter throws an exception when no value is set. For the former the behavior is undefined then. So working around this issue is trivial. But I don't see a way to poison `.value()` usages on non-macOS systems. So if we don't change to macOS 10.14 min requirement, we just have to avoid the function and fix build errors if somebody still uses it. There may be other C++14/C++17 features unavailable for 10.12. According to the [compiler support overview ](https://en.cppreference.com/w/cpp/compiler_support) the filesystem library is available in XCode 11.0 only (macOS 10.14.4+). (I can open a new task for this if needed)

Brecht Van Lommel commented

2020-07-02 13:20:38 +02:00

We will upgrade the macOS minimum version to 10.13. I will make that change shortly along with related changes for the precompiled libraries.

I think dropping 10.13 is too early. Filesystem seems like it requires 10.15, which is definitely too recent.
https://stackoverflow.com/questions/42633477/macos-clang-c17-filesystem-header-not-found

We will upgrade the macOS minimum version to 10.13. I will make that change shortly along with related changes for the precompiled libraries. I think dropping 10.13 is too early. Filesystem seems like it requires 10.15, which is definitely too recent. https://stackoverflow.com/questions/42633477/macos-clang-c17-filesystem-header-not-found

Jaggz H commented

2020-07-18 07:36:18 +02:00

Added subscriber: @jaggz

Jaggz H commented

2020-07-18 07:36:18 +02:00

There are potential errors with the pre-built libs if depending on GCC 9.x, but I don't see a resolution to know if this is 100% the cause.
See: https://devtalk.blender.org/t/blender-2-90-local-build-fails/14393/4

I am experiencing the same undefined symbol references in Debian Stable, which is at GCC 8.3 -- from the svn-retrieved libs (including linux_centos7_x86_64/llvm/lib/libLLVMMCJIT.a
(MCJIT.cpp.o):MCJIT.cpp).

(C++17 has been supported since GCC 5, and even C++2a features are available since GCC 8.)
https://gcc.gnu.org/projects/cxx-status.html#cxx17

(I'm just reporting. I don't have any suggestions or recommendations.)

There are potential errors with the pre-built libs if depending on GCC 9.x, but I don't see a resolution to know if this is 100% the cause. See: https://devtalk.blender.org/t/blender-2-90-local-build-fails/14393/4 I am experiencing the same undefined symbol references in Debian Stable, which is at GCC 8.3 -- from the svn-retrieved libs (including linux_centos7_x86_64/llvm/lib/libLLVMMCJIT.a (MCJIT.cpp.o):MCJIT.cpp). (C++17 has been supported since GCC 5, and even C++2a features are available since GCC 8.) https://gcc.gnu.org/projects/cxx-status.html#cxx17 (I'm just reporting. I don't have any suggestions or recommendations.)

Sergey Sharybin commented

2020-07-20 10:12:57 +02:00

Please don't report issues in the coordination task which is already marked as done.

GCC 8.3 is below the required one - [x]. It has different ABI, which causes linker error. It might be possible to solve the linker issue, but this compiler has a codegen bug which breaks Cycles rendering in some scenes.

https://wiki.blender.org/wiki/Building_Blender#Compiler_Versions

Please don't report issues in the coordination task which is already marked as done. GCC 8.3 is below the required one - [x]. It has different ABI, which causes linker error. It might be possible to solve the linker issue, but this compiler has a codegen bug which breaks Cycles rendering in some scenes. - [x] https://wiki.blender.org/wiki/Building_Blender#Compiler_Versions

Thomas Dinges added this to the 2.90 milestone 2023-02-08 16:27:21 +01:00

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

C++ Version Upgrade #76783