Build: add global precompiled header #111225

Aras Pranckevicius · 2023-08-17T14:34:44+02:00

2023-08-17 14:34:44 +02:00

PCH includes commonly used STL bits: <algorithm>, <atomic>, <memory>, <string>, <vector>. Build time tests:

Windows, VS2022 Release ("make developer") build of blender: 7m10s -> 6m30s (Ryzen 5950X)
Mac, ninja developer debug build: 5m31s -> 5m02s (Apple M1Max). Total cpu time spent parsing source files goes 2089sec -> 1809sec, and none of STL headers are at the top of expensive to parse headers anymore (previously <algorithm> and <string> were)

PCH includes commonly used STL bits: `<algorithm>`, `<atomic>`, `<memory>`, `<string>`, `<vector>`. Build time tests: - Windows, VS2022 Release ("make developer") build of blender: 7m10s -> 6m30s (Ryzen 5950X) - Mac, ninja developer debug build: 5m31s -> 5m02s (Apple M1Max). Total cpu time spent parsing source files goes 2089sec -> 1809sec, and none of STL headers are at the top of expensive to parse headers anymore (previously `<algorithm>` and `<string>` were)

Germano Cavalcante added a new dependency 2023-08-17 14:38:36 +02:00

#107854 UI: Add Placement Settings to the Options popover

Germano Cavalcante removed a dependency 2023-08-17 14:38:58 +02:00

#107854 UI: Add Placement Settings to the Options popover

Aras Pranckevicius force-pushed global_pch from 678d3050c5 to 4fb1ccfec0

2023-08-18 17:14:00 +02:00

Compare

Ray molenkamp commented

2023-08-18 22:12:18 +02:00

While playing with this patch a while ago, there was some uneasiness about it I just can't shake

It was meant as a global pch (ie build once) but what cmake is doing is adding a PCH step to all C++ projects (ie bf_xxx_yyy) to precompile these headers, so rather than the common headers getting parsed 2600ish.. times, it's down to 100-150ish times, which is better, but not down to 1 which i was hoping for.

cmake has an option to reuse a PCH using REUSE_FROM but I couldn't get that to work.

Now if it's gonna add a PCH step to every sub project anyhow, we can likely do better our selves, and manually craft precompiled headers finely tuned for the projects that need it most rather than feeling all of them a generic set of common offenders.

While playing with this patch a while ago, there was some uneasiness about it I just can't shake It was meant as a global pch (ie build once) but what cmake is doing is adding a PCH step to all C++ projects (ie bf_xxx_yyy) to precompile these headers, so rather than the common headers getting parsed 2600ish.. times, it's down to 100-150ish times, which is better, but not down to 1 which i was hoping for. cmake has an [option](https://cmake.org/cmake/help/latest/command/target_precompile_headers.html) to reuse a PCH using `REUSE_FROM` but I couldn't get that to work. Now if it's gonna add a PCH step to every sub project anyhow, we can likely do better our selves, and manually craft precompiled headers finely tuned for the projects that need it most rather than feeling all of them a generic set of common offenders.

Aras Pranckevicius commented

2023-08-19 09:22:19 +02:00

As far as I can tell, pretty much "everything" right now in Blender needs <algorithm> and <string>. Which is unfortunate since C++ decided to put tons of stuff into algorithm, even if majority of things would only need things like min and max (fine, these would be easy to do manually) or things like copy_n (again trivial to do for simple things, but STL might have specializations for large loops w/ SIMD or whatever). On this machine, including algorithm and string results between 27 thousand and 35 thousand (depending on C++ language version enabled) non-whitespace lines of code.

So I'd think that even putting these into PCHs to be used everywhere would be a good first step.

That said, I'll take a look at REUSE_FROM, indeed it would be better if we'd find a way to make it work.

As far as I can tell, pretty much "everything" right now in Blender needs `<algorithm>` and `<string>`. Which is unfortunate since C++ decided to put _tons of stuff_ into algorithm, even if majority of things would only need things like `min` and `max` (fine, these would be easy to do manually) or things like `copy_n` (again trivial to do for simple things, but STL _might_ have specializations for large loops w/ SIMD or whatever). On this machine, including algorithm and string results between 27 thousand and 35 thousand (depending on C++ language version enabled) non-whitespace lines of code. So I'd think that even putting these into PCHs to be used everywhere would be a good first step. That said, I'll take a look at `REUSE_FROM`, indeed it would be better if we'd find a way to make it work.

Aras Pranckevicius force-pushed global_pch from 4fb1ccfec0 to ee11f3a3fa

2023-08-19 09:24:59 +02:00

Compare

Aras Pranckevicius added 1 commit 2023-08-20 15:26:54 +02:00

buildbot/vexp-code-patch-coordinator Build done. Details

010353a46e Build: share precompiled headers from base bf_pch library

On Apple M1 Max "make debug developer ninja" the ninja part
of empty build goes 331s -> 302s with PCH usage (no measurable
difference between shared and non-shared PCHs).

Editing BKE_attribute.hh and building "blender" target afterwards
goes 71s -> 68s.

Aras Pranckevicius changed title from ~~WIP: Build: add global precompiled header~~ to Build: add global precompiled header

2023-08-20 18:49:21 +02:00

Aras Pranckevicius requested review from Ray molenkamp 2023-08-20 18:49:41 +02:00

Ray molenkamp commented

2023-08-22 03:25:25 +02:00

more of a todo for my self (unless you want to look at this) but i'm getting

LINK : fatal error LNK1181: cannot open input file 'K:\BlenderGit\msvc_2022_lite\source\blender\pch\bf_pch.dir\Debug\cmake_pch.obj'
Done building project "bf_gpu_shaders.vcxproj" -- FAILED.

during a debug/vs ide build

more of a todo for my self (unless you want to look at this) but i'm getting ``` LINK : fatal error LNK1181: cannot open input file 'K:\BlenderGit\msvc_2022_lite\source\blender\pch\bf_pch.dir\Debug\cmake_pch.obj' Done building project "bf_gpu_shaders.vcxproj" -- FAILED. ``` during a debug/vs ide build

Aras Pranckevicius added 1 commit 2023-08-24 13:15:54 +02:00

buildbot/vexp-code-patch-coordinator Build done. Details

ec27abd5d5 Build: revert "share precompiled headers from base bf_pch library"

REUSE_FROM causes some issues with transitive library dependencies
on MSVC generator (see https://gitlab.kitware.com/cmake/cmake/-/issues/22630
where a similar issue was solved, but the developer comments indicate
that in more intricate cases the issue remains).

With REUSE_FROM, some targets do not link, but only in Debug config.
e.g. bf_blenlib wrongly tries to link with
build_windows_x64_vc17_Release\source\blender\pch\bf_pch.dir\Debug\cmake_pch.obj
which simply does not exist since nothing produces that.

Aras Pranckevicius commented

2023-08-24 13:17:48 +02:00

I'm getting during a debug/vs ide build

Ah, looks like cmake generates flat out wrong MSVC project files, but only for the Debug config, and only for some projects 🤦 Seems that this is hitting "complex enough case that REUSE_FROM does not handle" from https://gitlab.kitware.com/cmake/cmake/-/issues/22630 -- I've pushed a commit that reverts to previous "each library just rebuilds their own pch" without using REUSE_FROM. Conceptually worse, but in my testing (on both Windows & Mac) that does not actually affect build time in a measurable way.

> I'm getting during a debug/vs ide build Ah, looks like cmake generates flat out wrong MSVC project files, but _only_ for the Debug config, and only for _some_ projects 🤦 Seems that this is hitting "complex enough case that REUSE_FROM does not handle" from https://gitlab.kitware.com/cmake/cmake/-/issues/22630 -- I've pushed a commit that reverts to previous "each library just rebuilds their own pch" without using REUSE_FROM. Conceptually worse, but in my testing (on both Windows & Mac) that does not actually affect build time in a measurable way.

Aras Pranckevicius added 1 commit 2023-08-24 13:49:56 +02:00

buildbot/vexp-code-patch-coordinator Build done. Details

29393e3fc2 Merge branch 'main' into global_pch

Aras Pranckevicius commented

2023-08-24 13:53:16 +02:00

@blender-bot build

buildbot/vexp-code-patch-coordinator Build done.

Details

Merge conflict checking is in progress. Try again in few moments.

View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u global_pch:aras_p-global_pch

git checkout aras_p-global_pch

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Build: add global precompiled header #111225

Checkout