Cycles: Test Embree huge pages #117175

New Issue

Open

opened 2024-01-16 16:36:39 +01:00 by Brecht Van Lommel · 20 comments

Brecht Van Lommel commented

2024-01-16 16:36:39 +01:00

Owner

In #116663 huge pages were enabled in jemalloc on Linux, which helps for Embree too.

But Embree also has its own option, which may or may not provide additional benefit. This is to be tested.
https://github.com/embree/embree/blob/master/doc/src/api.md#huge-page-support

On Windows it is also possible to use huge pages, but this requires Blender to run in elevated mode. However it means memory can't be paged out. While an option to enable it could be added, it seems too obscure to be worth it.

In #116663 huge pages were enabled in jemalloc on Linux, which helps for Embree too. But Embree also has its own option, which may or may not provide additional benefit. This is to be tested. https://github.com/embree/embree/blob/master/doc/src/api.md#huge-page-support On Windows it is also possible to use huge pages, but this requires Blender to run in elevated mode. However it means memory can't be paged out. While an option to enable it could be added, it seems too obscure to be worth it.

Brecht Van Lommel added the

Good First Issue

Render & Cycles

labels 2024-01-16 16:36:40 +01:00

Brecht Van Lommel added this to the Render & Cycles project 2024-01-16 16:36:42 +01:00

Brecht Van Lommel referenced this issue

2024-01-16 16:38:20 +01:00

Linux: Use huge pages in jemalloc to speed up allocations #116663

Holly Liu commented

2024-01-28 23:23:58 +01:00

Hey Brecht, I'd like to try this out if you don't mind.

Hey Brecht, I'd like to try this out if you don't mind.

Brecht Van Lommel commented

2024-01-29 11:01:11 +01:00

Author

Owner

Sure, you can try it. Did you want to check this for Windows or Linux?

Sure, you can try it. Did you want to check this for Windows or Linux?

Daksh-Kaushik commented

2024-01-31 22:06:27 +01:00

Hey Brecht, I checked out #116663 and understood huge pages, to enable huge pages we need to pass 'hugepages=1' in 'rtcNewDevice' (which is a function) , but I was not able to find the files where Embree was included to pass the parameter value(I tried finding it in blender/source/blender and blender/intern/cycles), also I am checking for Linux, as I just set it up on my PC and this is my first good first issue.

Hey Brecht, I checked out #116663 and understood huge pages, to enable huge pages we need to pass 'hugepages=1' in 'rtcNewDevice' (which is a function) , but I was not able to find the files where Embree was included to pass the parameter value(I tried finding it in blender/source/blender and blender/intern/cycles), also I am checking for Linux, as I just set it up on my PC and this is my first good first issue.

Holly Liu commented

2024-02-01 00:22:54 +01:00

I assume we need to install Embree and add it to Blender ourselves. I also built Blender for Linux but I have Windows dual-booted so I can test this out on Windows too if needed.

I assume we need to install Embree and add it to Blender ourselves. I also built Blender for Linux but I have Windows dual-booted so I can test this out on Windows too if needed.

Brecht Van Lommel commented

2024-02-01 02:35:59 +01:00

Author

Owner

Blender builds with Embree by default, it is included in the precompiled libraries. It should be possible to find rtcNewDevice by searching the code, with grep or search functionality in an IDE.

Blender builds with Embree by default, it is included in the precompiled libraries. It should be possible to find `rtcNewDevice` by searching the code, with grep or search functionality in an IDE.

Ray molenkamp commented

2024-02-01 02:42:12 +01:00

Member

On Windows it is also possible to use huge pages, but this requires Blender to run in elevated mode. It would be worth investigating if we can enable this automatically for admin users.

I don't think this is a good idea, huge pages on windows cannot be paged out, which is i suppose a performance benefit, but having a single memory hungry process on the system, that can't be paged out feels like a bad idea to automatically enable. i could live with a cli switch i suppose, but automatic seems like a bit of an over reach.

> On Windows it is also possible to use huge pages, but this requires Blender to run in elevated mode. It would be worth investigating if we can enable this automatically for admin users. I don't think this is a good idea, huge pages on windows cannot be paged out, which is i suppose a performance benefit, but having a single memory hungry process on the system, that can't be paged out feels like a bad idea to automatically enable. i could live with a cli switch i suppose, but automatic seems like a bit of an over reach.

Brecht Van Lommel commented

2024-02-01 11:29:41 +01:00

Author

Owner

I was not aware of that, indeed we should not do this on Windows then.

I was not aware of that, indeed we should not do this on Windows then.

Holly Liu commented

2024-02-01 15:52:19 +01:00

Ah, thanks for the heads up.

Ah, thanks for the heads up.

Ray molenkamp commented

2024-02-01 23:26:53 +01:00

Member

I took a closer look here, and it seems someone already did the tbbmalloc changes required to support this [1]. I applied the patch to our copy of tbb and build a new tbbmalloc.dll

I'm in no way set up to run any kind of repeatable performance benchmark though, so beyond validating large pages are actually being allocated i can't really say if there is a performance benefit or not.

To test:

Make sure your user has the Lock pages in memory privilege in the local security policy [2]. If you just assigned this privilege you need to logoff or reboot otherwise the new policy will not apply.
Replace blenders tbbmalloc.dll with the copy attached to this post.
Open up a command prompt with administrator privileges
Run set TBBMALLOC_USE_HUGE_PAGES=1
Run blender from this same command prompt
Validate with rammap [3] large pages are allocated.

[1] https://github.com/oneapi-src/oneTBB/compare/master...Hardcode84:oneTBB:huge-pages-win
[2] https://learn.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/lock-pages-in-memory
[3] https://learn.microsoft.com/en-us/sysinternals/downloads/rammap

I took a closer look here, and it seems someone already did the tbbmalloc changes required to support this [[1]](https://github.com/oneapi-src/oneTBB/compare/master...Hardcode84:oneTBB:huge-pages-win). I applied the patch to our copy of tbb and build a new `tbbmalloc.dll` I'm in no way set up to run any kind of repeatable performance benchmark though, so beyond validating large pages are actually being allocated i can't really say if there is a performance benefit or not. To test: 1) Make sure your user has the `Lock pages in memory` privilege in the local security policy [[2]](https://learn.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/lock-pages-in-memory). If you _just_ assigned this privilege you need to logoff or reboot otherwise the new policy _will not apply_. 2) Replace blenders `tbbmalloc.dll` with the copy attached to this post. 3) Open up a command prompt with administrator privileges 4) Run `set TBBMALLOC_USE_HUGE_PAGES=1` 5) Run blender from this same command prompt 6) Validate with rammap [[3]](https://learn.microsoft.com/en-us/sysinternals/downloads/rammap) large pages are allocated. [1] https://github.com/oneapi-src/oneTBB/compare/master...Hardcode84:oneTBB:huge-pages-win [2] https://learn.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/lock-pages-in-memory [3] https://learn.microsoft.com/en-us/sysinternals/downloads/rammap

tbbmalloc_win64_huge_page_support.zip

27 KiB

Daksh-Kaushik commented

2024-02-02 00:06:42 +01:00

(For Linux) Well, I understood what I need to do, as in Linux huge pages are enabled by default, I need to check it with disabling THP using ' echo /sys/kernel/mm/transparent_hugepage/disabled ' and compare timings for both enabled and disabled THP using the ' huge_page_test.cc ' given in #116663, but I do not know how to unit test, so I'll try to learn it.

(For Linux) Well, I understood what I need to do, as in Linux huge pages are enabled by default, I need to check it with disabling THP using ' echo /sys/kernel/mm/transparent_hugepage/disabled ' and compare timings for both enabled and disabled THP using the ' huge_page_test.cc ' given in #116663, but I do not know how to unit test, so I'll try to learn it.

Holly Liu commented

2024-02-04 15:54:12 +01:00

(Linux)
I tried to recreate Eugene's benchmark and may or may not see a speedup with Embree.
I ran his command although I could not figure out how to run it as 4 samples per pixel:
time ./build_linux/bin/blender -b monkey_test.blend -a -- --cycles-device CPU

I enabled Embree thp by:
echo always > /sys/kernel/mm/transparent_hugepage/enabled
restart

and disabled Embree thp by:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
restart

I tried to disable jemalloc thp by undoing commit 10dfa07e36

Version	time (real) for monkey_test.blend (s)	time (real) for rug0113.blend (s)
embree without jemalloc	26.976, 25.944, 26.260	15.870, 15.867, 15.833
jemalloc without embree	26.413, 26.329, 26.225	15.909, 15.864, 16.037
no embree no jemalloc	26.240, 26.326, 26.482	15.906, 16.044, 15.952

Could not figure out how to recreate @brecht 's cycles cpu rendering benchmark with these conditions.

Is this on the right track?

(Linux) I tried to recreate Eugene's benchmark and may or may not see a speedup with Embree. I ran his command although I could not figure out how to run it as 4 samples per pixel: `time ./build_linux/bin/blender -b monkey_test.blend -a -- --cycles-device CPU` I enabled Embree thp by: `echo always > /sys/kernel/mm/transparent_hugepage/enabled` restart and disabled Embree thp by: `echo never > /sys/kernel/mm/transparent_hugepage/enabled` restart I tried to disable jemalloc thp by undoing commit https://projects.blender.org/blender/blender/commit/10dfa07e361b6a8b488a9263440bd69e64ae67ff | Version | time (real) for monkey_test.blend (s) | time (real) for rug0113.blend (s) | |---|---|---| | embree without jemalloc | 26.976, 25.944, 26.260 | 15.870, 15.867, 15.833 | | jemalloc without embree | 26.413, 26.329, 26.225 | 15.909, 15.864, 16.037 | | no embree no jemalloc | 26.240, 26.326, 26.482 | 15.906, 16.044, 15.952 | Could not figure out how to recreate @brecht 's cycles cpu rendering benchmark with these conditions. Is this on the right track?

Brecht Van Lommel commented

2024-02-04 17:13:52 +01:00

Author

Owner

It's not really what I meant for this task.

It's a bit confusing because the Embree docs contain some information about how to change the system settings to enable huge pages. But there is nothing Embree specific about, and we can't change those system settings from Blender.

What I wanted to investigate is if enabling huge pages in the rtcNewDevice call provides any additional benefit over jemalloc huge pages. It likely does not provide a benefit, but I wanted to check to be sure.

It's not really what I meant for this task. It's a bit confusing because the Embree docs contain some information about how to change the system settings to enable huge pages. But there is nothing Embree specific about, and we can't change those system settings from Blender. What I wanted to investigate is if enabling huge pages in the `rtcNewDevice` call provides any additional benefit over jemalloc huge pages. It likely does not provide a benefit, but I wanted to check to be sure.

Ray molenkamp commented

2024-02-04 17:27:56 +01:00

Member

Took a closer look. Embree calls mmap (linux/mac) and VirtualAlloc (windows) directly for its memory management, bypassing jemalloc / tbbmalloc so i'm expecting it to make a difference.

Took a closer look. Embree calls `mmap` (linux/mac) and `VirtualAlloc` (windows) directly for its memory management, bypassing jemalloc / tbbmalloc so i'm expecting it to make a difference.

Holly Liu commented

2024-02-04 23:07:53 +01:00

I see, I seem to have misunderstood some things. Are the benchmark tests that I ran the right idea or how should we do that?
I looked at testing for Windows mentioned above by following the steps in that comment. Though had some trouble building Blender on Windows.

I see, I seem to have misunderstood some things. Are the benchmark tests that I ran the right idea or how should we do that? I looked at testing for Windows mentioned above by following the steps in that comment. Though had some trouble building Blender on Windows.

Alaska commented

2024-02-05 05:03:25 +01:00

Member

Just to get some information out, primarily for @holly-l . Linux supports an option called "Transparent Huge Pages". With this feature enabled, the Linux kernel will try to combine the normal 4KB pages into 2MB pages. This can improve performance in some applications, and this can be enabled with sudo echo always > /sys/kernel/mm/transparent_hugepage/enabled. Enabling and disabling this is practically out of the control of the Blender foundation, and so it's not worth testing here.

An application also has the option to direclty allocate huge pages. So instead of the app creating 4KB pages and the kernel combining them into 2MB pages, the app just creates 2MB pages. This is the feature we want to test. And Embree has an option for this.

It seems like to let Embree directly allocate 2MB huge pages on Linux, you need to create a "huge page pool" (run echo number_of_huge_pages_in_pool > /proc/sys/vm/nr_overcommit_hugepages as root) So if you want 1000 pages in that pool (roughly 2GB of avaliable space for Embree to allocate into), then you need to run echo 1000 > /proc/sys/vm/nr_overcommit_hugepages. And then you need to tell Embree to use huge pages (which the Embree readme claims is enabled by default for Linux).

But just so it's clear. To tell Embree to use or not to use huge pages, you need to pass hugepages=1 or hugepages=0 to rtcNewDevice in the Cycles code. An example of how to do this can be found here: f5d09cce0c

There might be something I'm missing. So please correct me if I'm wrong.

How would you test this? Assuming the Embree readme is right, there's no point testing this on Linux. Huge pages offers a performance boost, and it's enabled by default on Linux. Nothing needs to be done on the Blender/Cycles side.

But if the Embree readme is wrong, or you just want to test this on Linux then you need too:

Setup a build environment for Blender so you can compile it yourself. https://developer.blender.org/docs/handbook/building_blender/
Compile Blender with hugepages=1 in rtcNewDevice. This is your version of Blender for testing huge pages.
Compile Blender with hugepages=0 in rtcNewDevice. This is your version of Blender for testing without huge pages.
Maybe compile Blender with neither set? This is just to verify if it's enabled by default.
Create a huge page pool for Embree to use (E.G. echo 1000 > /proc/sys/vm/nr_overcommit_hugepages as root).
Then you test.

Testing should consist of:

Try and reduce variables that could impact performance (E.G. Close background applications, ensure your computer is adequately cooled, don't interact with your computer while testing is running)
Render a Cycles scene on your CPU (you can use the command line or GUI) with the two versions of Blender you compiled earlier (the huge pages and non huge pages version).
Record the render time.
Repeat a few times to try and eliminate noise from testing (I usually do 3 times).
Repeat steps 2-4 on a few different scenes.
If you find a file where huge pages offers a noticeable performance benefit, test that version of Blender that doesn't have anything set and see where it's performance lies. If performance is close to huge pages, then huge pages is probably enabled by default on Linux. If it's close to the non huge pages option, then huge pages is not enabled by default. This is useful information for the Blender foundation as it lets them know whether or not they need to do something for Linux.

Here's some scenes you can use for testing:

The Blender demos files. https://www.blender.org/download/demo-files/
Cycles benchmark files. https://svn.blender.org/svnroot/bf-blender/trunk/lib/benchmarks/cycles/

I should note that Blender does come with a benchmark script for automatically benchmarking different Blender versions with the Cycles benchmark files. Information about it can be found here: https://developer.blender.org/docs/handbook/testing/performance/

And lastly report your results.

Just to get some information out, primarily for @holly-l . Linux supports an option called "Transparent Huge Pages". With this feature enabled, the Linux kernel will try to combine the normal 4KB pages into 2MB pages. This can improve performance in some applications, and this can be enabled with `sudo echo always > /sys/kernel/mm/transparent_hugepage/enabled`. Enabling and disabling this is practically out of the control of the Blender foundation, and so it's not worth testing here. An application also has the option to direclty allocate huge pages. So instead of the app creating 4KB pages and the kernel combining them into 2MB pages, the app just creates 2MB pages. This is the feature we want to test. And Embree has an option for this. It seems like to let Embree directly allocate 2MB huge pages on Linux, you need to create a "huge page pool" (run `echo number_of_huge_pages_in_pool > /proc/sys/vm/nr_overcommit_hugepages` as root) So if you want 1000 pages in that pool (roughly 2GB of avaliable space for Embree to allocate into), then you need to run `echo 1000 > /proc/sys/vm/nr_overcommit_hugepages`. And then you need to tell Embree to use huge pages (which the Embree readme claims is enabled by default for Linux). But just so it's clear. To tell Embree to use or not to use huge pages, you need to pass `hugepages=1` or `hugepages=0` to `rtcNewDevice` in the Cycles code. An example of how to do this can be found here: https://projects.blender.org/Alaska/blender/commit/f5d09cce0c2996a03046766d191f59f9fa7fe062 There might be something I'm missing. So please correct me if I'm wrong. --- How would you test this? Assuming the Embree readme is right, there's no point testing this on Linux. Huge pages offers a performance boost, and it's enabled by default on Linux. Nothing needs to be done on the Blender/Cycles side. But if the Embree readme is wrong, or you just want to test this on Linux then you need too: 1. Setup a build environment for Blender so you can compile it yourself. https://developer.blender.org/docs/handbook/building_blender/ 2. Compile Blender with `hugepages=1` in `rtcNewDevice`. This is your version of Blender for testing huge pages. 3. Compile Blender with `hugepages=0` in `rtcNewDevice`. This is your version of Blender for testing without huge pages. 4. Maybe compile Blender with neither set? This is just to verify if it's enabled by default. 5. Create a huge page pool for Embree to use (E.G. `echo 1000 > /proc/sys/vm/nr_overcommit_hugepages` as root). 5. Then you test. Testing should consist of: 1. Try and reduce variables that could impact performance (E.G. Close background applications, ensure your computer is adequately cooled, don't interact with your computer while testing is running) 2. Render a Cycles scene on your CPU (you can use the command line or GUI) with the two versions of Blender you compiled earlier (the huge pages and non huge pages version). 3. Record the render time. 4. Repeat a few times to try and eliminate noise from testing (I usually do 3 times). 5. Repeat steps 2-4 on a few different scenes. 6. If you find a file where huge pages offers a noticeable performance benefit, test that version of Blender that doesn't have anything set and see where it's performance lies. If performance is close to huge pages, then huge pages is probably enabled by default on Linux. If it's close to the non huge pages option, then huge pages is not enabled by default. This is useful information for the Blender foundation as it lets them know whether or not they need to do something for Linux. Here's some scenes you can use for testing: 1. The Blender demos files. https://www.blender.org/download/demo-files/ 2. Cycles benchmark files. https://svn.blender.org/svnroot/bf-blender/trunk/lib/benchmarks/cycles/ I should note that Blender does come with a benchmark script for automatically benchmarking different Blender versions with the Cycles benchmark files. Information about it can be found here: https://developer.blender.org/docs/handbook/testing/performance/ And lastly report your results.

Alaska commented

2024-02-05 07:02:21 +01:00

Member

I ran some quick tests on Windows using the Cycles benchmark files. And I could be doing something wrong. With huge pages I saw a average performance increase of 0.68%. The maximum performance increase was 2.04%. And only one scene saw a performance deficient of 0.05%.

I should note that it seems the tools to enable huge pages on Windows is limited to Pro editions and higher. Which once again limits the availability of huge pages on Windows.

Hardware used for testing:
CPU: AMD Ryzen 5 5600
RAM: 32GB of 3200mhz DDR4 RAM (unsure of timings)

I ran some quick tests on Windows using the Cycles benchmark files. And I could be doing something wrong. With huge pages I saw a average performance increase of 0.68%. The maximum performance increase was 2.04%. And only one scene saw a performance deficient of 0.05%. I should note that it seems the tools to enable huge pages on Windows is limited to Pro editions and higher. Which once again limits the availability of huge pages on Windows. Hardware used for testing: CPU: AMD Ryzen 5 5600 RAM: 32GB of 3200mhz DDR4 RAM (unsure of timings)

Brecht Van Lommel commented

2024-02-05 13:09:18 +01:00

Author

Owner

Thanks for the explanation and test.

Comparing render times on a few scenes is fine, they just have to be complicated enough like the benchmark ones we use. Small scenes may fit entirely in the cache and would not get much benefit from this change.

For the Windows performance impact, hard to say if anything was wrong in that test. It's quite possible the impact is really that small. If so it's not worth adding support for that.

Thanks for the explanation and test. Comparing render times on a few scenes is fine, they just have to be complicated enough like the benchmark ones we use. Small scenes may fit entirely in the cache and would not get much benefit from this change. For the Windows performance impact, hard to say if anything was wrong in that test. It's quite possible the impact is really that small. If so it's not worth adding support for that.

Alaska commented

2024-02-05 17:21:13 +01:00

Member

I did some testing for Linux using the Cycles benchmark files.

The Embree readme is correct. Hugepages is enabled on Linux, even if you don't explictly request for hugepages to be enabled. This can be verified with grep -i HugePages_Total /proc/meminfo while running a Cycles render on an unmodified main version of Blender (requires a hugepage pool to be setup).

Even with that, performance seems to be different between rtcNewDevice("verbose=0") and rtcNewDevice("hugepages=1,verbose=0") (they're supposed to be the same since hugepages is enabled on both). Typically hugepages=1,verbose=0 is faster.

Finally performance:

With rtcNewDevice("verbose=0") (as found in main), It provides a 0.15% performance uplift on average over explicitly disabling hugepages. The max performance uplift was 1.13%. And some scenes saw a performance deficit, with the worse being 0.72%.

With rtcNewDevice("hugepages=1,verbose=0"), It provides a 0.50% performance uplift on average over explicitly disabling hugepages. The max performance uplift was 1.67%. And some scenes saw a performance deficit, with the worse being 1.11%.

Hardware used for testing:
CPU: AMD Ryzen 5 5600
RAM: 32GB of 3200mhz DDR4 RAM (unsure of timings)

I did some testing for Linux using the Cycles benchmark files. The Embree readme is correct. Hugepages is enabled on Linux, even if you don't explictly request for hugepages to be enabled. This can be verified with `grep -i HugePages_Total /proc/meminfo` while running a Cycles render on an unmodified main version of Blender (requires a hugepage pool to be setup). Even with that, performance seems to be different between `rtcNewDevice("verbose=0")` and `rtcNewDevice("hugepages=1,verbose=0")` (they're supposed to be the same since hugepages is enabled on both). Typically `hugepages=1,verbose=0` is faster. --- Finally performance: With `rtcNewDevice("verbose=0")` (as found in main), It provides a 0.15% performance uplift on average over explicitly disabling hugepages. The max performance uplift was 1.13%. And some scenes saw a performance deficit, with the worse being 0.72%. With `rtcNewDevice("hugepages=1,verbose=0")`, It provides a 0.50% performance uplift on average over explicitly disabling hugepages. The max performance uplift was 1.67%. And some scenes saw a performance deficit, with the worse being 1.11%. Hardware used for testing: CPU: AMD Ryzen 5 5600 RAM: 32GB of 3200mhz DDR4 RAM (unsure of timings)

Brecht Van Lommel commented

2024-02-07 19:44:23 +01:00

Author

Owner

Thanks for the tests. If it's 0.5% and it's worse in some files, I think it's too small to be a clear win, and may depend more on specifics of the scene and coincidences in favorable memory layout. So I would just stay with the jemalloc huge pages then.

Thanks for the tests. If it's 0.5% and it's worse in some files, I think it's too small to be a clear win, and may depend more on specifics of the scene and coincidences in favorable memory layout. So I would just stay with the jemalloc huge pages then.

Holly Liu commented

2024-02-10 01:37:32 +01:00

@Alaska I looked at your explanation and found it to be very clear. Thanks.

@Alaska I looked at your explanation and found it to be very clear. Thanks.

Sign in to join this conversation.

No Label

Animation & Rigging

Asset Browser Project Overview

Automated Testing

Blender Asset Bundle

Dependency Graph

Development Management

EEVEE & Viewport

Images & Movies

Motion Tracking

Nodes & Physics

Pipeline, Assets & IO

Platforms, Builds & Tests

Render & Cycles

Render Pipeline

Sculpt, Paint & Texture

Video Sequencer

Virtual Reality

Blender 2.8 Project

Milestone 1: Basic, Local Asset Browser

Good First Issue

Animation & Rigging

Development Management

EEVEE & Viewport

Nodes & Physics

Pipeline, Assets & IO

Platforms, Builds & Tests

Render & Cycles

Sculpt, Paint & Texture

Needs Info from Developers

Needs Information from User

No Milestone

No project

No Assignees

5 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#117175

No description provided.