Points not rendered when indices are higher than 67.5 million(roughly) #108901

New Issue

Open

opened 2023-06-12 16:53:21 +02:00 by Miro Horváth · 13 comments

Miro Horváth commented

2023-06-12 16:53:21 +02:00

System Information
Operating system: Windows-10-10.0.19044-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 3070 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 516.94

Blender Version
Broken: version: 3.6.0 Beta, branch: blender-v3.6-release, commit date: 2023-05-22 21:38, hash: 43fe7bec4fa5

Short description of error
I encountered this issue when importing huge point clouds(~100 million points).

I prepared a simple file to reproduce the issue, it seems that points stop rendering roughly around index 67.5 million, see the gap between points and 3D cursor.

Repro:

unhide Plane object in Outliner, you should see gap between points and 3D cursor, that means some points didn't render
change integer from 67.500.000 to say 40.000.000, see that all points along the X axis up to 10(3D cursor) got rendered

Edit: when you unmute Points to Vertices node you can see vertices along whole 0-10 range

**System Information** Operating system: Windows-10-10.0.19044-SP0 64 Bits Graphics card: NVIDIA GeForce RTX 3070 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 516.94 **Blender Version** Broken: version: 3.6.0 Beta, branch: blender-v3.6-release, commit date: 2023-05-22 21:38, hash: `43fe7bec4fa5` **Short description of error** I encountered this issue when importing huge point clouds(~100 million points). ![image](/attachments/e7c8b642-e194-4ce4-9b64-3d56b0a82b46) I prepared a simple file to reproduce the issue, it seems that points stop rendering roughly around index 67.5 million, see the gap between points and 3D cursor. ![image](/attachments/e2326f79-f133-400b-84c2-20dd5fa814f2) Repro: - unhide Plane object in Outliner, you should see gap between points and 3D cursor, that means some points didn't render - change integer from 67.500.000 to say 40.000.000, see that all points along the X axis up to 10(3D cursor) got rendered Edit: when you unmute Points to Vertices node you can see vertices along whole 0-10 range

863 KiB

990 KiB

1.7 MiB

Miro Horváth added the

labels 2023-06-12 16:53:22 +02:00

Iliya Katushenock added the

EEVEE & Viewport

Nodes & Physics

labels 2023-06-12 16:59:18 +02:00

Miro Horváth commented

2023-06-12 17:04:54 +02:00

Author

Jeroen asked me to add also system-info.txt

Jeroen asked me to add also system-info.txt

system-info.txt

26 KiB

Jeroen Bakker commented

2023-06-12 17:55:24 +02:00

Member

What I forgot to ask, if the points are also missing in Cycles it would indicate an issue in GN, otherwise it would be a Viewport issue.

What I forgot to ask, if the points are also missing in Cycles it would indicate an issue in GN, otherwise it would be a Viewport issue.

Miro Horváth commented

2023-06-12 18:52:19 +02:00

Author

@Jeroen-Bakker experienced in both Eevee and Cycles engine

@Jeroen-Bakker experienced in both Eevee and Cycles engine

Pratik Borhade commented

2023-06-13 07:08:17 +02:00

Member

Can confirm.

Can confirm.

Pratik Borhade added

Nodes & Physics

and removed

Nodes & Physics

labels 2023-06-13 07:08:36 +02:00

Lukas Tönne commented

2023-06-19 11:07:18 +02:00

Member

I'm getting a crash in release mode in drawing code, full backtrace attached

1   ??                                                0x7fffebb92929 
2   ??                                                0x7fffca036786 
3   ??                                                0x7fffc9c678a2 
4   ??                                                0x7fffc9c68115 
5   ??                                                0x7fffc9c6d022 
6   ??                                                0x7fffc9b0b2c7 
7   blender::gpu::GLIndexBuf::bind()                  0xb2edff0      
8   blender::gpu::GLVaoCache::vao_get(GPUBatch *)     0xb2e8909      
9   blender::gpu::GLBatch::bind(int)                  0xb2e8a34      
10  blender::gpu::GLBatch::draw(int, int, int, int)   0xb2e8a8d      
11  draw_call_single_do                               0x57433db      
12  drw_draw_pass_ex                                  0x574445f      
13  DRW_draw_pass                                     0x5744faf      
14  workbench_draw_sample                             0x57e3561      
15  workbench_draw_scene                              0x57e3698      
16  drw_engines_draw_scene                            0x5738232      
17  DRW_draw_render_loop_ex                           0x573860a      
18  view3d_main_region_draw                           0x66173d3      
19  ED_region_do_draw                                 0x5b427ce      
20  wm_draw_update                                    0x54f6f65      
... <More>

I'm getting a crash in release mode in drawing code, full backtrace attached ``` 1 ?? 0x7fffebb92929 2 ?? 0x7fffca036786 3 ?? 0x7fffc9c678a2 4 ?? 0x7fffc9c68115 5 ?? 0x7fffc9c6d022 6 ?? 0x7fffc9b0b2c7 7 blender::gpu::GLIndexBuf::bind() 0xb2edff0 8 blender::gpu::GLVaoCache::vao_get(GPUBatch *) 0xb2e8909 9 blender::gpu::GLBatch::bind(int) 0xb2e8a34 10 blender::gpu::GLBatch::draw(int, int, int, int) 0xb2e8a8d 11 draw_call_single_do 0x57433db 12 drw_draw_pass_ex 0x574445f 13 DRW_draw_pass 0x5744faf 14 workbench_draw_sample 0x57e3561 15 workbench_draw_scene 0x57e3698 16 drw_engines_draw_scene 0x5738232 17 DRW_draw_render_loop_ex 0x573860a 18 view3d_main_region_draw 0x66173d3 19 ED_region_do_draw 0x5b427ce 20 wm_draw_update 0x54f6f65 ... <More> ```

backtrace.txt

116 KiB

Lukas Tönne commented

2023-06-19 11:09:42 +02:00

Member

I'm guessing we're hitting some machine-specific limit like GL_MAX_ELEMENTS_INDICES here (post on this topic). May be feasible to do by splitting the point cloud into more manageable chunks.

I'm guessing we're hitting some machine-specific limit like `GL_MAX_ELEMENTS_INDICES` here ([post on this topic](https://community.khronos.org/t/gldrawelements-buffer-size-limit/73746)). May be feasible to do by splitting the point cloud into more manageable chunks.

Miro Horváth commented

2023-06-19 11:30:39 +02:00

Author

Hi @LukasTonne , while it's not a problem to split point clouds before they reach some limit, Viewer node can be active only on one object at the time(only one object would be colored).

Possibility to have Viewer node working on multiple objects or other way how to render colored point clouds in the viewport would great feature for upcoming release of Blender 4.0.

Hi @LukasTonne , while it's not a problem to split point clouds before they reach some limit, Viewer node can be active only on one object at the time(only one object would be colored). ![image](/attachments/d364cb42-e771-4d96-9461-46d90b73ec6c) Possibility to have Viewer node working on multiple objects or other way how to render colored point clouds in the viewport would great feature for upcoming release of Blender 4.0.

742 KiB

Lukas Tönne commented

2023-06-19 11:38:11 +02:00

Member

@MiroHorvath I didn't mean this as a solution in geometry nodes. It appears to be primarily an OpenGL problem with limited buffer sizes. If you push it too far you'll hit a memory limit in GN as well but i expect that to be a few orders of magnitude bigger.

@MiroHorvath I didn't mean this as a solution in geometry nodes. It appears to be primarily an OpenGL problem with limited buffer sizes. If you push it too far you'll hit a memory limit in GN as well but i expect that to be a few orders of magnitude bigger.

Miro Horváth commented

2023-06-19 11:45:34 +02:00

Author

@MiroHorvath I didn't mean this as a solution in geometry nodes. It appears to be primarily an OpenGL problem with limited buffer sizes. If you push it too far you'll hit a memory limit in GN as well but i expect that to be a few orders of magnitude bigger.

Ah right, that sounds good.

> @MiroHorvath I didn't mean this as a solution in geometry nodes. It appears to be primarily an OpenGL problem with limited buffer sizes. If you push it too far you'll hit a memory limit in GN as well but i expect that to be a few orders of magnitude bigger. Ah right, that sounds good.

Lukas Tönne commented

2023-06-19 11:56:41 +02:00

Member

@fclem @Jeroen-Bakker Can you confirm that this is an OpenGL buffer limitation?

@fclem @Jeroen-Bakker Can you confirm that this is an OpenGL buffer limitation?

Jeroen Bakker commented

2023-06-19 12:17:37 +02:00

Member

Can you add your system_info.txt just to be sure? And yes it is something to be aware of https://registry.khronos.org/OpenGL-Refpages/gl4/html/glDrawRangeElements.xhtml

Note that doesn't change the buffer size, just the number of elements/indexes that can be drawn in a single command. You should still be able to allocate the memory, just the drawing needs to be split in multiple drawing commands. This could be responsible for the GPU backend, and not for the code that is calling the GPU code, but currently isn't.

Can you add your system_info.txt just to be sure? And yes it is something to be aware of https://registry.khronos.org/OpenGL-Refpages/gl4/html/glDrawRangeElements.xhtml Note that doesn't change the buffer size, just the number of elements/indexes that can be drawn in a single command. You should still be able to allocate the memory, just the drawing needs to be split in multiple drawing commands. This could be responsible for the GPU backend, and not for the code that is calling the GPU code, but currently isn't.

Lukas Tönne commented

2023-06-19 12:21:19 +02:00

Member

No description provided.

system-info.txt

13 KiB

Clément Foucault commented

2023-06-22 23:50:10 +02:00

Member

I think this has to do with this snippet:

int pointcloud_get_point_id()
{
  return gl_VertexID / 32;
}

Basically, I reserve some bits inside the vertex index to store the point shape vertices index (the half octahedron).
The maximum allowed index is 0x8FFFFFFF = 2,147,483,647 because 0xFFFFFFFF is considered the restart index. So we have 31 - 5 = 26 bits for point IDs which gives a maximum of 0x3FFFFFF = 67,108,863 points.

Now the real question is, why did I reserve 5 bits instead of the currently used 3 bits (indices of the octahedron only goes to 4). Maybe I was future proofing and thought we will add more resolution when needed.

Since this day hasn't come yet, we could very well reduce the number of reserved bits to 3. That would allow a maximum of 0x0FFFFFFF = 268,435,455 points.

Note that the 32 constant is contained in both common_pointcloud_lib.glsl and draw_cache_impl_pointcloud.cc.

I think this has to do with this snippet: ``` int pointcloud_get_point_id() { return gl_VertexID / 32; } ``` Basically, I reserve some bits inside the vertex index to store the point shape vertices index (the half octahedron). The maximum allowed index is `0x8FFFFFFF = 2,147,483,647` because `0xFFFFFFFF` is considered the restart index. So we have 31 - 5 = 26 bits for point IDs which gives a maximum of `0x3FFFFFF = 67,108,863` points. Now the real question is, why did I reserve 5 bits instead of the currently used 3 bits (indices of the octahedron only goes to 4). Maybe I was future proofing and thought we will add more resolution when needed. Since this day hasn't come yet, we could very well reduce the number of reserved bits to 3. That would allow a maximum of `0x0FFFFFFF = 268,435,455` points. Note that the 32 constant is contained in both `common_pointcloud_lib.glsl` and `draw_cache_impl_pointcloud.cc`.

Sign in to join this conversation.

No Label

Animation & Rigging

Asset Browser Project Overview

Automated Testing

Blender Asset Bundle

Dependency Graph

Development Management

EEVEE & Viewport

Images & Movies

Motion Tracking

Nodes & Physics

Pipeline, Assets & IO

Platforms, Builds & Tests

Render & Cycles

Render Pipeline

Sculpt, Paint & Texture

Video Sequencer

Virtual Reality

Blender 2.8 Project

Milestone 1: Basic, Local Asset Browser

Good First Issue

Animation & Rigging

Development Management

EEVEE & Viewport

Nodes & Physics

Pipeline, Assets & IO

Platforms, Builds & Tests

Render & Cycles

Sculpt, Paint & Texture

Needs Info from Developers

Needs Information from User

No Milestone

No project

No Assignees

5 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#108901

No description provided.