BLI: Group Deduce functions for GroupedSpan #111081

Iliya Katushenock · 2023-08-13T02:02:41+02:00

Iliya Katushenock commented

2023-08-13 02:02:41 +02:00

This commit add the general implementations of an algorithms to deal with the geometry groups. Group is a set of the same values in a geometry. The values can be randomly distributed in whole domain. Deduced group is a set of an indices to all the same values in geometry. This indices can be used for read, change, write, copy, edit the currect geometry or to create other one.

This commit is based on 'Grouping of random points by it curve index' paper [1].

One of the main advantages of this solution is its stability with a wide range of group sizes and its amount.
The fluctuation range for 10.000.000 points amount is only ~50 milliseconds .
So, the main parameter of the algorithm, to did predict its time costs, is the amount of data, but not data grouping.

Changes:

New BLI header: BLI_group_deduce.hh. The file contains base declarations of algorithm and helpful general tools to deal with the groups. All the implementations are multithreaded. All the implementation are in new group_deduce.cc file. All the algorithms is non-template and consider int as main type.
New IndexMask::from_groups method to construct the set of an IndexMask's to represend all groups. It is generally preferable to use a more general iterator instead of span of indices for most other algorithms.
A Index of Nearest node are redone to use new api.
Benchmark result below.
New function partially is continue of changes from 226359ec48.
Based on exploring of other ways, parallel algorithm, used in 226359ec48, is the best solution for many really small groups. But it can be faster via using tempolar buffer (Copy all group indices to this buffer. Rewrite each one by it result index, instead of write it to actual position. This provide better cache in the main loop).
Solution from 226359ec48 also used in new code, but only for data with really small groups (is_fragmented function is used to deduce this fact).
Benchmark of comparing old-new speed of code for build edge-to-corner maps provided below.
A Points to Curves node are redone to use new api. This allow to reduce a lot of code in node implementation. Old version also was use not the fastest code to build curves (groups) and benchmark of comparing old-new speed are below.

Benchmarks:

Index of Nearest node redone. Replace IndexMask::from_groups by grouped_indices::from_indices and IndexMask::from_indices (implementation of new IndexMask::from_groups). Test for 1.500.000 points. Example File index_of_nearest_groups_bench. Milliseconds. Timings of whole node results.

Max random value Old New Improvement

500000 354 175 2x

50000 256 150 1.6x

5000 180 170 1.0..x

500 280 240 1.10..x

50 430 406 1.0..x

5 846 829 1.0..x
build_edge_to_loop_map and build_vert_to_loop_map functions redone. Tested as part of an Edge Split node. Cube primitive node was used as data. Example File edge_to_corner_map_groups_bench. Milliseconds. Timings of function calls only.

Cuboid vertices Old New Improvement

800x800x800 261 232 1.1x

500x500x500 72 63 1.4x

100x100x100 2 1.8 1.1x
Points to Curves node redone. Tested for 10.000.000 points. Example File points_to_curves_groups_bench. Milliseconds. Timings of group-function calls only.

Max random value Old New Improvement

100000 235 116 1.9x

10000 142 120 1.1x

1000 121 123 1.0...x

100 146 97 1.4x

1 284 67 4.6x

https://hackmd.io/@s0TMIS4lTAGwHVO20ECwpw/points_grouping_methods

This commit add the general implementations of an algorithms to deal with the geometry groups. Group is a set of the same values in a geometry. The values can be randomly distributed in whole domain. Deduced group is a set of an indices to all the same values in geometry. This indices can be used for read, change, write, copy, edit the currect geometry or to create other one. This commit is based on 'Grouping of random points by it curve index' paper [1]. One of the main advantages of this solution is its stability with a wide range of group sizes and its amount. The fluctuation range for 10.000.000 points amount is only ~50 milliseconds . So, the main parameter of the algorithm, to did predict its time costs, is the amount of data, but not data grouping. Changes: 1. New BLI header: `BLI_group_deduce.hh`. The file contains base declarations of algorithm and helpful general tools to deal with the groups. All the implementations are multithreaded. All the implementation are in new `group_deduce.cc` file. All the algorithms is non-template and consider `int` as main type. 2. New `IndexMask::from_groups` method to construct the set of an IndexMask's to represend all groups. It is generally preferable to use a more general iterator instead of span of indices for most other algorithms. 3. A Index of Nearest node are redone to use new api. Benchmark result below. 4. New function partially is continue of changes from 226359ec48c820df59eb948d6d32a5560bffd685. Based on exploring of other ways, parallel algorithm, used in 226359ec48c820df59eb948d6d32a5560bffd685, is the best solution for many really small groups. But it can be faster via using tempolar buffer (Copy all group indices to this buffer. Rewrite each one by it result index, instead of write it to actual position. This provide better cache in the main loop). Solution from 226359ec48c820df59eb948d6d32a5560bffd685 also used in new code, but only for data with really small groups (`is_fragmented` function is used to deduce this fact). Benchmark of comparing old-new speed of code for build edge-to-corner maps provided below. 5. A Points to Curves node are redone to use new api. This allow to reduce a lot of code in node implementation. Old version also was use not the fastest code to build curves (groups) and benchmark of comparing old-new speed are below. Benchmarks: 1. Index of Nearest node redone. Replace `IndexMask::from_groups` by `grouped_indices::from_indices` and `IndexMask::from_indices` (implementation of new `IndexMask::from_groups`). Test for 1.500.000 points. Example File `index_of_nearest_groups_bench`. Milliseconds. Timings of whole node results. | Max random value | Old | New | Improvement | | -- | -- | -- | -- | | 500000 | 354 | 175 | 2x | | 50000 | 256 | 150 | 1.6x | | 5000 | 180 | 170 | 1.0..x | | 500 | 280 | 240 | 1.10..x | | 50 | 430 | 406 | 1.0..x | | 5 | 846 | 829 | 1.0..x | 2. `build_edge_to_loop_map` and `build_vert_to_loop_map` functions redone. Tested as part of an Edge Split node. Cube primitive node was used as data. Example File `edge_to_corner_map_groups_bench`. Milliseconds. Timings of function calls only. | Cuboid vertices | Old | New | Improvement | | -- | -- | -- | -- | | 800x800x800 | 261 | 232 | 1.1x | | 500x500x500 | 72 | 63 | 1.4x | | 100x100x100 | 2 | 1.8 | 1.1x | 3. Points to Curves node redone. Tested for 10.000.000 points. Example File `points_to_curves_groups_bench`. Milliseconds. Timings of group-function calls only. | Max random value | Old | New | Improvement | | -- | -- | -- | -- | | 100000 | 235 | 116 | 1.9x | | 10000 | 142 | 120 | 1.1x | | 1000 | 121 | 123 | 1.0...x | | 100 | 146 | 97 | 1.4x | | 1 | 284 | 67 | 4.6x | --- 1. https://hackmd.io/@s0TMIS4lTAGwHVO20ECwpw/points_grouping_methods

edge_to_corner_map_groups_bench.blend

874 KiB

points_to_curves_groups_bench.blend

880 KiB

index_of_nearest_groups_bench.blend

950 KiB

Iliya Katushenock added 1 commit 2023-08-13 02:02:53 +02:00

71494c65a4 init

Iliya Katushenock added 1 commit 2023-08-13 02:03:53 +02:00

e9b7453cd2 format

Iliya Katushenock added 2 commits 2023-08-14 00:37:04 +02:00

9a7a386a76 Merge branch 'main' into group_deduce

5282801c71 progress

Iliya Katushenock added 2 commits 2023-08-14 00:46:56 +02:00

9a7e49b113 Merge branch 'main' into group_deduce

201cc584f6 format

Iliya Katushenock added 1 commit 2023-08-14 17:31:08 +02:00

09d7dd11b0 progress

Iliya Katushenock added 1 commit 2023-08-15 00:40:04 +02:00

bd8c2791af progress

Iliya Katushenock added 2 commits 2023-08-16 03:03:46 +02:00

a508b24e22 Merge branch 'main' into group_deduce

1022d558b1 progress

Iliya Katushenock added 1 commit 2023-08-16 03:05:28 +02:00

c3aa87f3e4 progress

Iliya Katushenock added 4 commits 2023-08-18 22:48:06 +02:00

7a29aede6e Merge branch 'main' into group_deduce

88f101441a progress

5ff074ce31 Merge branch 'main' into group_deduce

5a7cd16923 progress

Iliya Katushenock added 1 commit 2023-08-18 23:24:08 +02:00

2b55d9119b cleanup

Iliya Katushenock changed title from ~~WIP: BLI: Group Deduce functions for GroupedSpan~~ to BLI: Group Deduce functions for GroupedSpan

2023-08-18 23:24:38 +02:00

Iliya Katushenock added a new dependency 2023-08-19 23:39:00 +02:00

#109610 Geometry Nodes: Points to Curves node

Iliya Katushenock removed a dependency 2023-08-23 22:32:31 +02:00

#109610 Geometry Nodes: Points to Curves node

Iliya Katushenock added 3 commits 2023-08-30 01:53:30 +02:00

7403995535 Merge branch 'main' into group_deduce

4be82ee412 Merge branch 'main' into group_deduce

2edb030f91 progress

Iliya Katushenock added 1 commit 2023-08-30 02:13:45 +02:00

3f6584296a Fix CLI

Iliya Katushenock added 2 commits 2023-09-03 02:23:39 +02:00

70ae4b3dc6 merge main

473fc89e88 fix empty groups case

Iliya Katushenock added this to the Core Libraries project 2023-12-01 15:04:36 +01:00

Iliya Katushenock referenced this pull request

2023-12-03 17:38:58 +01:00

Geometry Nodes: Rewrite Scale Elements node #115142

Iliya Katushenock commented

2023-12-08 14:28:51 +01:00

TODO:

Move all to header / template arg's for all parts of functions.
Handle:
- Too many groups (atomic lists).
- Many groups (need to test if that will be faster than ^).
- Large groups (sort segments and copy part-by-part).
- Too large groups (just try to copy part-by-part until first occure element of other group).
Generalize as BLI_sort.hh implementations of radix sort with specific heuristics?
IndexMask?
More benchmarks with nods which is also can use this function (Scale Elements, Points to Curves, Mesh to Curves, Sort Elements, ...)?
parallel_for_weighted.
SIMD?

TODO: - [ ] Move all to header / template arg's for all parts of functions. - [ ] Handle: - [ ] Too many groups (atomic lists). - [X] Many groups (need to test if that will be faster than ^). - [X] Large groups (sort segments and copy part-by-part). - [ ] Too large groups (just try to copy part-by-part until first occure element of other group). - [ ] Generalize as `BLI_sort.hh` implementations of radix sort with specific heuristics? - [ ] IndexMask? - [ ] More benchmarks with nods which is also can use this function (`Scale Elements`, `Points to Curves`, `Mesh to Curves`, `Sort Elements`, ...)? - [ ] `parallel_for_weighted`. - [ ] SIMD?

Iliya Katushenock referenced this pull request

2023-12-26 14:17:00 +01:00

Improved Surface Deform performance #116545

Iliya Katushenock referenced this pull request

2024-01-06 12:18:42 +01:00

Geometry Nodes: Sort Elements node #114194

Iliya Katushenock referenced this pull request

2024-04-08 13:42:34 +02:00

Geometry Nodes: Rewrite Scale Elements node #115142

Jacques Lucke commented

2024-04-08 17:21:27 +02:00

One thing I don't like about the API right now is that it forces the creation of new integer arrays, even for the trivial cases where everything is in the same group (and other cases).

Maybe an API like the following could allow for more even efficiency. Note that we might also need a good way to iterate over Span and IndexMask in the same way to avoid code duplication.

const IndexMask &selection = ...;
const VArray<int> &group_ids = ...;

foreach_group(selection, group_ids, [&](auto indices) {
  /* #indices is either a #Span<int> or #IndexMask. */
});

One thing I don't like about the API right now is that it forces the creation of new integer arrays, even for the trivial cases where everything is in the same group (and other cases). Maybe an API like the following could allow for more even efficiency. Note that we might also need a good way to iterate over `Span` and `IndexMask` in the same way to avoid code duplication. ``` const IndexMask &selection = ...; const VArray<int> &group_ids = ...; foreach_group(selection, group_ids, [&](auto indices) { /* #indices is either a #Span<int> or #IndexMask. */ }); ```

This pull request has changes conflicting with the target branch.

source/blender/blenkernel/intern/mesh_mapping.cc
source/blender/blenlib/BLI_index_mask.hh
source/blender/nodes/geometry/nodes/node_geo_index_of_nearest.cc

View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u group_deduce:mod_moder-group_deduce

git checkout mod_moder-group_deduce

Sign in to join this conversation.

No reviewers

No Label

Max random value	Old	New	Improvement
500000	354	175	2x
50000	256	150	1.6x
5000	180	170	1.0..x
500	280	240	1.10..x
50	430	406	1.0..x
5	846	829	1.0..x

Max random value	Old	New	Improvement
100000	235	116	1.9x
10000	142	120	1.1x
1000	121	123	1.0...x
100	146	97	1.4x
1	284	67	4.6x

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

BLI: Group Deduce functions for GroupedSpan #111081

Checkout