Libraries: integrate xxHash library for fast hashing #120139

Jacques Lucke · 2024-04-01T11:51:11+02:00

Jacques Lucke commented

2024-04-01 11:51:11 +02:00

xxHash is a fast non-cryptographic hashing library. It significantly outperforms md5 which we use in some places currently while also having great collision resistance if not attacked explicitly.

The library is added to extern because that was the easiest way to do it and has the least impact on others. I expect this library to become a required dependency instead of an optional one. It's licence is BSD 2-Clause which seems to be the first of its kind in Blender (there is BSD 3-Clause a couple of times). From what I can tell, this licence is compatible with ours.

For now, I used the library only for data deduplication when baking geometry nodes. The time to bake the setup below goes down from >6s to <1s (note that this includes more than just the hashing time).

[xxHash](https://xxhash.com/) is a fast non-cryptographic hashing library. It significantly outperforms md5 which we use in some places currently while also having great [collision resistance](https://github.com/Cyan4973/xxHash/tree/dev/tests/collisions) if not attacked explicitly. The library is added to `extern` because that was the easiest way to do it and has the least impact on others. I expect this library to become a required dependency instead of an optional one. It's licence is `BSD 2-Clause` which seems to be the first of its kind in Blender (there is `BSD 3-Clause` a couple of times). From what I can tell, this licence is compatible with ours. For now, I used the library only for data deduplication when baking geometry nodes. The time to bake the setup below goes down from >6s to <1s (note that this includes more than just the hashing time). ![image](/attachments/71003c96-8c83-4bb0-9948-25b175883633)

image.png

36 KiB

image.png

45 KiB

👍 1 ❤️ 3 🎉 1 🚀 3

Jacques Lucke added 1 commit 2024-04-01 11:51:16 +02:00

fa319928c3 integrate xxhash

Jacques Lucke added 1 commit 2024-04-01 12:00:38 +02:00

buildbot/vexp-code-patch-lint Build done. Details

buildbot/vexp-code-patch-linux-x86_64 Build done. Details

buildbot/vexp-code-patch-darwin-x86_64 Build done. Details

buildbot/vexp-code-patch-darwin-arm64 Build done. Details

buildbot/vexp-code-patch-windows-amd64 Build done. Details

buildbot/vexp-code-patch-coordinator Build done. Details

5e11b04994 use XXH3_64bits

Jacques Lucke commented

2024-04-01 12:01:27 +02:00

@blender-bot build

Jacques Lucke requested review from Brecht Van Lommel 2024-04-01 12:26:58 +02:00

Jacques Lucke requested review from Ray molenkamp 2024-04-01 12:26:59 +02:00

Jacques Lucke requested review from Hans Goudey 2024-04-01 12:26:59 +02:00

Jacques Lucke requested review from Aras Pranckevicius 2024-04-01 12:26:59 +02:00

Aras Pranckevicius reviewed 2024-04-01 16:22:16 +02:00

extern/xxhash/README.blender Outdated

						
				@ -0,0 +1,6 @@

				Project: xxHash

				URL: https://xxhash.com/

				License: BSD 2-Clause

				Upstream version: 2024-04-01

Aras Pranckevicius commented

2024-04-01 16:22:16 +02:00

Minor: for me it's clearer if version also indicates the commit and/or release version. Like with 2024-04-01 it's not clear whether it is d28b0bf from dev branch (2024-03-24) or bbb27a5 from release branch (2023-07-21, also shipped as version v0.8.2)

Minor: for me it's clearer if version also indicates the commit and/or release version. Like with 2024-04-01 it's not clear whether it is `d28b0bf` from dev branch (2024-03-24) or `bbb27a5` from release branch (2023-07-21, also shipped as version v0.8.2)

👍 1

JacquesLucke marked this conversation as resolved

Aras Pranckevicius approved these changes 2024-04-01 16:24:19 +02:00

Dismissed

Aras Pranckevicius left a comment

Nice! One minor comment wrt which version exactly is used from upstream.

Would be good for someone to look at some point whether this could be used in some other places where today some slow(er) hash functions are used.

Nice! One minor comment wrt which version exactly is used from upstream. Would be good for someone to look at some point whether this could be used in some other places where today some slow(er) hash functions are used.

Ray molenkamp approved these changes 2024-04-01 17:08:09 +02:00

Ray molenkamp left a comment

I agree with @aras_p here, please provide either the exact version or hash you used, also gitea doesn't have a blocking reviewer concept but @brecht or someone else from core/admin has to accept this before landing, do not land without this.

also cc: @ThomasDinges as he maintains the compliance documentation.

I agree with @aras_p here, please provide either the exact version or hash you used, also gitea doesn't have a blocking reviewer concept but @brecht or someone else from core/admin _has_ to accept this before landing, do not land without this. also cc: @ThomasDinges as he maintains the compliance documentation.

👍 1

Jacques Lucke added 1 commit 2024-04-01 18:07:50 +02:00

buildbot/vexp-code-patch-lint Build done. Details

buildbot/vexp-code-patch-linux-x86_64 Build done. Details

buildbot/vexp-code-patch-darwin-x86_64 Build done. Details

buildbot/vexp-code-patch-darwin-arm64 Build done. Details

buildbot/vexp-code-patch-windows-amd64 Build done. Details

buildbot/vexp-code-patch-coordinator Build done. Details

762c3b04aa use latest xxhash release (0.8.2) instead of dev branch

Jacques Lucke commented

2024-04-01 18:12:50 +02:00

Would be good for someone to look at some point whether this could be used in some other places where today some slow(er) hash functions are used.

I think this could at least replace all of our usages of md5 which we could potentially also remove. It might also replace our usages of murmur hash, but that needs to be checked in more detail.

> Would be good for someone to look at some point whether this could be used in some other places where today some slow(er) hash functions are used. I think this could at least replace all of our usages of `md5` which we could potentially also remove. It might also replace our usages of murmur hash, but that needs to be checked in more detail.

👍 2

Jacques Lucke commented

2024-04-01 19:08:14 +02:00

@blender-bot build

Aras Pranckevicius commented

2024-04-02 06:19:28 +02:00

I think this could at least replace all of our usages of md5 which we could potentially also remove. It might also replace our usages of murmur hash, but that needs to be checked in more detail.

Hardest part is usually knowing whether "something" depends on exact hash values not changing, ever. Like if the hashes are used to generate some number identifiers that are then saved into files, etc. If it is strictly just runtime hashtables or some serialized caches that by design can be "regenerated", it is easier -- but you have to look and determine if that is the case or not.

Last time I did hash performance comparisons was 8 years ago, back then xxHash (before XXH3 existed) was generally faster than murmur hash family, including for very small data. So I suspect XXH3 might be even better, considering that one of the "things" about XXH3 is how it is faster to compute for small data sizes.

I suspect hash_string might also be a candidate for replacement, but probably needs to get evaluated what are the usual string lengths that it is called on. For very short strings the current inline function might be better, even if it hashes one byte at a time.

> I think this could at least replace all of our usages of `md5` which we could potentially also remove. It might also replace our usages of murmur hash, but that needs to be checked in more detail. Hardest part is usually knowing whether "something" depends on exact hash values not changing, ever. Like if the hashes are used to generate some number identifiers that are then saved into files, etc. If it is strictly just runtime hashtables or some serialized caches that by design can be "regenerated", it is easier -- but you have to look and determine if that is the case or not. Last time I did [hash performance comparisons](https://aras-p.info/blog/2016/08/09/More-Hash-Function-Tests/) was 8 years ago, back then xxHash (before XXH3 existed) was generally faster than murmur hash family, including for very small data. So I suspect XXH3 might be even better, considering that one of the "things" about XXH3 is how it is faster to compute for small data sizes. I suspect `hash_string` might also be a candidate for replacement, but probably needs to get evaluated what are the usual string lengths that it is called on. For *very* short strings the current inline function might be better, even if it hashes one byte at a time.

👍 1

Aras Pranckevicius approved these changes 2024-04-02 08:31:58 +02:00

Hans Goudey approved these changes 2024-04-02 13:56:33 +02:00

Jacques Lucke requested review from Thomas Dinges 2024-04-02 16:19:52 +02:00

Jacques Lucke requested review from Sergey Sharybin 2024-04-02 16:20:16 +02:00

Sergey Sharybin commented

2024-04-03 09:59:39 +02:00

I think its fine to have this library. Having it in extern also minimizes an impact on people who do not use precompiled libraries.

To be fully ready the remaining part is to add release/license/BSD-2-Clause.txt.

I think its fine to have this library. Having it in `extern` also minimizes an impact on people who do not use precompiled libraries. To be fully ready the remaining part is to add `release/license/BSD-2-Clause.txt`.

🎉 1

Jacques Lucke added 2 commits 2024-04-03 10:05:30 +02:00

c82f2950b4 Merge branch 'main' into xxhash

a872131408 add BSD-2-Clause license

Sergey Sharybin approved these changes 2024-04-03 10:14:56 +02:00

Jacques Lucke merged commit e2d170f685 into main

2024-04-03 10:23:04 +02:00

Jacques Lucke referenced this issue from a commit

2024-04-03 10:23:06 +02:00

Libraries: integrate xxHash library for fast hashing

Jacques Lucke deleted branch xxhash

2024-04-03 10:23:07 +02:00