This function was documented to return the length but returned an
error value for WIN32. While this doesn't cause any bugs at the moment,
it could cause problems in the future.
Oversight in 5496d8cd36.
Besides helping to avoid buffer overflow errors this reduces complexity
of BLI_str_utf32_as_utf8 which needed a special loop for the last 6
characters to avoid writing past the buffer bounds.
Also add BLI_str_utf8_from_unicode_len which only returns the length.
Various changes to reduce risk of out of bounds errors in utf8 seeking.
- Remove BLI_str_prev_char_utf8
This function could potentially scan past the beginning of a string.
Use BLI_str_find_prev_char_utf8 instead which takes a limiting
string start argument.
- Swap arguments for BLI_str_find_prev_char_utf8 so the stepping
argument is first and the limiting argument is last.
This matches BLI_str_find_next_char_utf8.
- Change behavior of these functions to return it the start or end
pointers instead of NULL, which complicated use of these functions
to calculate offsets.
Callers that need to check if the limits were reached can compare
the return value with the start/end pointers.
- Return 'const char *' from these functions
so they don't remove const from the input arguments.
Remove BLI_str_utf8_as_unicode_and_size and
BLI_str_utf8_as_unicode_and_size_safe.
Use BLI_str_utf8_as_unicode_step instead since it takes
a buffer bounds argument to prevent buffer over-reading.
There were multiple utf8 functions which treated
errors slightly differently.
Split BLI_str_utf8_as_unicode_step into two functions.
- BLI_str_utf8_as_unicode_step_or_error returns error value
when decoding fails and doesn't step.
- BLI_str_utf8_as_unicode_step always steps forward at least one
returning the byte value without decoding
(needed to display some latin1 file-paths).
Font drawing uses BLI_str_utf8_as_unicode_step and no longer
check for error values.
Add a string length argument to BLI_str_utf8_as_unicode_step to prevent
reading past the buffer bounds or the intended range since some callers
of this function take a string length to operate on part of the string.
Font drawing for example didn't respect the length argument,
potentially causing a buffer over-read with multi-byte characters
that could read past the end of the string.
The following command would read 5 bytes past the end of the input.
`BLF_draw(font_id, (char[]){252}, 1);`
In practice strings are typically null terminated so this didn't crash
reading past buffer bounds.
Nevertheless, this wasn't correct and could cause bugs in the future.
Clamping by the length now has the same behavior as a null byte.
Add test to ensure this is working as intended.
BLI_strncpy_wchar_from_utf8 made the assumption that
wchar_t is UTF-32 bit regardless of environment, while
this holds true on both mac and linux, on windows
wchar_t is actually actually UTF-16.
This resulted in the upper 16 bits being dropped from
from some string conversions and prevented blender
from starting when installed in a path with unicode
code-points over 0xffff.
There was also a fair bit of code duplication between
BLI_strncpy_wchar_from_utf8 and BLI_str_utf8_as_unicode_and_size
this change essentially removes all logic from
BLI_strncpy_wchar_from_utf8 and calls the right function
for the right environment.
Reviewed By: brecht , Robert Guetzkow
Differential Revision: https://developer.blender.org/D9822
This addresses warnings from Clang-Tidy's `readability-else-after-return`
rule in the `source/blender/blenlib` module. Not all warnings are
addressed in this commit.
No functional changes.
`BLI_strncpy_wchar_from_utf8` internally assumes `wchar_t` is 32 bits
which is not the case on windows.
The solution is to replace `wchar_t` with `char32_t`.
Thanks to @robbott for compatibility on macOS.
Differential Revision: https://developer.blender.org/D6198
BF-admins agree to remove header information that isn't useful,
to reduce noise.
- BEGIN/END license blocks
Developers should add non license comments as separate comment blocks.
No need for separator text.
- Contributors
This is often invalid, outdated or misleading
especially when splitting files.
It's more useful to git-blame to find out who has developed the code.
See P901 for script to perform these edits.
That code was a joke, letting some invalid utf8 bytes pass, returning
wrong offset for some invalid sequences, not to mention length and
pointer easily going out of sync, NULL final byte being 'forgotten' by
memcpy, etc. etc.
The miracle here is that we could survive using this for so long!
Probably because we do not use utf-8 sanitizing enough in Blender,
actually... :/
Own mistake in refactoring of `BLI_strncpy_wchar_as_utf8()`, if given size was exactly
the one needed, we'd lost last char (off-by-one error).
Many thanks to plasmasolutions (Thomas Beck) who found the issue and did
all the investigation work here!
