Unicode Block Not Working on Windows #71273

New Issue

Mehmet Oguz Derin · 2019-11-01T19:44:01+01:00

Mehmet Oguz Derin commented

2019-11-01 19:44:01 +01:00

System Information
Operating system: Windows 10

Blender Version
Broken: 2.80 Release
Worked: Never

Short description of error
Text objects do not display Old Turkic Unicode block (Old Turkic, U+10C00 - U+10C4F) when using Segoe UI Historic or Google Noto Sans Old Turkic.

Exact steps for others to reproduce the error

Open:
font.blend

Or follow the steps:

Create a text object and name it "Text"
Load either ttf or otf version of either Segoe UI Historic or Google Noto Sans Old Turkic

You can obtain second font from https://www.google.com/get/noto/#sans-orkh

Run the following code

import bpy
bpy.data.objects["Text"].data.body = u'\U00010c45'

The result is a box instead of the desired ? character

Expected result (obtained on Linux):

Developer Notes
The code assumes wchar_t is 32 bit, which is not the case on Windows. See comments for more investigation details.

**System Information** Operating system: Windows 10 **Blender Version** Broken: 2.80 Release Worked: Never **Short description of error** Text objects do not display Old Turkic Unicode block (Old Turkic, U+10C00 - U+10C4F) when using Segoe UI Historic or Google Noto Sans Old Turkic. **Exact steps for others to reproduce the error** Open: [font.blend](https://archive.blender.org/developer/F7952320/font.blend) Or follow the steps: 1. Create a text object and name it "Text" 2. Load either `ttf` or `otf` version of either Segoe UI Historic or Google Noto Sans Old Turkic - You can obtain second font from https://www.google.com/get/noto/#sans-orkh 3. Run the following code ``` import bpy bpy.data.objects["Text"].data.body = u'\U00010c45' ``` 4. The result is a box instead of the desired ? character Expected result (obtained on Linux): ![image.png](https://archive.blender.org/developer/F7952329/image.png) **Developer Notes** The code assumes `wchar_t` is 32 bit, which is not the case on Windows. See comments for more investigation details.

Mehmet Oguz Derin commented

2019-11-01 19:44:01 +01:00

Added subscriber: @mehmetoguzderin

Germano Cavalcante commented

2019-11-04 19:50:59 +01:00

Added subscribers: @mont29, @mano-wii

Germano Cavalcante commented

2019-11-04 19:50:59 +01:00

Analyzing the code I realize that, at least in windows, the supported character encoding is utf-16.
u'\U00010c45' is from a 32-bit hex value that is only supported by utf-32.
A possible solution would be to change the value of the built-in type wchar_t to 4-byte.

If I'm not mistaken this is the size of wchar_t in linux.

I think this is a known limitation.
@mont29, what do you think?

Analyzing the code I realize that, at least in windows, the supported character encoding is utf-16. `u'\U00010c45'` is from a 32-bit hex value that is only supported by utf-32. A possible solution would be to change the value of the built-in type `wchar_t` to 4-byte. If I'm not mistaken this is the size of `wchar_t` in linux. I think this is a known limitation. @mont29, what do you think?

Ray molenkamp commented

2019-11-04 20:02:19 +01:00

Added subscriber: @LazyDodo

Ray molenkamp commented

2019-11-04 20:02:19 +01:00

sizeof(wchar_t) is implementation dependent and cannot be changed for some implementations, C11 added char16_t and char32_t types for if you have size requirements.

`sizeof(wchar_t)` is implementation dependent and cannot be changed for some implementations, C11 added `char16_t` and `char32_t` types for if you have size requirements.

Ray molenkamp commented

2019-11-05 20:33:45 +01:00

Yeahhh this is a mess....

Debugged a little into this, the input to vfont_to_curve is correctly UTF8, however, it then feeds this to BLI_strncpy_wchar_from_utf8 which internally assumes wchar_t is 32 bits which is decisively not the case on windows (16 bits there, cannot be changed. specs say implementation is free to do whatever with wchar_t) so the upper 16 bits get lost and you and up with the wrong codepoint.

Only way to fix this is to replace the parts of the codebase where the assumption is made that wchar_t is 32 bit to use uint32_t instead (I really would have preferred char32_t but msvc does not support this type in c mode)

Yeahhh this is a mess.... Debugged a little into this, the input to `vfont_to_curve` is correctly UTF8, however, it then feeds this to `BLI_strncpy_wchar_from_utf8` which internally assumes wchar_t is 32 bits which is decisively not the case on windows (16 bits there, cannot be changed. specs say implementation is free to do whatever with wchar_t) so the upper 16 bits get lost and you and up with the wrong codepoint. Only way to fix this is to replace the parts of the codebase where the assumption is made that `wchar_t` is 32 bit to use `uint32_t` instead (I really would have preferred `char32_t` but msvc does not support this type in c mode)

Dalai Felinto commented

2019-11-05 22:20:10 +01:00

Added subscriber: @dfelinto

Dalai Felinto commented

2019-11-05 22:20:10 +01:00

Here (Ubuntu Linux, official Blender 2.80) it works fine:
font.blend

Here (Ubuntu Linux, official Blender 2.80) it works fine: [font.blend](https://archive.blender.org/developer/F7952320/font.blend) ![image.png](https://archive.blender.org/developer/F7952329/image.png)

Ray molenkamp commented

2019-11-05 22:23:24 +01:00

wchar_t is 32 bit on gcc, so yeah you wouldn't see the issue there.

Dalai Felinto changed title from ~~Unicode Block Not Working~~ to Unicode Block Not Working on Windows

2019-11-05 22:31:08 +01:00

blender-admin commented

2019-11-22 16:26:54 +01:00

This issue was referenced by 177dfc6384

This issue was referenced by 177dfc6384b926dd19e3b7e98a995ccb4da9167c

Germano Cavalcante commented

2019-11-22 16:28:05 +01:00

Changed status from 'Open' to: 'Resolved'

Germano Cavalcante closed this issue

2019-11-22 16:28:05 +01:00

Germano Cavalcante self-assigned this 2019-11-22 16:28:05 +01:00

Milan Jaros commented

2020-02-17 16:56:41 +01:00

Added subscriber: @MilanJaros

Milan Jaros commented

2020-02-17 16:56:41 +01:00

I have problem with building of this patch on Linux (CentOS) with GCC/4.9 or GCC/7.1 with blender_lite.cmake. I have got the error message: "uchar.h: No such file or directory" in BLI_sys_types.h and in wcwidth.h. After fix it I have got the next error message: "unknown type name size_t" in wcwidth.h and wcwidth.c. After changes on my site in BLI_sys_types.h (uchar.h -> add typedef like for macos), wcwidth.h (size_t -> unsigned long long int) and wcwidth.c (size_t -> unsigned long long int) it works fine.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Unicode Block Not Working on Windows #71273