Fix: operators can have invalid utf-8 strings #129209

Jacques Lucke · 2024-10-18T11:40:28+02:00

Jacques Lucke commented

2024-10-18 11:40:28 +02:00

#129167 showed an issue where code failed because an operator name was unexpectedly not valid utf-8. This patch adds new checks to prevent this from happening when the operator is registered.

When the idname is not valid utf-8, the registration will fail with an error.
If the name, description or translation_context is invalid utf-8, it will be fixed automatically and the registration succeeds. There is a warning though. Failing operator registration in this case could break existing add-ons, so better handle the situation more gracefully.

#129167 showed an issue where code failed because an operator name was unexpectedly not valid utf-8. This patch adds new checks to prevent this from happening when the operator is registered. * When the `idname` is not valid utf-8, the registration will fail with an error. * If the `name`, `description` or `translation_context` is invalid utf-8, it will be fixed automatically and the registration succeeds. There is a warning though. Failing operator registration in this case could break existing add-ons, so better handle the situation more gracefully.

Jacques Lucke added 1 commit 2024-10-18 11:40:38 +02:00

ensure utf8

buildbot/vexp-code-patch-lint Build done.

Details

buildbot/vexp-code-patch-linux-x86_64 Build done.

Details

buildbot/vexp-code-patch-darwin-arm64 Build done.

Details

buildbot/vexp-code-patch-darwin-x86_64 Build done.

Details

buildbot/vexp-code-patch-windows-amd64 Build done.

Details

buildbot/vexp-code-patch-coordinator Build done.

Details

8c621709de

Jacques Lucke commented

2024-10-18 11:41:24 +02:00

@blender-bot build

Jacques Lucke requested review from Bastien Montagne 2024-10-18 11:41:35 +02:00

Jacques Lucke referenced this pull request

2024-10-18 11:42:46 +02:00

fix search string in a endless loop #129167

Bastien Montagne approved these changes 2024-10-18 11:58:58 +02:00

Bastien Montagne left a comment

LGTM, will add @ideasman42 for FYI and in case he has better ideas to solve this?

Also wonder if other type of registerable data (like Panels, UILists...) do not have the same issue currently?

LGTM, will add @ideasman42 for FYI and in case he has better ideas to solve this? Also wonder if other type of registerable data (like Panels, UILists...) do not have the same issue currently?

Bastien Montagne requested review from Campbell Barton 2024-10-18 11:59:08 +02:00

Campbell Barton requested changes 2024-10-18 12:31:03 +02:00

Campbell Barton left a comment

Is there a simple test case to validate this?

If I try to reproduce the error using the input from the report:

diff --git a/scripts/startup/bl_operators/file.py b/scripts/startup/bl_operators/file.py
index 08c2cde1146..e7e7b32f5f6 100644
--- a/scripts/startup/bl_operators/file.py
+++ b/scripts/startup/bl_operators/file.py
@@ -19,7 +19,13 @@ from bpy.app.translations import pgettext_rpt as rpt_
 
 class WM_OT_previews_batch_generate(Operator):
     """Generate selected .blend file's previews"""
-    bl_idname = "wm.previews_batch_generate"
+    bl_idname = (
+        "\xe7\x84\xb6\xe5\x90\x8e\xe5\x9c"
+        "\xa8\xe6\xa8\xa1\xe6\x80\x81\xe6"
+        "\xa8\xa1\xe5\xbc\x8f\xe4\xb8\x8b"
+        "\xe7\x9b\xb4\xe6\x8e\xa5\xe6\x9b"
+        "\xb4\xe6\x94\xb9\xe4"
+    )
     bl_label = "Batch-Generate Previews"
     bl_options = {'REGISTER'}

Blender fails to register the class with the following error:

Traceback (most recent call last):                                     
  File "/src/cmake_debug/bin/4.4/scripts/modules/bpy/utils/__init__.py", line 208, in _register_module_call                                    
    register()                                                         
    ~~~~~~~~^^                                                         
  File "/src/cmake_debug/bin/4.4/scripts/startup/bl_operators/__init__.py", line 64, in register                                               
    register_class(cls)                                                
    ~~~~~~~~~~~~~~^^^^^                                                
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 149: invalid continuation byte

Requesting changes because it would be good to have a reproducible test case, so it's possible to know if the same problem occurs elsewhere.

Is there a simple test case to validate this? If I try to reproduce the error using the input from the report: ```diff diff --git a/scripts/startup/bl_operators/file.py b/scripts/startup/bl_operators/file.py index 08c2cde1146..e7e7b32f5f6 100644 --- a/scripts/startup/bl_operators/file.py +++ b/scripts/startup/bl_operators/file.py @@ -19,7 +19,13 @@ from bpy.app.translations import pgettext_rpt as rpt_ class WM_OT_previews_batch_generate(Operator): """Generate selected .blend file's previews""" - bl_idname = "wm.previews_batch_generate" + bl_idname = ( + "\xe7\x84\xb6\xe5\x90\x8e\xe5\x9c" + "\xa8\xe6\xa8\xa1\xe6\x80\x81\xe6" + "\xa8\xa1\xe5\xbc\x8f\xe4\xb8\x8b" + "\xe7\x9b\xb4\xe6\x8e\xa5\xe6\x9b" + "\xb4\xe6\x94\xb9\xe4" + ) bl_label = "Batch-Generate Previews" bl_options = {'REGISTER'} ``` Blender fails to register the class with the following error: ``` Traceback (most recent call last): File "/src/cmake_debug/bin/4.4/scripts/modules/bpy/utils/__init__.py", line 208, in _register_module_call register() ~~~~~~~~^^ File "/src/cmake_debug/bin/4.4/scripts/startup/bl_operators/__init__.py", line 64, in register register_class(cls) ~~~~~~~~~~~~~~^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 149: invalid continuation byte ``` Requesting changes because it would be good to have a reproducible test case, so it's possible to know if the same problem occurs elsewhere.

dereck-4 commented

2024-10-18 16:24:38 +02:00

Is there a simple test case to validate this?

If I try to reproduce the error using the input from the report:

diff --git a/scripts/startup/bl_operators/file.py b/scripts/startup/bl_operators/file.py
index 08c2cde1146..e7e7b32f5f6 100644
--- a/scripts/startup/bl_operators/file.py
+++ b/scripts/startup/bl_operators/file.py
@@ -19,7 +19,13 @@ from bpy.app.translations import pgettext_rpt as rpt_
 
 class WM_OT_previews_batch_generate(Operator):
     """Generate selected .blend file's previews"""
-    bl_idname = "wm.previews_batch_generate"
+    bl_idname = (
+        "\xe7\x84\xb6\xe5\x90\x8e\xe5\x9c"
+        "\xa8\xe6\xa8\xa1\xe6\x80\x81\xe6"
+        "\xa8\xa1\xe5\xbc\x8f\xe4\xb8\x8b"
+        "\xe7\x9b\xb4\xe6\x8e\xa5\xe6\x9b"
+        "\xb4\xe6\x94\xb9\xe4"
+    )
     bl_label = "Batch-Generate Previews"
     bl_options = {'REGISTER'}

Blender fails to register the class with the following error:

Traceback (most recent call last):                                     
  File "/src/cmake_debug/bin/4.4/scripts/modules/bpy/utils/__init__.py", line 208, in _register_module_call                                    
    register()                                                         
    ~~~~~~~~^^                                                         
  File "/src/cmake_debug/bin/4.4/scripts/startup/bl_operators/__init__.py", line 64, in register                                               
    register_class(cls)                                                
    ~~~~~~~~~~~~~~^^^^^                                                
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 149: invalid continuation byte

Requesting changes because it would be good to have a reproducible test case, so it's possible to know if the same problem occurs elsewhere.

So It is really strange,I found if I directly print the translate content in the python's print,It is OK,while in the search function ,it is an invalid string.
the operator's idname is scatter5.add_psy_modal

> Is there a simple test case to validate this? > > If I try to reproduce the error using the input from the report: > > ```diff > diff --git a/scripts/startup/bl_operators/file.py b/scripts/startup/bl_operators/file.py > index 08c2cde1146..e7e7b32f5f6 100644 > --- a/scripts/startup/bl_operators/file.py > +++ b/scripts/startup/bl_operators/file.py > @@ -19,7 +19,13 @@ from bpy.app.translations import pgettext_rpt as rpt_ > > class WM_OT_previews_batch_generate(Operator): > """Generate selected .blend file's previews""" > - bl_idname = "wm.previews_batch_generate" > + bl_idname = ( > + "\xe7\x84\xb6\xe5\x90\x8e\xe5\x9c" > + "\xa8\xe6\xa8\xa1\xe6\x80\x81\xe6" > + "\xa8\xa1\xe5\xbc\x8f\xe4\xb8\x8b" > + "\xe7\x9b\xb4\xe6\x8e\xa5\xe6\x9b" > + "\xb4\xe6\x94\xb9\xe4" > + ) > bl_label = "Batch-Generate Previews" > bl_options = {'REGISTER'} > ``` > > Blender fails to register the class with the following error: > > ``` > Traceback (most recent call last): > File "/src/cmake_debug/bin/4.4/scripts/modules/bpy/utils/__init__.py", line 208, in _register_module_call > register() > ~~~~~~~~^^ > File "/src/cmake_debug/bin/4.4/scripts/startup/bl_operators/__init__.py", line 64, in register > register_class(cls) > ~~~~~~~~~~~~~~^^^^^ > UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 149: invalid continuation byte > ``` > > Requesting changes because it would be good to have a reproducible test case, so it's possible to know if the same problem occurs elsewhere. So It is really strange,I found if I directly print the translate content in the python's print,It is OK,while in the search function ,it is an invalid string. the operator's idname is scatter5.add_psy_modal

Jacques Lucke commented

2024-10-18 17:16:18 +02:00

This reproduces the issue for me. I don't quite get what is special about this string yet. For Python it seems to be all fine. I can encode as utf8 and decode without problem and the before/after are equal. Maybe it's also an issue in our utf8 processing?

diff --git a/scripts/startup/bl_operators/file.py b/scripts/startup/bl_operators/file.py
index 08c2cde1146..6203646d094 100644
--- a/scripts/startup/bl_operators/file.py
+++ b/scripts/startup/bl_operators/file.py
@@ -20,7 +20,7 @@ from bpy.app.translations import pgettext_rpt as rpt_
 class WM_OT_previews_batch_generate(Operator):
     """Generate selected .blend file's previews"""
     bl_idname = "wm.previews_batch_generate"
-    bl_label = "Batch-Generate Previews"
+    bl_label = "快速散佈所選物件, 然後在模態模式下直接更改一些設定。"
     bl_options = {'REGISTER'}
 
     # -----------

This reproduces the issue for me. I don't quite get what is special about this string yet. For Python it seems to be all fine. I can encode as utf8 and decode without problem and the before/after are equal. Maybe it's also an issue in our utf8 processing? ```diff diff --git a/scripts/startup/bl_operators/file.py b/scripts/startup/bl_operators/file.py index 08c2cde1146..6203646d094 100644 --- a/scripts/startup/bl_operators/file.py +++ b/scripts/startup/bl_operators/file.py @@ -20,7 +20,7 @@ from bpy.app.translations import pgettext_rpt as rpt_ class WM_OT_previews_batch_generate(Operator): """Generate selected .blend file's previews""" bl_idname = "wm.previews_batch_generate" - bl_label = "Batch-Generate Previews" + bl_label = "快速散佈所選物件, 然後在模態模式下直接更改一些設定。" bl_options = {'REGISTER'} # ----------- ```

dereck-4 commented

2024-10-18 23:44:39 +02:00

This reproduces the issue for me. I don't quite get what is special about this string yet. For Python it seems to be all fine. I can encode as utf8 and decode without problem and the before/after are equal. Maybe it's also an issue in our utf8 processing?

diff --git a/scripts/startup/bl_operators/file.py b/scripts/startup/bl_operators/file.py
index 08c2cde1146..6203646d094 100644
--- a/scripts/startup/bl_operators/file.py
+++ b/scripts/startup/bl_operators/file.py
@@ -20,7 +20,7 @@ from bpy.app.translations import pgettext_rpt as rpt_
 class WM_OT_previews_batch_generate(Operator):
     """Generate selected .blend file's previews"""
     bl_idname = "wm.previews_batch_generate"
-    bl_label = "Batch-Generate Previews"
+    bl_label = "快速散佈所選物件, 然後在模態模式下直接更改一些設定。"
     bl_options = {'REGISTER'}
 
     # -----------

Yes,you are right.I think the string in blender has a max length,and the hard cut logic cut a utf8 string with an end of a incomplete character,so the window_end is larger then full_end,because window_end is calculated by the first 8 bit of the character.I am not sure if it is the problem.

> This reproduces the issue for me. I don't quite get what is special about this string yet. For Python it seems to be all fine. I can encode as utf8 and decode without problem and the before/after are equal. Maybe it's also an issue in our utf8 processing? > > ```diff > diff --git a/scripts/startup/bl_operators/file.py b/scripts/startup/bl_operators/file.py > index 08c2cde1146..6203646d094 100644 > --- a/scripts/startup/bl_operators/file.py > +++ b/scripts/startup/bl_operators/file.py > @@ -20,7 +20,7 @@ from bpy.app.translations import pgettext_rpt as rpt_ > class WM_OT_previews_batch_generate(Operator): > """Generate selected .blend file's previews""" > bl_idname = "wm.previews_batch_generate" > - bl_label = "Batch-Generate Previews" > + bl_label = "快速散佈所選物件, 然後在模態模式下直接更改一些設定。" > bl_options = {'REGISTER'} > > # ----------- > > ``` Yes,you are right.I think the string in blender has a max length,and the hard cut logic cut a utf8 string with an end of a incomplete character,so the window_end is larger then full_end,because window_end is calculated by the first 8 bit of the character.I am not sure if it is the problem.

无标题.png

72 KiB

Campbell Barton commented

2024-10-25 05:40:51 +02:00

Committed alternate fix 1d6add574d, closing.

Committed alternate fix 1d6add574d7547cc76217040b49075bad8c7f9e7, closing.

Campbell Barton closed this pull request

2024-10-25 05:40:54 +02:00

buildbot/vexp-code-patch-lint Build done.

Details

buildbot/vexp-code-patch-linux-x86_64 Build done.

Details

buildbot/vexp-code-patch-darwin-arm64 Build done.

Details

buildbot/vexp-code-patch-darwin-x86_64 Build done.

Details

buildbot/vexp-code-patch-windows-amd64 Build done.

Details

buildbot/vexp-code-patch-coordinator Build done.

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Fix: operators can have invalid utf-8 strings #129209

Pull request closed