Registering more than 3070 operators caused overriden built-in operators to fail #105063

Closed
opened 2023-02-22 05:00:07 +01:00 by Dion Moult · 13 comments

If you register more than 3070 operators via bpy.utils.register_class, then an operator overriding a built-in operator stops working (i.e. doesn't run at all, no error message).

First, run this script to do a count of registered operators:

import bpy
import types

total = 0

def count_operators(obj):
    global total
    for name in dir(obj):
        subthing = getattr(obj, name)
        if isinstance(subthing, types.ModuleType):
            count_operators(subthing)
        else:
            total += 1
        
count_operators(bpy.ops)
print(total)

This gives me 2154 operators on my machine. This may be different on your machine depending if you have add-ons enabled.

I've determined the threshold where operators stop working at 3070.

See the attached buggysmall.py add-on with minimal code. The add-on simply contains one OverrideDelete operator, with the bl_idname = "object.delete" (i.e. the same operator that runs when you press "x" or delete an object). It also contains another Asdf1 operator that does absolutely nothing (i.e. just return {"FINISHED"} immediately).

If you install / enable the buggysmall.py the add-on, as expected when you delete an object, the override will be activated and instead of the object being deleted, you'll get a message printed in the console. This works completely as expected and demonstrates correct behaviour.

However, if you install / enable the buggy.py add-on, this is almost the same as buggysmall.py except it has 1000 operators that do absolutely nothing instead of 1. On my machine, 2154 + 1000 > 3070, so if you enable this and try to delete an object, the object will be deleted (i.e. the OverrideDelete operator will stop working).

I came across this when developing the BlenderBIM Add-on - it's a very, very large and complex add-on with about 900 operators so far and growing, so users have started reporting bugs if they have other add-ons installed that hit this limit.

If you register more than 3070 operators via bpy.utils.register_class, then an operator overriding a built-in operator stops working (i.e. doesn't run at all, no error message). First, run this script to do a count of registered operators: ```python import bpy import types total = 0 def count_operators(obj): global total for name in dir(obj): subthing = getattr(obj, name) if isinstance(subthing, types.ModuleType): count_operators(subthing) else: total += 1 count_operators(bpy.ops) print(total) ``` This gives me 2154 operators on my machine. This may be different on your machine depending if you have add-ons enabled. I've determined the threshold where operators stop working at 3070. See the attached `buggysmall.py` add-on with minimal code. The add-on simply contains one `OverrideDelete` operator, with the bl_idname = "object.delete" (i.e. the same operator that runs when you press "x" or delete an object). It also contains another `Asdf1` operator that does absolutely nothing (i.e. just return {"FINISHED"} immediately). If you install / enable the `buggysmall.py` the add-on, as expected when you delete an object, the override will be activated and instead of the object being deleted, you'll get a message printed in the console. This works completely as expected and demonstrates correct behaviour. However, if you install / enable the `buggy.py` add-on, this is almost the same as `buggysmall.py` except it has 1000 operators that do absolutely nothing instead of 1. On my machine, 2154 + 1000 > 3070, so if you enable this and try to delete an object, the object will be deleted (i.e. the `OverrideDelete` operator will stop working). I came across this when developing the BlenderBIM Add-on - it's a very, very large and complex add-on with about 900 operators so far and growing, so users have started reporting bugs if they have other add-ons installed that hit this limit.
Dion Moult added the
Priority
Normal
Type
Report
Status
Needs Triage
labels 2023-02-22 05:00:08 +01:00

None of those overrides work for me, instead in a debug build I get an assert in

BLI_ghash_insert(global_ops_hash, (void *)ot->idname, ot);

at

BLI_assert((gh->flag & GHASH_FLAG_ALLOW_DUPES) || (BLI_ghash_haskey(gh, key) == 0));

which seems to suggest it's not meant to work to override in this way.

By adding this code above the ghash insert line above in wm_operatortype_append__end, it removes any old operator first and it works:

  if (BLI_ghash_haskey(global_ops_hash, (void *)ot->idname)) {
    WM_operatortype_remove_ptr(BLI_ghash_lookup(global_ops_hash, (void *)ot->idname));
  }

But I'm not sure if this is correct, someone with more experience will have to check this.

None of those overrides work for me, instead in a debug build I get an assert in `BLI_ghash_insert(global_ops_hash, (void *)ot->idname, ot);` at `BLI_assert((gh->flag & GHASH_FLAG_ALLOW_DUPES) || (BLI_ghash_haskey(gh, key) == 0));` which seems to suggest it's not meant to work to override in this way. By adding this code above the ghash insert line above in `wm_operatortype_append__end`, it removes any old operator first and it works: ``` if (BLI_ghash_haskey(global_ops_hash, (void *)ot->idname)) { WM_operatortype_remove_ptr(BLI_ghash_lookup(global_ops_hash, (void *)ot->idname)); } ``` But I'm not sure if this is correct, someone with more experience will have to check this.
Author

@erik85 that seems odd, because the override has an effect for me as well as other users who are using the add-on. Here's also a SO post describing how they use an override: https://blender.stackexchange.com/questions/184245/how-to-re-register-a-built-in-operator

So does Blender not allow us to override built-ins?

@erik85 that seems odd, because the override has an effect for me as well as other users who are using the add-on. Here's also a SO post describing how they use an override: https://blender.stackexchange.com/questions/184245/how-to-re-register-a-built-in-operator So does Blender not allow us to override built-ins?
Iliya Katushenock added
Module
Python API
Status
Needs Info from Developers
and removed
Status
Needs Triage
labels 2023-02-22 11:26:41 +01:00

Added a core as I'm not sure who is responsible for the built-in operators

Added a core as I'm not sure who is responsible for the built-in operators
Iliya Katushenock added the
Interest
Core
label 2023-02-22 11:33:19 +01:00
Contributor

Made some other script that you can just run in blender text editor to identify the operators limit.

Blender 3.4, Windows 11 it starts with 2122 operators (without any addons enabled) and at register of 3075th (what an interesting number) overriding built-in operators stops working and throws an error below.

Error: Python: Traceback (most recent call last):
  File "\Text", line 6072, in <module>
  File "\Text", line 6064, in register
RuntimeError: Error: Type identifier 'OBJECT_OT_delete' is already in use: 'OBJECT_OT_delete(Operator)'.

Btw there is no limit for non built-in operators. I've replaced object.delete with test.test in the attached script and managed to install 5000+ operators and it still was working.

@erik85 how many operators are there in debug build? Maybe there is some extra operators and this is why it hits the limit right away?

Just tested it on both 3.5 and 3.6 latest builds (3.5 c760248fc5 and 3.6 e497a50aa2) - same result, it stops working at registration of 3075th operator.

Made some other script that you can just run in blender text editor to identify the operators limit. Blender 3.4, Windows 11 it starts with 2122 operators (without any addons enabled) and at register of 3075th (what an interesting number) overriding built-in operators stops working and throws an error below. ``` Error: Python: Traceback (most recent call last): File "\Text", line 6072, in <module> File "\Text", line 6064, in register RuntimeError: Error: Type identifier 'OBJECT_OT_delete' is already in use: 'OBJECT_OT_delete(Operator)'. ``` Btw there is no limit for non built-in operators. I've replaced `object.delete` with `test.test` in the attached script and managed to install 5000+ operators and it still was working. @erik85 how many operators are there in debug build? Maybe there is some extra operators and this is why it hits the limit right away? Just tested it on both 3.5 and 3.6 latest builds (3.5 c760248fc5aa and 3.6 e497a50aa207) - same result, it stops working at registration of 3075th operator.

The assert that triggers on duplicate operator names is a debug assert and is only active in Debug builds which is why you don't see erik85's particular crash on normal builds.

That said, the system does indeed not expect more than one operator with the same name to be added. Imagine 2 addons that attempt to add their own operator like OBJECT_OT_delete (a 3rd if you count Blender itself). Which one wins?

Even if things kind of work, there is no telling what other Addons a user of yours might load, before or after, your addon. Confusion will result and lots of time will be wasted determining what is going on.

The assert that triggers on duplicate operator names is a debug assert and is only active in Debug builds which is why you don't see erik85's particular crash on normal builds. That said, the system does indeed not expect more than one operator with the same name to be added. Imagine 2 addons that attempt to add their own operator like OBJECT_OT_delete (a 3rd if you count Blender itself). Which one wins? Even if things kind of work, there is no telling what other Addons a user of yours might load, before or after, your addon. Confusion will result and lots of time will be wasted determining what is going on.

In the end this seems like a question about if built in operators should be overridable or not. Personally I think it would make sense to be able to "break" Blender if you want by overriding.

In the end this seems like a question about if built in operators should be overridable or not. Personally I think it would make sense to be able to "break" Blender if you want by overriding.
Contributor

Is there any reasons to prevent overriding built-ins? If user installed Blender addon that's changing Blender core functionality - isn't it part of the deal?

On other hand allowing overriding built-ins is great for addon developers to add things on top of Blender and keep it seamless for the users.
And it also might be critical for some addons - because otherwise you'll need to rebuild built-ins you're trying to override by different names and change keymaps to new operators to keep it usable. Which will inevitably go in conflict with some other code that's using real built-ins and there may not be a way to fix it.

The solutions seems to be:

  • keep the option to override builtins;

  • add some notification that built-in operator is overriden (I think it should be only visible in console in Blender --debug mode since it's not critical, it's just an information that could be useful for debug later on);

  • add some non-debug warning in console that built-in operator was overridden twice or some non-built in operator was overriden even once.
    It's important to keep this warning visible even in non debug mode of Blender because it basically means that there is addon conflict and without warning it might go under the radar until you actually meet the functionality issue and spend awhile debugging it.
    If there will be no warning of this kind in Blender itself then it would be reasonable for any addon that's using operators override to add the warning themselves to make sure it doesn't break. So it's reasonable to just add it directly to Blender.

Is there any reasons to prevent overriding built-ins? If user installed Blender addon that's changing Blender core functionality - isn't it part of the deal? On other hand allowing overriding built-ins is great for addon developers to add things on top of Blender and keep it seamless for the users. And it also might be critical for some addons - because otherwise you'll need to rebuild built-ins you're trying to override by different names and change keymaps to new operators to keep it usable. Which will inevitably go in conflict with some other code that's using real built-ins and there may not be a way to fix it. The solutions seems to be: - keep the option to override builtins; - add some notification that built-in operator is overriden (I think it should be only visible in console in Blender `--debug` mode since it's not critical, it's just an information that could be useful for debug later on); - add some non-debug warning in console that built-in operator was overridden twice or some non-built in operator was overriden even once. It's important to keep this warning visible even in non debug mode of Blender because it basically means that there is addon conflict and without warning it might go under the radar until you actually meet the functionality issue and spend awhile debugging it. If there will be no warning of this kind in Blender itself then it would be reasonable for any addon that's using operators override to add the warning themselves to make sure it doesn't break. So it's reasonable to just add it directly to Blender.
Blender Bot added
Status
Resolved
and removed
Status
Needs Info from Developers
labels 2023-02-28 04:13:11 +01:00

Note that overriding built-ins should have never been possible, doing this causes duplicate operators with the same name in the hash causing the result of the lookup to be undefined as this report notes.
Committed a fix that causes registering duplicate types to raise an error.

Note that overriding built-ins should have never been possible, doing this causes duplicate operators with the same name in the hash causing the result of the lookup to be undefined as this report notes. Committed a fix that causes registering duplicate types to raise an error.
Author

Oh wow, this significantly breaks our add-on :( Is there a reason @erik 's solution of removing the old operator and overriding it not an acceptable solution?

We use it to hook into events like object deletion or object duplication. E.g. if you delete a Blender object, a correlating dataset is also deleted elsewhere. Is a workflow like this now no longer possible?

Oh wow, this significantly breaks our add-on :( Is there a reason @erik 's solution of removing the old operator and overriding it not an acceptable solution? We use it to hook into events like object deletion or object duplication. E.g. if you delete a Blender object, a correlating dataset is also deleted elsewhere. Is a workflow like this now no longer possible?
Author

To clarify, we'd need to reliably detect the deletion, or copy / paste, or duplication of objects. (FYI add-on in question is https://blenderbim.org/ which turns Blender into an authoring platform for digital building models for the architecture, engineering, and construction industry, which has their own ISO standardised object data structure).

One option is to define our own keymaps and override the "x" and "delete" and "alt-d" and "shift-d", but then there's the issue of people using menus in various locations in the viewport and outliner and people who have their own keymaps who then have to do extra work. It also breaks people calling delete from scripts.

Alternatively, could Blender provide a pre/post delete or pre/post duplicate callback? Or something else we could use the msgbus subscribe to to listen to?

What other options are there?

I don't think we're alone in doing this override, this StackExchange post https://blender.stackexchange.com/questions/184245/how-to-re-register-a-built-in-operator shows others are also doing similar workflows.

To clarify, we'd need to reliably detect the deletion, or copy / paste, or duplication of objects. (FYI add-on in question is https://blenderbim.org/ which turns Blender into an authoring platform for digital building models for the architecture, engineering, and construction industry, which has their own ISO standardised object data structure). One option is to define our own keymaps and override the "x" and "delete" and "alt-d" and "shift-d", but then there's the issue of people using menus in various locations in the viewport and outliner and people who have their own keymaps who then have to do extra work. It also breaks people calling delete from scripts. Alternatively, could Blender provide a pre/post delete or pre/post duplicate callback? Or something else we could use the msgbus subscribe to to listen to? What other options are there? I don't think we're alone in doing this override, this StackExchange post https://blender.stackexchange.com/questions/184245/how-to-re-register-a-built-in-operator shows others are also doing similar workflows.
Author

Apologies if this is not the right etiquette, but I've reopened the bug due to the side effects (even if not initially designed that way).

Apologies if this is not the right etiquette, but I've reopened the bug due to the side effects (even if not initially designed that way).
Blender Bot added
Status
Needs Triage
and removed
Status
Resolved
labels 2023-02-28 23:00:20 +01:00

Firstly, overriding the object delete operator isn't a reliable way to track removal - objects could be removed from the outliner, Python scripts can unlink object or their creation can be undone.

So this report is requesting support for something that's not closely related to the bug report (which related to duplicate entries in a hash which should not have happened).

While the request is understandable, it's not a bug - so it would be better to handle this as a separate addition to the API. Investigate if it's even reasonable to support this.

Firstly, overriding the object delete operator isn't a reliable way to track removal - objects could be removed from the outliner, Python scripts can unlink object or their creation can be undone. So this report is requesting support for something that's not closely related to the bug report (which related to duplicate entries in a hash which should not have happened). While the request is understandable, it's not a bug - so it would be better to handle this as a separate addition to the API. Investigate if it's even reasonable to support this.
Blender Bot added
Status
Archived
and removed
Status
Needs Triage
labels 2023-03-01 02:00:58 +01:00
Author

Thanks for the explanation!

Thanks for the explanation!
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
6 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#105063
No description provided.