Flamenco: Failed: command exited abnormally with code 11 #103673

New Issue

ManBlender · 2023-01-05T22:49:23+01:00

ManBlender commented

2023-01-05 22:49:23 +01:00

I'm not sure where to report this case around Flamenco.

2023-01-05T18:31:47+01:00 Task assigned to worker DESKTOP-X (bla bla)
2023-01-05T18:31:47+01:00 task changed status queued -> active
going to run: E:\blender\blender.exe ["-b" "-y" "E:/Flamenco/2023-01-05-183145.219205-test1/test.flamenco.blend" "--render-output" "E:/Flamenco/Test/Pictures/test1/2023-01-05_183145/######" "--render-format" "PNG" "--render-frame" "1"]
pid=19336 > CAD_Sketcher-main:{INFO}: Logging into: C:\Users\ManBlender\AppData\Local\Temp\CAD_Sketcher-main.log
pid=19336 > INFO:CAD_Sketcher-main:Logging into: C:\Users\ManBlender\AppData\Local\Temp\CAD_Sketcher-main.log
pid=19336 > CAD_Sketcher-main:{WARNING}: Solvespace module isn't available, only base modules registered
pid=19336 > WARNING:CAD_Sketcher-main:Solvespace module isn't available, only base modules registered
pid=19336 > Error   : EXCEPTION_ACCESS_VIOLATION
pid=19336 > Address : 0x0000000000000000
pid=19336 > Module  : 
pid=19336 > Thread  : 0000263c
pid=19336 > Blender 3.4.0 (hash a95bf1ac01be built 2022-12-07 02:06:25)
pid=19336 > Read prefs: C:\Users\ManBlender\AppData\Roaming\Blender Foundation\Blender\3.4\config\userpref.blend
pid=19336 > Writing: C:\Users\ManBlender\AppData\Local\Temp\blender.crash.txt
Failed: command exited abnormally with code 11
2023-01-05T18:31:49+01:00 Task failed by 1 worker, Manager will mark it as soft failure. 2 more failures will cause hard failure.
2023-01-05T18:31:49+01:00 task changed status active -> soft-failed

blender.crash.txt
Similar for Blender 3.2 - flamenco-3.1-windows-amd64.zip

The blender file was just a cube standard in test.blend file

I'm not sure where to report this case around Flamenco. ``` 2023-01-05T18:31:47+01:00 Task assigned to worker DESKTOP-X (bla bla) 2023-01-05T18:31:47+01:00 task changed status queued -> active going to run: E:\blender\blender.exe ["-b" "-y" "E:/Flamenco/2023-01-05-183145.219205-test1/test.flamenco.blend" "--render-output" "E:/Flamenco/Test/Pictures/test1/2023-01-05_183145/######" "--render-format" "PNG" "--render-frame" "1"] pid=19336 > CAD_Sketcher-main:{INFO}: Logging into: C:\Users\ManBlender\AppData\Local\Temp\CAD_Sketcher-main.log pid=19336 > INFO:CAD_Sketcher-main:Logging into: C:\Users\ManBlender\AppData\Local\Temp\CAD_Sketcher-main.log pid=19336 > CAD_Sketcher-main:{WARNING}: Solvespace module isn't available, only base modules registered pid=19336 > WARNING:CAD_Sketcher-main:Solvespace module isn't available, only base modules registered pid=19336 > Error : EXCEPTION_ACCESS_VIOLATION pid=19336 > Address : 0x0000000000000000 pid=19336 > Module : pid=19336 > Thread : 0000263c pid=19336 > Blender 3.4.0 (hash a95bf1ac01be built 2022-12-07 02:06:25) pid=19336 > Read prefs: C:\Users\ManBlender\AppData\Roaming\Blender Foundation\Blender\3.4\config\userpref.blend pid=19336 > Writing: C:\Users\ManBlender\AppData\Local\Temp\blender.crash.txt Failed: command exited abnormally with code 11 2023-01-05T18:31:49+01:00 Task failed by 1 worker, Manager will mark it as soft failure. 2 more failures will cause hard failure. 2023-01-05T18:31:49+01:00 task changed status active -> soft-failed ``` [blender.crash.txt](https://archive.blender.org/developer/F14116800/blender.crash.txt) Similar for Blender 3.2 - flamenco-3.1-windows-amd64.zip The blender file was just a cube standard in test.blend file

ManBlender commented

2023-01-05 22:49:23 +01:00

Added subscriber: @ManBlender

ManBlender commented

2023-01-05 22:51:38 +01:00

it be nice to download old version https://flamenco.blender.org/downloads/

Sybren A. Stüvel added

and removed

labels 2023-02-17 11:09:21 +01:00

Sybren A. Stüvel commented

2023-02-28 11:46:39 +01:00

Please update the task description with steps to reproduce this issue. If it cannot be reproduced with standard Blender but requires a specific Blender add-on, then it's considerably harder to figure out what's going wrong.

it be nice to download old version https://flamenco.blender.org/downloads/

That's a good idea. Until that's there, you can edit the URL of the packages to download other versions -- just change 3.2 to 3.1 or 3.0.

Please update the task description with steps to reproduce this issue. If it cannot be reproduced with standard Blender but requires a specific Blender add-on, then it's considerably harder to figure out what's going wrong. > it be nice to download old version https://flamenco.blender.org/downloads/ That's a good idea. Until that's there, you can edit the URL of the packages to download other versions -- just change `3.2` to `3.1` or `3.0`.

Sybren A. Stüvel added

Status

Needs Information from User

and removed

Status

Needs Triage

labels 2023-02-28 11:48:25 +01:00

BlackElkX commented

2023-07-12 22:06:49 +02:00

I have the same issue when I set up a multi platform farm:

1 imac i9 9900k 64GB ram (gives the order) has a worker and works great
1 debian 11 celeron (generation like i9 9900k) has a docker container with the manager and a worker (works but very slow)
1 windows 11 with i5 8250U 12GB ram has a worker and works slow
1 windows 10 with i5 5th gen 8GB ram has a worker but does not work (or very very slow)
1 windows 10 with amd rysen 5 5600U 8GB ram has a worker and gives this error (code 11)
They all start correctly and all see the correct share and files.
Blender version is 3.6 (whish is great and stable on all platforms) and I configured Eevee with a resolution of 1920x1080.
The scene has 1 object from its own, a landscape. 2 objects (linked 1 object 2 times) both with different textures and animations.
1 flying and 1 static, with moving parts.
2 persons are linked without animation as extra details for the scene.
2 cameras, each following a different path: 1 following the flying object, 1 starting from within the fixed object, going out and end with a panoramic shot.
I can share the files if really needed, they are combined somewhat around 115MB
Renderings are correct.
I hope this can help for this bug / problem. If I find a sollution myself, I update this message.
I added 3 loggings, one containing this error, 2 others contained other errors.

I have the same issue when I set up a multi platform farm: - 1 imac i9 9900k 64GB ram (gives the order) has a worker and works great - 1 debian 11 celeron (generation like i9 9900k) has a docker container with the manager and a worker (works but very slow) - 1 windows 11 with i5 8250U 12GB ram has a worker and works slow - 1 windows 10 with i5 5th gen 8GB ram has a worker but does not work (or very very slow) - 1 windows 10 with amd rysen 5 5600U 8GB ram has a worker and gives this error (code 11) They all start correctly and all see the correct share and files. - Blender version is 3.6 (whish is great and stable on all platforms) and I configured Eevee with a resolution of 1920x1080. - The scene has 1 object from its own, a landscape. 2 objects (linked 1 object 2 times) both with different textures and animations. 1 flying and 1 static, with moving parts. - 2 persons are linked without animation as extra details for the scene. - 2 cameras, each following a different path: 1 following the flying object, 1 starting from within the fixed object, going out and end with a panoramic shot. I can share the files if really needed, they are combined somewhat around 115MB Renderings are correct. I hope this can help for this bug / problem. If I find a sollution myself, I update this message. I added 3 loggings, one containing this error, 2 others contained other errors.

task-87d226a1-1105-4425-a9cc-7862fd5d4323.txt

65 KiB

task-4ac25bc9-33e0-4cea-a759-475370a98224.txt

61 KiB

task-ccc77155-c21a-4723-af8e-ef417c9ff4fd.txt

25 KiB

MichaelC commented

2023-07-13 03:38:53 +02:00

@BlackElkX thanks for the details on the setup and crash logs.

There is a discussion about "exit 11" in Flamenco chat here: https://blender.chat/channel/flamenco?msg=QEmqpZMJbQCb8h9Rp

The short of it is that this is a Blender exit code and not related to Flamenco. Flamenco spawns Blender with the .blend project file, with some parameters. It simply waits until Blender is done assuming if non-error exit code it was successful. If Blender signals it was not successful, Flamenco will simple reschedule the task again a number of attempts.

The OP mention Blender 3.2 on one machine. You list Blender 3.6 and various different machines.

The crash looks similar and looks to be a NULL pointer exception.

pid=2832 > Win32 Error# (127): Kan opgegeven procedure niet vinden.
pid=2832 > Error   : EXCEPTION_ACCESS_VIOLATION
pid=2832 > Address : 0x0000000000000000
pid=2832 > Module  : 
pid=2832 > Thread  : 000024f4

Do you get the same exception or crash just running Blender manually on a machine... with these parameters?

/Applications/Blender.app/Contents/MacOS/Blender ["-b" "-y" "/Volumes/pe00063/projects/yoko-tsuno/blender/objects/Spacecrafts/simpleTestScene1.flamenco.blend" "--render-output" "/Volumes/pe00063/projects/yoko-tsuno/renderings/simpleTestScene1/2023-07-12_163631/######" "--render-format" "PNG" "--render-frame" "3"]

I can't think how Flamenco could influence this type of crash unless related to how Flamenco launches sub-processes (ie: Blender) when it has some work to do. It is more likely memory corruption or a bug in Blender.

If this crash doesn't happen by running Blender manually but only happens when a project is rendered using Flamenco that would be a very useful data point.

Let us know if this only happens when using Flamenco. If yes, then please provide more configuration on how Flamenco is provisioned in your farm. The specific project files would be very useful too to see if someone else on different hardware can reproduce the same crash.

@BlackElkX thanks for the details on the setup and crash logs. There is a discussion about "exit 11" in Flamenco chat here: https://blender.chat/channel/flamenco?msg=QEmqpZMJbQCb8h9Rp The short of it is that this is a Blender exit code and not related to Flamenco. Flamenco spawns Blender with the .blend project file, with some parameters. It simply waits until Blender is done assuming if non-error exit code it was successful. If Blender signals it was not successful, Flamenco will simple reschedule the task again a number of attempts. The OP mention Blender 3.2 on one machine. You list Blender 3.6 and various different machines. The crash looks similar and looks to be a NULL pointer exception. ``` pid=2832 > Win32 Error# (127): Kan opgegeven procedure niet vinden. pid=2832 > Error : EXCEPTION_ACCESS_VIOLATION pid=2832 > Address : 0x0000000000000000 pid=2832 > Module : pid=2832 > Thread : 000024f4 ``` Do you get the same exception or crash just running Blender manually on a machine... with these parameters? ``` /Applications/Blender.app/Contents/MacOS/Blender ["-b" "-y" "/Volumes/pe00063/projects/yoko-tsuno/blender/objects/Spacecrafts/simpleTestScene1.flamenco.blend" "--render-output" "/Volumes/pe00063/projects/yoko-tsuno/renderings/simpleTestScene1/2023-07-12_163631/######" "--render-format" "PNG" "--render-frame" "3"] ``` I can't think how Flamenco could influence this type of crash unless related to how Flamenco launches sub-processes (ie: Blender) when it has some work to do. It is more likely memory corruption or a bug in Blender. If this crash doesn't happen by running Blender manually but only happens when a project is rendered using Flamenco that would be a very useful data point. Let us know if this only happens when using Flamenco. If yes, then please provide more configuration on how Flamenco is provisioned in your farm. The specific project files would be very useful too to see if someone else on different hardware can reproduce the same crash.

MichaelC commented

2023-07-13 03:44:26 +02:00

@BlackElkX - a quick search on Blender and there are four open issues with specifically in the title but none look like this exactly type of null pointer exception (all zeros memory address).

You can see them using this query:
https://projects.blender.org/blender/blender/issues?type=all&state=open&labels=&milestone=0&project=0&assignee=0&poster=0&q=EXCEPTION_ACCESS_VIOLATION

@BlackElkX - a quick search on Blender and there are four open issues with specifically in the title but none look like this exactly type of null pointer exception (all zeros memory address). You can see them using this query: https://projects.blender.org/blender/blender/issues?type=all&state=open&labels=&milestone=0&project=0&assignee=0&poster=0&q=EXCEPTION_ACCESS_VIOLATION

MichaelC commented

2023-07-13 03:56:57 +02:00

I just noticed, two of your crash logs have multiple EXCEPTION_ACCESS_VIOLATIONs as well as the null pointer one.

Eg:

pid=14952 > Error   : EXCEPTION_ACCESS_VIOLATION
pid=14952 > Address : 0x00007FF7841DD0E0
pid=14952 > Error   : EXCEPTION_ACCESS_VIOLATION
pid=14952 > Address : 0x00007FF7841DD0E0
pid=14952 > Error   : EXCEPTION_ACCESS_VIOLATION
pid=14952 > Address : 0x00007FF7841DD0E0
pid=14952 > Error   : EXCEPTION_ACCESS_VIOLATION
pid=14952 > Address : 0x00007FF7841DD0E0
pid=14952 > Error   : EXCEPTION_ACCESS_VIOLATION
pid=14952 > Address : 0x00007FF7841DD0E0

These happen near the start of the Blender run. I dont know how but these bug reports include a crash attachment that includes a stack trace which would be very helpful. Example of this type of log can be found in blender/blender#109903 at https://projects.blender.org/attachments/51027c56-71bc-4748-9210-d11c159cf852. (Note this may not be the exact same root cause... just an example of a blender crash log vs a Flamenco crash log).

This Blender crash file (not Flamenco) would be very helpful. Flamenco is only logging the output of Blender logs and not the crash log itself. On Windows, you would want to find a file .crash.txt. (https://docs.blender.org/manual/en/latest/troubleshooting/crash.html#windows)

I just noticed, two of your crash logs have multiple EXCEPTION_ACCESS_VIOLATIONs as well as the null pointer one. Eg: ``` pid=14952 > Error : EXCEPTION_ACCESS_VIOLATION pid=14952 > Address : 0x00007FF7841DD0E0 pid=14952 > Error : EXCEPTION_ACCESS_VIOLATION pid=14952 > Address : 0x00007FF7841DD0E0 pid=14952 > Error : EXCEPTION_ACCESS_VIOLATION pid=14952 > Address : 0x00007FF7841DD0E0 pid=14952 > Error : EXCEPTION_ACCESS_VIOLATION pid=14952 > Address : 0x00007FF7841DD0E0 pid=14952 > Error : EXCEPTION_ACCESS_VIOLATION pid=14952 > Address : 0x00007FF7841DD0E0 ``` These happen near the start of the Blender run. I dont know how but these bug reports include a crash attachment that includes a stack trace which would be very helpful. Example of this type of log can be found in https://projects.blender.org/blender/blender/issues/109903 at https://projects.blender.org/attachments/51027c56-71bc-4748-9210-d11c159cf852. (Note this may not be the exact same root cause... just an example of a blender crash log vs a Flamenco crash log). This Blender crash file (not Flamenco) would be very helpful. Flamenco is only logging the output of Blender logs and not the crash log itself. On Windows, you would want to find a file <project-name>.crash.txt. (https://docs.blender.org/manual/en/latest/troubleshooting/crash.html#windows)

ManBlender commented

2023-07-13 10:27:02 +02:00

This issue can be more observed around nvidia, etc. and in relation to GPU.
In normal mode eg. Blender or other 3d app is focused to run with GPU processor as priority and default, etc.
https://pureinfotech.com/set-default-gpu-apps-windows-11/
https://www.youtube.com/watch?v=hlyFjjJGaow
But when command line running flamenco or blender instance this setup can be omitted and render process if advanced (big project) failed. Then.. if small project working, that is more predicted whats going on.. if new sub-process.
I not solved this problem.

This issue can be more observed around nvidia, etc. and in relation to GPU. In normal mode eg. Blender or other 3d app is focused to run with GPU processor as priority and default, etc. https://pureinfotech.com/set-default-gpu-apps-windows-11/ https://www.youtube.com/watch?v=hlyFjjJGaow But when command line running flamenco or blender instance this setup can be omitted and render process if advanced (big project) failed. Then.. if small project working, that is more predicted whats going on.. if new sub-process. I not solved this problem.

Sybren A. Stüvel commented

2023-07-13 12:45:34 +02:00

Do you get the same exception or crash just running Blender manually on a machine... with these parameters?

/Applications/Blender.app/Contents/MacOS/Blender ["-b" "-y" "/Volumes/pe00063/projects/yoko-tsuno/blender/objects/Spacecrafts/simpleTestScene1.flamenco.blend" "--render-output" "/Volumes/pe00063/projects/yoko-tsuno/renderings/simpleTestScene1/2023-07-12_163631/######" "--render-format" "PNG" "--render-frame" "3"]

This is the main question to answer now. Just run this command in a terminal and see what happens. Just make sure all text ends up on the same line.

For @ManBlender to run:

E:\blender\blender.exe -b -y "E:/Flamenco/2023-01-05-183145.219205-test1/test.flamenco.blend" --render-output "E:/Flamenco/Test/Pictures/test1/2023-01-05_183145/######" --render-format PNG --render-frame 1

For @BlackElkX to run:

/Applications/Blender.app/Contents/MacOS/Blender -b -y "/Volumes/pe00063/projects/yoko-tsuno/blender/objects/Spacecrafts/simpleTestScene1.flamenco.blend" --render-output "/Volumes/pe00063/projects/yoko-tsuno/renderings/simpleTestScene1/2023-07-12_163631/######" --render-format PNG --render-frame 3

> Do you get the same exception or crash just running Blender manually on a machine... with these parameters? > > ``` > /Applications/Blender.app/Contents/MacOS/Blender ["-b" "-y" "/Volumes/pe00063/projects/yoko-tsuno/blender/objects/Spacecrafts/simpleTestScene1.flamenco.blend" "--render-output" "/Volumes/pe00063/projects/yoko-tsuno/renderings/simpleTestScene1/2023-07-12_163631/######" "--render-format" "PNG" "--render-frame" "3"] > ``` This is the main question to answer now. Just run this command in a terminal and see what happens. Just make sure all text ends up on the same line. For @ManBlender to run: ``` E:\blender\blender.exe -b -y "E:/Flamenco/2023-01-05-183145.219205-test1/test.flamenco.blend" --render-output "E:/Flamenco/Test/Pictures/test1/2023-01-05_183145/######" --render-format PNG --render-frame 1 ``` For @BlackElkX to run: ``` /Applications/Blender.app/Contents/MacOS/Blender -b -y "/Volumes/pe00063/projects/yoko-tsuno/blender/objects/Spacecrafts/simpleTestScene1.flamenco.blend" --render-output "/Volumes/pe00063/projects/yoko-tsuno/renderings/simpleTestScene1/2023-07-12_163631/######" --render-format PNG --render-frame 3 ```

BlackElkX commented

2023-07-19 13:47:33 +02:00

I did some extra checks: Added a very recent asus rog laptop with 2060: works after I installed the flamenco plugin.
I did updates of windows for all windows machines: no change
I run the manager in a docker container on the linux i9 machine. Worker does not work either direct or in docker on same machine.

The AMD machine is still giving same errors: I check in Blender.
The Linux machine (both docker and direct): I cannot install the plugin, it has no X11 interface. I will try another machine to configure and copy the plugins and settings to this machine.

Status: 3 of 6 working: 1 iMac and 2 Win11

To be continued...

I did some extra checks: Added a very recent asus rog laptop with 2060: works after I installed the flamenco plugin. I did updates of windows for all windows machines: no change I run the manager in a docker container on the linux i9 machine. Worker does not work either direct or in docker on same machine. The AMD machine is still giving same errors: I check in Blender. The Linux machine (both docker and direct): I cannot install the plugin, it has no X11 interface. I will try another machine to configure and copy the plugins and settings to this machine. Status: 3 of 6 working: 1 iMac and 2 Win11 To be continued...

Sybren A. Stüvel commented

2023-07-20 11:24:18 +02:00

I installed the flamenco plugin.

What 'plugin' is that? Do you mean the Blender add-on?

I cannot install the plugin, it has no X11 interface.

This is why I'm puzzled about the word "plugin". If this is about the Flamenco add-on for Blender, this is only necessary for submitting jobs to Flamenco. It does not need to be installed on the Workers.

Worker does not work either direct or in docker on same machine.

If there's no info about what doesn't work, there's little I can do to help. As a general recommendation: avoid complicating your setup until you have things working. Containerising things in Docker adds complexity and I issues with such setups are off topic for this tracker. If you want to run things in Docker that's of course fine, but issues there cannot be handled as issues with Flamenco.

> I installed the flamenco plugin. What 'plugin' is that? Do you mean the Blender add-on? > I cannot install the plugin, it has no X11 interface. This is why I'm puzzled about the word "plugin". If this is about the Flamenco add-on for Blender, this is only necessary for submitting jobs to Flamenco. It does *not* need to be installed on the Workers. > Worker does not work either direct or in docker on same machine. If there's no info about what doesn't work, there's little I can do to help. As a general recommendation: avoid complicating your setup until you have things working. Containerising things in Docker adds complexity and I issues with such setups are off topic for this tracker. If you want to run things in Docker that's of course fine, but issues there cannot be handled as issues with Flamenco.

Sybren A. Stüvel commented

2024-01-19 11:34:42 +01:00

Closing due to inactivity of the reporter.

Sybren A. Stüvel closed this issue

2024-01-19 11:34:43 +01:00

Sign in to join this conversation.