Added ToNative to the blendfile args to resolve workers failing due to windows -> linux pathing #104295

Open
Jamie-Newbon wants to merge 1 commits from Jamie-Newbon/flamenco:main into main

When changing the target branch, be careful to rebase the branch in your fork to match. See documentation.
First-time contributor

This is a fix for #104237 and #104246 where the blend file would use the wrong slash's when using a windows manager and Linux workers

I have not tested this with linux manager and windows workers

This is a fix for #104237 and #104246 where the blend file would use the wrong slash's when using a windows manager and Linux workers I have not tested this with linux manager and windows workers
Jamie-Newbon added 1 commit 2024-04-02 21:29:56 +02:00

Thanks for the PR, and your willingness to help the project :)

The referenced issues were already closed, with one actually being fixed and the other where the original reporter was unresponsive, so I don't know exactly how to deal with this PR. If this issue is still there (in the main branch, that is), do let me know.

Thanks for the PR, and your willingness to help the project :) The referenced issues were already closed, with one actually being fixed and the other where the original reporter was unresponsive, so I don't know exactly how to deal with this PR. If this issue is still there (in the `main` branch, that is), do let me know.
Sybren A. Stüvel added the
Status
Needs Information from User
label 2024-04-04 10:33:50 +02:00

PS: Please test with 3.5-beta0, I've just released that to the 'experimental' download section.

PS: Please test with 3.5-beta0, I've just released that to [the 'experimental' download section](https://flamenco.blender.org/download/#go-experimental).
First-time contributor

In 3.5-beta1
When submitting to manager on Windows and when the file and renderoutput is inside the shared folder, I get this error on my Mac worker:
2024-04-17T08:36:18+02:00 INF going to run: blender -b -y '/Volumes/192.168.0.210/render\Project1\Projectblend.flamenco.blend' --render-output '/Volumes/192.168.0.210/render/Project1/Render/Project1/2024-04-17_084036/######' --render-format PNG --render-frame 3,4 command=blender-render task=84c530af-9bc9-436c-a84e-d1bde891ca02
(...)
2024-04-17T08:36:19+02:00 WRN error executing task error="command exited abnormally with code 1" task={"commands":[{"name":"blender-render","parameters":{"args":["--render-output","/Volumes/192.168.0.210/render/Project1/Render/Project1/2024-04-17_084036/######","--render-format","PNG","--render-frame","3,4"],"argsBefore":[],"blendfile":"/Volumes/192.168.0.210/render\\Project1\\Projectblend.flamenco.blend","exe":"blender","exeArgs":"-b -y"}}],"job":"28c4a99a-8157-40c0-9047-9c06fad80b2f","job_priority":50,"job_type":"simple-blender-render","name":"render-3,4","priority":50,"status":"active","task_type":"blender","uuid":"84c530af-9bc9-436c-a84e-d1bde891ca02"}
When the blender-file is outside the shared folder, everything works. It seems the path to the blender-file gets the wrong slashes still, the output path looks ok. It seems the PR would address this, but I haven't built it myself.

This issue seems related: #100540

In 3.5-beta1 When submitting to manager on Windows and when the file and renderoutput is inside the shared folder, I get this error on my Mac worker: `2024-04-17T08:36:18+02:00 INF going to run: blender -b -y '/Volumes/192.168.0.210/render\Project1\Projectblend.flamenco.blend' --render-output '/Volumes/192.168.0.210/render/Project1/Render/Project1/2024-04-17_084036/######' --render-format PNG --render-frame 3,4 command=blender-render task=84c530af-9bc9-436c-a84e-d1bde891ca02` (...) `2024-04-17T08:36:19+02:00 WRN error executing task error="command exited abnormally with code 1" task={"commands":[{"name":"blender-render","parameters":{"args":["--render-output","/Volumes/192.168.0.210/render/Project1/Render/Project1/2024-04-17_084036/######","--render-format","PNG","--render-frame","3,4"],"argsBefore":[],"blendfile":"/Volumes/192.168.0.210/render\\Project1\\Projectblend.flamenco.blend","exe":"blender","exeArgs":"-b -y"}}],"job":"28c4a99a-8157-40c0-9047-9c06fad80b2f","job_priority":50,"job_type":"simple-blender-render","name":"render-3,4","priority":50,"status":"active","task_type":"blender","uuid":"84c530af-9bc9-436c-a84e-d1bde891ca02"} ` When the blender-file is outside the shared folder, everything works. It seems the path to the blender-file gets the wrong slashes still, the output path looks ok. It seems the PR would address this, but I haven't built it myself. This issue seems related: #100540
Author
First-time contributor

Sorry for the delay, When I submitted the PR I had got the issue fixed, then to confirm it actually did fix the issue I reverted my changes to see the the job would render which it didn't thus confirming to me that the issue was still there in main branch at time of submission.

We have now swapped the Manager over to run on Linux and have noticed that instead of the blend file having the wrong slashes, its the render output instead and only on the Windows machines.

Windows Box

going to run:

    'C:\Program Files\Blender Foundation\Blender 4.1\blender.exe' -b -y 'V:\blenderrender\Projects\Donut\donut2.flamenco.blend' --render-output 'V:\blenderrender\Projects/Donut/donut2/2024-04-17_042428/######' --render-format PNG --render-frame 30

Linux Box (Running my changes)

going to run:

    blender -b -y /mnt/my_windows_stuffs/blenderrender/Projects/Donut/donut2.flamenco.blend --render-output '/mnt/my_windows_stuffs/blenderrender/Projects/Donut/donut2/2024-04-17_042428/######' --render-format PNG --render-frame 18

However as windows seems the handles both type of slashes, the render still works.

Sorry for the delay, When I submitted the PR I had got the issue fixed, then to confirm it actually did fix the issue I reverted my changes to see the the job would render which it didn't thus confirming to me that the issue was still there in main branch at time of submission. We have now swapped the Manager over to run on Linux and have noticed that instead of the blend file having the wrong slashes, its the render output instead and only on the Windows machines. Windows Box ``` going to run: 'C:\Program Files\Blender Foundation\Blender 4.1\blender.exe' -b -y 'V:\blenderrender\Projects\Donut\donut2.flamenco.blend' --render-output 'V:\blenderrender\Projects/Donut/donut2/2024-04-17_042428/######' --render-format PNG --render-frame 30 ``` Linux Box (Running my changes) ``` going to run: blender -b -y /mnt/my_windows_stuffs/blenderrender/Projects/Donut/donut2.flamenco.blend --render-output '/mnt/my_windows_stuffs/blenderrender/Projects/Donut/donut2/2024-04-17_042428/######' --render-format PNG --render-frame 18 ``` However as windows seems the handles both type of slashes, the render still works.

Thanks for coming back with more info :)

2024-04-17T08:36:19+02:00 WRN error executing task error="command exited abnormally with code 1" task={"commands":[{"name":"blender-render","parameters":{"args":["--render-output","/Volumes/192.168.0.210/render/Project1/Render/Project1/2024-04-17_084036/######","--render-format","PNG","--render-frame","3,4"],"argsBefore":[],"blendfile":"/Volumes/192.168.0.210/render\\Project1\\Projectblend.flamenco.blend","exe":"blender","exeArgs":"-b -y"}}],"job":"28c4a99a-8157-40c0-9047-9c06fad80b2f","job_priority":50,"job_type":"simple-blender-render","name":"render-3,4","priority":50,"status":"active","task_type":"blender","uuid":"84c530af-9bc9-436c-a84e-d1bde891ca02"}

This is an interesting one. Where does the double \\ come from, in "/Volumes/192.168.0.210/render\\Project1\\Projectblend.flamenco.blend"? never mind, that's just a regular single backslash, but escaped with another backslash by the logger.

On Windows, always use drive letters to point to the shared storage. Flamenco does not support UNC notation.

Please attach your flamenco-manager.yaml so that I have some more info about how it's set up. That'll help me to dig in further. It's good to know that this PR fixes this issue, but I'd like it even more if it could be fixed in a more general way (without relying on each command knowing exactly what is a filename and what isn't).

Thanks for coming back with more info :) > `2024-04-17T08:36:19+02:00 WRN error executing task error="command exited abnormally with code 1" task={"commands":[{"name":"blender-render","parameters":{"args":["--render-output","/Volumes/192.168.0.210/render/Project1/Render/Project1/2024-04-17_084036/######","--render-format","PNG","--render-frame","3,4"],"argsBefore":[],"blendfile":"/Volumes/192.168.0.210/render\\Project1\\Projectblend.flamenco.blend","exe":"blender","exeArgs":"-b -y"}}],"job":"28c4a99a-8157-40c0-9047-9c06fad80b2f","job_priority":50,"job_type":"simple-blender-render","name":"render-3,4","priority":50,"status":"active","task_type":"blender","uuid":"84c530af-9bc9-436c-a84e-d1bde891ca02"}` ~~This is an interesting one. Where does the double `\\` come from, in `"/Volumes/192.168.0.210/render\\Project1\\Projectblend.flamenco.blend"`?~~ never mind, that's just a regular single backslash, but escaped with another backslash by the logger. ~~On Windows, always use drive letters to point to the shared storage. Flamenco does not support [UNC notation](https://en.wikipedia.org/wiki/Path_(computing)#Universal_Naming_Convention).~~ Please attach your `flamenco-manager.yaml` so that I have some more info about how it's set up. That'll help me to dig in further. It's good to know that this PR fixes this issue, but I'd like it even more if it could be fixed in a more general way (without relying on each command knowing exactly what is a filename and what isn't).
First-time contributor

This is the storage setup:

shared_storage_path: F:\render
...
my_storage:
    is_twoway: true
    values:
    - platform: linux
      value: /media/192.168.0.210/render
    - platform: windows
      value: F:\render
    - platform: darwin
      value: /Volumes/192.168.0.210/render

The manager is on the Windows machine, which uses drive letters, I only have a mac worker, no linux.

This is the storage setup: ```yaml shared_storage_path: F:\render ... my_storage: is_twoway: true values: - platform: linux value: /media/192.168.0.210/render - platform: windows value: F:\render - platform: darwin value: /Volumes/192.168.0.210/render ``` The manager is on the Windows machine, which uses drive letters, I only have a mac worker, no linux.

I've tried a slightly different approach to fix the same issue, in 7ffbd72dce. This should avoid the need to call crosspath.ToNative() on every use of a path, as the two-way variables system should already take care of this.

Could you give the latest Flamenco main branch a test, to see if that also solves your problems?

I've tried a slightly different approach to fix the same issue, in 7ffbd72dce1284ce81ff11949aa7ef50ad7f1d32. This should avoid the need to call `crosspath.ToNative()` on every use of a path, as the two-way variables system should already take care of this. Could you give the latest Flamenco `main` branch a test, to see if that also solves your problems?

Nope, my approach wasn't the right one. I think your approach is better, as then the command impementation itself can decide which parameters to adjust and which to keep as-is.

Nope, my approach wasn't the right one. I think your approach is better, as then the command impementation itself can decide which parameters to adjust and which to keep as-is.

As it turns out, my approach was fine apart from the little bug I introduced (#104331). Now that that's fixed, please re-test with the latest revision on the main branch to see if your issue is properly addressed now.

As it turns out, my approach was fine apart from the little bug I introduced (#104331). Now that that's fixed, please re-test with the latest revision on the `main` branch to see if your issue is properly addressed now.
Merge conflict checking is in progress. Try again in few moments.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u main:Jamie-Newbon-main
git checkout Jamie-Newbon-main
Sign in to join this conversation.
No description provided.