Check for number of workers before soft failing the task. #104195

Merged
Sybren A. Stüvel merged 15 commits from Nitin-Rawat-1/flamenco:104190-job-stuck into main 2023-04-20 11:53:43 +02:00
6 changed files with 187 additions and 8 deletions
Showing only changes of commit c6974e97d2 - Show all commits

View File

@ -40,21 +40,30 @@ file][workercfg].
## Can I change the paths/names of the rendered files? ## Can I change the paths/names of the rendered files?
Where Flamenco places the rendered files is determined by the job type. You can Where Flamenco places the rendered files is determined by the job type. You can
create [your own custom job type][jobtypes] to change this. With that, you can create [your own custom job type][jobtypes] or check the existing
[third-party job types][thirdpartyjobs] to change this. With that, you can
even add your own custom job settings like a sequence identifier and use that to even add your own custom job settings like a sequence identifier and use that to
determine the location of rendered files. determine the location of rendered files.
## Can I use the Compositor to output multiple EXR files? ## Can I use the Compositor to output multiple EXR files or Passes?
This is possible with Flamenco, but it takes a bit of work. It's not managed by This is possible with Flamenco, but it takes a bit of work. Although it's not
Flamenco's default job types. You can create [your own custom job managed by Flamenco's default job types, you can use a [custom job type][jobtypes]
type][jobtypes] for this, though. With that, you have control over the arguments for this.
that get used before and/or after the filename on the CLI.
If you have this working, please [share your job compiler script with us][getinvolved]! With that, you have control over the arguments that get used before and/or after
the filename on the CLI.
There are Flamenco jobs out there that support compositor nodes,
multi-platform, and multiple pass outputs. You can check our [third-party jobs
section][thirdpartyjobs].
If you wish to contribute to the project, you're invited to
[get involved with Flamenco][getinvolved]!
[jobtypes]: {{< ref "usage/job-types" >}} [jobtypes]: {{< ref "usage/job-types" >}}
[thirdpartyjobs]: {{< ref "third-party-jobs" >}}
[getinvolved]: {{< ref "development/get-involved" >}} [getinvolved]: {{< ref "development/get-involved" >}}
@ -98,3 +107,21 @@ complex project, and relies on a lot of components
([source](https://www.opencue.io/docs/getting-started/)), whereas Flamenco is ([source](https://www.opencue.io/docs/getting-started/)), whereas Flamenco is
made for simplicity and use in small studios or at home, running on your own made for simplicity and use in small studios or at home, running on your own
hardware. hardware.
## Why do I get an Error Performing BAT Pack Message?
As of yet, we've only encountered the issue below on Windows installations.
```
Error performing BAT pack: [WinError 267] The directory name is invalid:
'C:\\The\\Path\\To\\Your\\Project.blend'
```
This is most likely some sort of incompatibility that occurs in some cases where
you might be using linked assets from an asset library in your project.
*To fix this you may attempt the following fix:*
* Go to **File -> External Data -> Make Paths Relative**
* Submit your job again.
That should fix the issue.

View File

@ -0,0 +1,4 @@
---
title: Third-Party Jobs
weight: 30
---

View File

@ -0,0 +1,22 @@
---
title: Available Third-Party Jobs
weight: 1
---
This section contains third-party jobs for Flamenco submitted by the community.
If you wish to contribute you might consider joining our
[Blender Chat channel][flamencochannel] and chime-in there.
## How can I create my own Job?
Check how to create a custom job in our [Job Types][jobtypes] section.
It's recommended to use the [built-in scripts][built-in-scripts] as examples and build from there.
## Third-Party Job List
{{< flamenco/toc-children >}}
[jobtypes]: {{< ref "usage/job-types" >}}
[built-in-scripts]: https://projects.blender.org/studio/flamenco/src/branch/main/internal/manager/job_compilers/scripts
[flamencochannel]: https://blender.chat/channel/flamenco

View File

@ -0,0 +1,102 @@
---
title: Compositor Nodes and Multi-Platform Storage Paths
weight: 10
---
Job maintained by: Dylan Blanqué
If you need to use Blender's **Compositor** Nodes with *Flamenco*,
a Python Script and a Flamenco Job have been contributed to the community.
You'll need to do the following changes to support this workflow:
(It's recommended to use a symbolic link to the git repo files)
1. Clone the [Flamenco Compositor Script repository][compositorrepo]
(you'll need to install **git**) or download the files manually to a directory
in your Flamenco Manager/Server.
```bash
git clone https://github.com/dblanque/flamenco-compositor-script.git
```
2. Copy or make a symbolic link of the **startup_script.py** file.
to the configured Blender File Folder in your *Network Attached Storage*.
3. Copy or make a symbolic link of the multipass javascript job to the *scripts*
folder in your Flamenco Manager Installation (Create it if it doesn't exist).
4. Add and configure the required variables from the *example Manager YAML*
*Config* to your Flamenco Manager YAML.
* **storagePath** - Your NAS path, multi-platform variable.
* **jobSubPath** - Where the jobs are stored inside storagePath.
* **renderSubpath** - Where the Render Output is stored inside storagePath.
* **deviceType** - Compute Device Type to force *do not set the variable if*
*you wish to use whatever is available*
5. Submit your job from a Blender Client with the corresponding Multi-Pass Job,
it should whatever compositor nodes you have set and correct the paths where
necessary.
[compositorrepo]: https://github.com/dblanque/flamenco-compositor-script.git
**This has only been tested in an environment with Flamenco Manager and**
**Shaman enabled, but it should work without Shaman as well.**
# Example Configuration Flamenco Manager YAML
```yaml
# Configuration file for Flamenco.
#
# For an explanation of the fields,
# refer to the original flamenco-manager-example.yaml
_meta:
version: 3
manager_name: Flamenco Manager
database: flamenco-manager.sqlite
listen: :8080
autodiscoverable: true
local_manager_storage_path: ./flamenco-manager-storage
shared_storage_path: /mnt/storage/project_files
shaman:
enabled: true
garbageCollect:
period: 24h0m0s
maxAge: 744h0m0s
extraCheckoutPaths: []
task_timeout: 10m0s
worker_timeout: 1m0s
blocklist_threshold: 3
task_fail_after_softfail_count: 3
variables:
blender:
values:
- platform: all
value: blender
- platform: linux
value: /usr/local/blender/blender
- platform: windows
value: C:/Program Files/Blender Foundation/Blender 3.4/blender.exe
- platform: darwin
value: /usr/bin/blender
blenderArgs:
values:
- platform: all
value: -b -y
storagePath:
values:
- platform: linux
value: /mnt/storage
- platform: windows
value: "Z:\\"
jobSubPath:
values:
- platform: all
value: project_files
renderSubPath:
values:
- platform: all
value: project_render
deviceType:
values:
- platform: all
value: "CUDA"
# Set the device type to FIRST or remove the variable definition
# to use whatever device type is detected first.
```

View File

@ -25,9 +25,13 @@ The difference with regular variables is that regular variables are one-way:
Two-way variables go both ways, as follows: Two-way variables go both ways, as follows:
- When submitting a job, values are replaced with variables. - When submitting a **job**, values **in the javascript jobs' command** are replaced
with the corresponding variables as it's executed on the client.
- When sending a task to a worker, variables are replaced with values again. - When sending a task to a worker, variables are replaced with values again.
*(Do keep in mind that if you perform changes to a job, you'll need to re-submit*
*it.)*
This may seem like a lot of unnecessary work. After all, why go through the This may seem like a lot of unnecessary work. After all, why go through the
trouble of replacing in one direction, when later the opposite is done? The trouble of replacing in one direction, when later the opposite is done? The
power lies in the fact that each replacement step can target a different power lies in the fact that each replacement step can target a different

View File

@ -0,0 +1,20 @@
<!-- For more info check the links below -->
<!-- Lists: https://gohugo.io/templates/lists/ -->
<!-- Taxonomy Templates: https://gohugo.io/templates/taxonomy-templates/ -->
<!-- Page Variables: https://gohugo.io/variables/pages/ -->
<!-- Cheers, Dylan -->
<!-- This TOC Excludes the current section index and the current page -->
<div>
{{ $current_page_title := .Page.Title }}
<ul>
{{ range .Page.CurrentSection.Data.Pages.ByWeight }}
{{ if ne .Page.Title $current_page_title }}
<li>
<a href="{{ .Permalink }}">{{ .LinkTitle }}</a>
</li>
{{ end }}
{{ end }}
</ul>
</div>