Job can get stuck with less workers than soft-fail threshold #104190

New Issue

Sybren A. Stüvel · 2023-02-28T11:56:08+01:00

Sybren A. Stüvel commented

2023-02-28 11:56:08 +01:00

Flamenco Version
Is Broken: 3.2
Worked OK: never

Short description of error
If a Worker fails a task, the task is marked as soft-failed. It only moves to failed status when more than 3 workers fail the same task. This means that if there are less than 3 workers in the farm, the job gets stuck.

Exact steps for others to reproduce the error

Set up Flamenco with one Worker
Submit a blend file that will fail (for example blender --python-expr 'raise SystemExit("fake failure")')
Start the job

Expected behaviour: after the failure, Flamenco detects that there are no workers left to retry the task, so the task immediately fails. Because the job cannot complete, the job fails as well.

Actual behaviour: after the failure the task sits indefinitely at soft-failed, and the job remains active.

Marked as high priority as on smaller render farms this can make any task failure a blocker of the entire job.

**Flamenco Version** Is Broken: 3.2  Worked OK: never  **Short description of error** If a Worker fails a task, the task is marked as `soft-failed`. It only moves to `failed` status when more than 3 workers fail the same task. This means that if there are less than 3 workers in the farm, the job gets stuck. **Exact steps for others to reproduce the error**  - Set up Flamenco with one Worker - Submit a blend file that will fail (for example `blender --python-expr 'raise SystemExit("fake failure")'`) - Start the job **Expected behaviour:** after the failure, Flamenco detects that there are no workers left to retry the task, so the task immediately fails. Because the job cannot complete, the job fails as well. **Actual behaviour:** after the failure the task sits indefinitely at `soft-failed`, and the job remains `active`. Marked as high priority as on smaller render farms this can make *any* task failure a blocker of the entire job.

Sybren A. Stüvel added the

labels 2023-02-28 11:56:21 +01:00

Sybren A. Stüvel added the

Good First Issue

label 2023-02-28 12:44:30 +01:00

Nitin-Rawat-1 referenced this issue

2023-03-09 19:19:25 +01:00

Check for number of workers before soft failing the task. #104195

Nitin-Rawat-1 commented

2023-03-09 19:24:58 +01:00

If number of workers capable of running the failed task again is "1" that means we have no worker besides the one that actually failed the task. In such condition we should just fail the job itself.

Sybren A. Stüvel referenced this issue from a commit

2023-04-20 11:53:43 +02:00

Check for number of workers before soft failing the task. (#104195)

Sybren A. Stüvel commented

2023-04-24 13:50:43 +02:00

This was actually fixed by @Nitin-Rawat-1 , thanks again for the patch!

Sybren A. Stüvel closed this issue

2023-04-24 13:50:44 +02:00

Sybren A. Stüvel added this to the v3.3 milestone 2023-04-24 13:50:52 +02:00

Sign in to join this conversation.