studio/flamenco
studio/flamenco
12
30
Fork 38
Code Issues 63 Pull Requests 5 Releases Wiki Activity

Manager should detect starvation due to blacklisting #59491

New Issue
Closed
opened 2018-12-17 13:57:17 +01:00 by Sybren A. Stüvel · 3 comments
Sybren A. Stüvel commented 2018-12-17 13:57:17 +01:00
Owner
Copy Link

The solution for #50981 does NOT include starvation detection. In other words, if a job has certain tasks that were failed by all available workers (and thus all workers are blacklisted for this job & task type) there is no detection that this happened. As a result, the job will be stuck in 'active' status without it ever having a chance of being finished.

We should have the Manager regularly inspect queued tasks, to see if there is still at least one worker that is not blacklisted and able to execute them. If not, we can mark those tasks as 'failed' to reflect the actual failure on each worker.

The solution for #50981 does NOT include starvation detection. In other words, if a job has certain tasks that were failed by all available workers (and thus all workers are blacklisted for this job & task type) there is no detection that this happened. As a result, the job will be stuck in 'active' status without it ever having a chance of being finished. We should have the Manager regularly inspect queued tasks, to see if there is still at least one worker that is not blacklisted and able to execute them. If not, we can mark those tasks as 'failed' to reflect the actual failure on each worker.
Sybren A. Stüvel self-assigned this 2018-12-17 13:57:17 +01:00
Sybren A. Stüvel commented 2018-12-17 13:57:17 +01:00
Author
Owner
Copy Link

Added subscriber: @dr.sybren

Added subscriber: @dr.sybren
blender-admin commented 2019-02-12 14:40:41 +01:00
Copy Link

This issue was referenced by archive/flamenco-manager@72c46706ea

This issue was referenced by archive/flamenco-manager@72c46706ea801c7400cf5659d888dcf9bb05a6f0
Sybren A. Stüvel commented 2019-02-12 16:09:58 +01:00
Author
Owner
Copy Link

Changed status from 'Open' to: 'Resolved'

Changed status from 'Open' to: 'Resolved'
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
Good First Issue
  
Priority
High

  
Priority
Low

  
Priority
Normal

  
Status
Archived

  
Status
Confirmed

  
Status
Needs Info from Developers

  
Status
Needs Information from User

  
Status
Needs Triage

  
Status
Resolved

  
Type
Bug

  
Type
Design

  
Type
Job Type
Custom Flamenco job type

  
Type
Known Issue

  
Type
Patch

  
Type
Report

  
Type
To Do

No Label
Good First Issue
Priority
High
Priority
Low
Priority
Normal
Status
Archived
Status
Confirmed
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Job Type
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
Sybren A. Stüvel
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: studio/flamenco#59491
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?