Idea: Task scheduling using random sampling #51197
Just an idea. Right now we have job & task priorities, and the task with the highest priority of the job with the highest priority is run first. As a result, jobs with higher priorities will completely starve jobs with lower priorities.
To give an example: if I want to render something on Flamenco, but with low priority to not disturb the Secret Lair renders, effectively I have to wait until the Secret Lair is completely done. Instead, I would rather see Flamenco pick up a task of a low-priority job every now and then. That way low-prio jobs are worked on, just slower than high-prio jobs.
I think this can be done using nonuniform random sampling. By using the job priority to determine the probability that a task of that job is chosen, there will still be a small probability for low-prio tasks to be executed. This of course should only be done for the job selection; tasks within a job should keep their relative execution order.
We could even take it one step further, and take job age into account. That way older jobs are more likely to be worked on than new jobs. Not sure if that's worth the effort, though.
not sure if this is still relevant: netrender had something like this based on rules, see https://developer.blender.org/diffusion/BA/browse/master/netrender/balancing.py
@lichtwerk thanks for thinking along with us :)
I'm not sure how relevant such a rule-based system is, though. We'll probably be able to do something suitable by changing the monster MongoDB aggregation query to include some randomisation of the job priority field.
@dr.sybren : still have to make myself familiar with flamenco [I will have to do this for an upcoming studio job...].
beauty of the rule-based thingie was [iirc], that you could add something easily without having to touch existing code.
So in the case of netrender I added something like "Priority-frames" [frames to be rendered first] without too much hassle. (I think the rules could even be mixed per job, but I would need to have a look again, this was ages ago...)
Anyways, looking forward to diving into Flamenco/Attract etc. (will probably bug you with some questions then...)
Feel free to poke me on #flamenco on Blender Chat ;-)
A new generation of Flamenco (version 3.0) is in development (in the
main branch at https://developer.blender.org/diffusion/F/), so all issues with Flamenco version 2 or older will be archived.
No due date set.
No dependencies set.
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?