# Idea: Task scheduling using random sampling #51197

Closed
opened 2017-04-13 14:15:02 +02:00 by Sybren A. Stüvel · 9 comments
Owner

Just an idea. Right now we have job & task priorities, and the task with the highest priority of the job with the highest priority is run first. As a result, jobs with higher priorities will completely starve jobs with lower priorities.

To give an example: if I want to render something on Flamenco, but with low priority to not disturb the Secret Lair renders, effectively I have to wait until the Secret Lair is completely done. Instead, I would rather see Flamenco pick up a task of a low-priority job every now and then. That way low-prio jobs are worked on, just slower than high-prio jobs.

I think this can be done using nonuniform random sampling. By using the job priority to determine the probability that a task of that job is chosen, there will still be a small probability for low-prio tasks to be executed. This of course should only be done for the job selection; tasks within a job should keep their relative execution order.

We could even take it one step further, and take job age into account. That way older jobs are more likely to be worked on than new jobs. Not sure if that's worth the effort, though.

Just an idea. Right now we have job & task priorities, and the task with the highest priority of the job with the highest priority is run first. As a result, jobs with higher priorities will completely starve jobs with lower priorities. To give an example: if I want to render something on Flamenco, but with low priority to not disturb the Secret Lair renders, effectively I have to wait until the Secret Lair is completely done. Instead, I would rather see Flamenco pick up a task of a low-priority job every now and then. That way low-prio jobs are worked on, just slower than high-prio jobs. I think this can be done using nonuniform random sampling. By using the job priority to determine the probability that a task of that job is chosen, there will still be a small probability for low-prio tasks to be executed. This of course should only be done for the job selection; tasks within a job should keep their relative execution order. We could even take it one step further, and take job age into account. That way older jobs are more likely to be worked on than new jobs. Not sure if that's worth the effort, though.
Author
Owner

Changed status to: 'Open'

Changed status to: 'Open'
Author
Owner

Added subscribers: @dr.sybren, @fsiddi, @eyecandy, @pablovazquez

Added subscribers: @dr.sybren, @fsiddi, @eyecandy, @pablovazquez
First-time contributor

First-time contributor

not sure if this is still relevant: netrender had something like this based on rules, see https://developer.blender.org/diffusion/BA/browse/master/netrender/balancing.py

not sure if this is still relevant: netrender had something like this based on rules, see https://developer.blender.org/diffusion/BA/browse/master/netrender/balancing.py
Author
Owner

@lichtwerk thanks for thinking along with us :)

I'm not sure how relevant such a rule-based system is, though. We'll probably be able to do something suitable by changing the monster MongoDB aggregation query to include some randomisation of the job priority field.

@lichtwerk thanks for thinking along with us :) I'm not sure how relevant such a rule-based system is, though. We'll probably be able to do something suitable by changing the [monster MongoDB aggregation query](https://developer.blender.org/source/flamenco-manager/browse/master/flamenco/scheduler.go;cfe561c79e755d0e58e9a79ef707fca35701dbee\$264) to include some randomisation of the job priority field.
First-time contributor

@dr.sybren : still have to make myself familiar with flamenco [I will have to do this for an upcoming studio job...].
beauty of the rule-based thingie was [iirc], that you could add something easily without having to touch existing code.
So in the case of netrender I added something like "Priority-frames" [frames to be rendered first] without too much hassle. (I think the rules could even be mixed per job, but I would need to have a look again, this was ages ago...)

Anyways, looking forward to diving into Flamenco/Attract etc. (will probably bug you with some questions then...)

@dr.sybren : still have to make myself familiar with flamenco [I will have to do this for an upcoming studio job...]. beauty of the rule-based thingie was [iirc], that you could add something easily without having to touch existing code. So in the case of netrender I added something like "Priority-frames" [frames to be rendered first] without too much hassle. (I think the rules could even be mixed per job, but I would need to have a look again, this was ages ago...) Anyways, looking forward to diving into Flamenco/Attract etc. (will probably bug you with some questions then...)
Author
Owner

Feel free to poke me on #flamenco on Blender Chat ;-)

Feel free to poke me on [#flamenco on Blender Chat](https://blender.chat/channel/flamenco) ;-)
Author
Owner

Changed status from 'Confirmed' to: 'Archived'

Changed status from 'Confirmed' to: 'Archived'
Author
Owner

A new generation of Flamenco (version 3.0) is in development (in the `main` branch at https://developer.blender.org/diffusion/F/), so all issues with Flamenco version 2 or older will be archived.

A new generation of Flamenco (version 3.0) is in development (in the `main` branch at https://developer.blender.org/diffusion/F/), so all issues with Flamenco version 2 or older will be archived.
No Label
No Milestone
No Assignees
2 Participants