Include final render time in task details #99731

New Issue

Sybren A. Stüvel · 2022-07-15T10:21:01+02:00

Sybren A. Stüvel commented

2022-07-15 10:21:01 +02:00

After a Blender render task is done, the task details should contain timing information.

It might be good to split this up into two bits of info:

- Generic timing of the task, which can be implemented for any task type.
- Specific Blender-reported render time, which of course only applies to Blender tasks. Maybe Flamenco should get a way for Workers to publish "statistics" of the commands they execute. Those commands can then determine for themselves how to get those statistics (for example looking for render time in Blender's output).

After a Blender render task is done, the task details should contain timing information. It might be good to split this up into two bits of info: - - [ ] Generic timing of the task, which can be implemented for any task type. - - [ ] Specific Blender-reported render time, which of course only applies to Blender tasks. Maybe Flamenco should get a way for Workers to publish "statistics" of the commands they execute. Those commands can then determine for themselves how to get those statistics (for example looking for render time in Blender's output).

Sybren A. Stüvel commented

2022-07-15 10:21:01 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Sybren A. Stüvel commented

2022-07-15 10:21:01 +02:00

Added subscriber: @dr.sybren

Sybren A. Stüvel added

and removed

labels 2023-02-17 11:12:45 +01:00

saurabh jadhav commented

2023-12-29 09:56:09 +01:00

Should i start crating this feature ?
it would be useful to show worker time statistics
also can be used by manager for worker task scheduling policy

the way i plan to implement this is by adding two more entries in the databse as :
started_at and completed_at in the task table of the database same as updated_at column

can you guide me how to change the schema of the database table in this project and necessary referances for it ?

Should i start crating this feature ? it would be useful to show worker time statistics also can be used by manager for worker task scheduling policy the way i plan to implement this is by adding two more entries in the databse as : started_at and completed_at in the task table of the database same as updated_at column can you guide me how to change the schema of the database table in this project and necessary referances for it ?

Sybren A. Stüvel commented

2023-12-29 11:14:29 +01:00

Hi @saurabhjadhav1911 , thanks for your offer to help!

Please also join #flamenco on Blender Chat, it's the best place to ask questions about Flamenco development.

the way i plan to implement this is by adding two more entries in the databse as :
started_at and completed_at in the task table of the database same as updated_at column

Although this could be a reasonable start, I think it might be better to have a new table task_execution_log or something like that, that contains task execution events. The thing is -- a task can be run multiple times. A Worker might fail at a task (for example because it doesn't have enough memory, crashing Blender), after which another Worker can pick it up and complete it. For such cases I think it would be easier to build, and easier to reason about, a table that has "events" for task status changes, something like:

`id`	`task_id`	`created_at`	`from_status`	`to_status`	`worker_id`	`description`
`1`	`421`	`2023-12-29T11:04:00+00:00`	`queued`	`active`	`555`	Worker ws-sybren started task
`1`	`421`	`2023-12-29T11:12:00+00:00`	`active`	`completed`	`555`	Worker ws-sybren completed task
`1`	`421`	`2023-12-29T11:30:00+00:00`	`completed`	`queued`	`NULL`	Task was requeued

There task_id is a foreign key to the tasks table, worker_id a nullable foreign ID to the workers table. Note that there is no updated_at, as entries in this table are not supposed to be ever updated.

This is just a rough idea that I have in my mind, if you have ideas to improve, please do discuss!

can you guide me how to change the schema of the database table in this project and necessary referances for it ?

That's done with Goose. Basically you create a new file in the internal/manager/persistence/migrations directory. Flamenco will pick it up and use Goose to change the database schema.

To test these files you can also use make db-migrate-status to get the migration status, make db-migrate-up to do a migration step, and make db-migrate-down to roll back a migration step. This makes it possible to test your migration without having to recompile & run Flamenco Manager. In order to run these commands you need the goose binary, which you can install with go install github.com/pressly/goose/v3/cmd/goose@latest.

That should get you started. If you have any more questions, feel free to reach out.

Hi @saurabhjadhav1911 , thanks for your offer to help! Please also join [#flamenco on Blender Chat](https://blender.chat/channel/flamenco), it's the best place to ask questions about Flamenco development. > the way i plan to implement this is by adding two more entries in the databse as : > `started_at` and `completed_at` in the task table of the database same as `updated_at` column Although this could be a reasonable start, I think it might be better to have a new table `task_execution_log` or something like that, that contains task execution events. The thing is -- a task can be run multiple times. A Worker might fail at a task (for example because it doesn't have enough memory, crashing Blender), after which another Worker can pick it up and complete it. For such cases I think it would be easier to build, and easier to reason about, a table that has "events" for task status changes, something like: | `id` | `task_id` | `created_at` | `from_status` | `to_status` | `worker_id` | `description` | | -- | -- | -- | -- | -- | -- | -- | | `1` | `421` | `2023-12-29T11:04:00+00:00` | `queued` | `active` | `555` | Worker ws-sybren started task | | `1` | `421` | `2023-12-29T11:12:00+00:00` | `active` | `completed` | `555` | Worker ws-sybren completed task | | `1` | `421` | `2023-12-29T11:30:00+00:00` | `completed` | `queued` | `NULL` | Task was requeued | There `task_id` is a foreign key to the `tasks` table, `worker_id` a nullable foreign ID to the `workers` table. Note that there is no `updated_at`, as entries in this table are not supposed to be ever updated. This is just a rough idea that I have in my mind, if you have ideas to improve, please do discuss! > can you guide me how to change the schema of the database table in this project and necessary referances for it ? That's done with [Goose](https://github.com/pressly/goose). Basically you create a new file in the `internal/manager/persistence/migrations` directory. Flamenco will pick it up and use Goose to change the database schema. To test these files you can also use `make db-migrate-status` to get the migration status, `make db-migrate-up` to do a migration step, and `make db-migrate-down` to roll back a migration step. This makes it possible to test your migration without having to recompile & run Flamenco Manager. In order to run these commands you need the goose binary, which you can install with `go install github.com/pressly/goose/v3/cmd/goose@latest`. That should get you started. If you have any more questions, feel free to reach out.

saurabh jadhav commented

2023-12-29 18:11:33 +01:00

I have added the migration with new table

240c5c4abb
please verify this

now do i need to add an API POST endpoint for directly writing to task_execution_log table from worker ?
or the database table need to be updated inside manager when state status changes through any of the existing worker/task endpoints ?

I have added the migration with new table https://projects.blender.org/saurabhjadhav1911/flamenco/commit/240c5c4abba8fa1fc6fccf847e19a8cecaf79eb5 please verify this now do i need to add an API POST endpoint for directly writing to task_execution_log table from worker ? or the database table need to be updated inside manager when state status changes through any of the existing worker/task endpoints ?

Sybren A. Stüvel commented

2023-12-30 18:24:14 +01:00

  CONSTRAINT `fk_task_execution_log_worker` FOREIGN KEY (`worker_id`) REFERENCES `workers`(`id`) ON DELETE CASCADE

Now that I think of it some more: here it's better to have an ON DELETE SET NULL for this constraint. Also add a worker_name column. That way, if the worker is ever removed, the log on the task remain valid. The fk_task_execution_log_task constraint is good -- the logs should disappear when the task gets removed.

or the database table need to be updated inside manager when state status changes through any of the existing worker/task endpoints ?

This one. As a design "rule of thumb" I try to keep the workers as dumb and simple as possible. So if there's bookkeeping that can be done by the Manager, it should be done by the Manager.

Please also take a look at #104273. Not that that is a requirement for this feature, but it might give you some ideas on how to implement this feature in a way that makes it possible to transition to some internal event system later.

```sql CONSTRAINT `fk_task_execution_log_worker` FOREIGN KEY (`worker_id`) REFERENCES `workers`(`id`) ON DELETE CASCADE ``` Now that I think of it some more: here it's better to have an `ON DELETE SET NULL` for this constraint. Also add a `worker_name` column. That way, if the worker is ever removed, the log on the task remain valid. The `fk_task_execution_log_task` constraint is good -- the logs should disappear when the task gets removed. > or the database table need to be updated inside manager when state status changes through any of the existing worker/task endpoints ? This one. As a design "rule of thumb" I try to keep the workers as dumb and simple as possible. So if there's bookkeeping that can be done by the Manager, it should be done by the Manager. Please also take a look at #104273. Not that that is a requirement for this feature, but it might give you some ideas on how to implement this feature in a way that makes it possible to transition to some internal event system later.

Sign in to join this conversation.