Sync branch magefile with main #104308

Merged
Sybren A. Stüvel merged 85 commits from abelli/flamenco:magefile into magefile 2024-05-13 16:26:32 +02:00

85 Commits

Author SHA1 Message Date
b8b1f8dcad Sync version 2024-05-12 11:14:57 -03:00
0f8e5152d2 Sync deps with go mod tidy 2024-05-12 11:14:36 -03:00
7c4659d9a6 Merge branch 'main' into magefile 2024-05-12 10:46:58 -03:00
ae0774b440 Manager: add unit tests
Add a few more unit tests for the persistence layer. The goal is to have
100% coverage of the happy flow, to aid in conversion from GORM to sqlc.

No functional changes.
2024-05-11 10:35:17 +02:00
df334deca5 Add shellSplit(someString) function to the job compiler scripts
Add a function `shellSplit(string)` to the global namespace of job
compiler scripts. It splits a string into an array of strings using
shell/CLI semantics.

For example: `shellSplit("--python-expr 'print(1 + 1)'")` will return
`["--python-expr", "print(1 + 1)"]`.
2024-05-07 12:39:13 +02:00
c5ae2916db Add label to job settings
This gives job type authors more control over how settings are presented
in Blender's job submission GUI. If a job setting does not define a
label, its `key` is used to generate one (like Flamenco 3.5 and older).

Note that this isn't used in the web interface yet.
2024-05-07 11:58:44 +02:00
5ee4871e39 OAPI: regenerate code 2024-05-07 11:55:56 +02:00
e712e9450c OAPI: add label to AvailableJobSetting 2024-05-07 11:54:49 +02:00
2146388abe Website: reword the 'worker cannot find manager' FAQ entry
Reword so that the section starts with the suggestion that each problem
has a solution. And make it an enumerated list to clarify the structure
of the answer.
2024-05-06 11:16:01 +02:00
153cfe7b5c Improve Troubleshooting FAQ: Add firewall and antivirus check
Updated the troubleshooting section of the FAQ to include guidance on checking the firewall and potential third-party antivirus issues when the Worker cannot connect to the Manager. This enhances the user experience by addressing common connectivity issues more comprehensively.
2024-05-04 18:19:44 +02:00
fde20af852 Changelog: mark 3.6 as 'in development' 2024-04-24 17:52:17 +02:00
85d7cb07e7 Website: bump experimental version to 3.6-alpha0 2024-04-24 17:46:05 +02:00
68ac3c03e3 Add-on: compatibility with Python 3.9
Remove some Python 3.10 features to make the add-on compatible with py39.
This is the Python version that's bundled with Blender 2.93 LTS, for which
I got a request to see if it could be supported.

The Blender version still isn't officially supported, but this should make
things at least not immediately fail.
2024-04-24 17:32:01 +02:00
d279f91549 Bumped version to 3.6-alpha0 2024-04-24 17:32:01 +02:00
f7aef5bfca Changelog: add link to MQTT documentation 2024-04-16 11:16:27 +02:00
b4cc9db2db Mark 3.5 as released today 2024-04-16 11:13:45 +02:00
bb772841b7 Revert "Bumped version to 3.6-alpha0"
This reverts commit 7f14e6705d. v3.5 still
needs today's date as release date in the changelog.
2024-04-16 11:12:48 +02:00
7f14e6705d Bumped version to 3.6-alpha0 2024-04-16 11:11:54 +02:00
81de246b48 Bumped version to 3.5 2024-04-16 11:03:27 +02:00
c90c1c1260 Website: bump available download version to 3.5 2024-04-16 11:02:40 +02:00
94fba20ef6 Worker: reduce log level of some internal components
Reduce the log level from 'info' to 'debug' on some internal components
of Flamenco Worker. This makes the console output slightly less noisy,
and it's unlikely that these particular messages are commonly needed.
2024-04-16 10:53:29 +02:00
30c92d2854 Website: document when the oom_score_adjust option was added 2024-04-16 10:46:04 +02:00
e2bca9ad61 Worker: add configuration for Linux out-of-memory killer
Add a Worker configuration option to configure the Linux out-of-memory
behaviour. Add `oom_score_adjust=500` to `flamenco-worker.yaml` to increase
the chance that Blender gets killed when the machine runs out of memory,
instead of Flamenco Worker itself.
2024-04-15 17:21:11 +02:00
3974770f36 Manager: refuse to delete workers without foreign key constraints
As a safety measure, refuse to delete Workers from the Manager's database
when foreign key constraints are disabled.

In the long term, the underlying problem should be solved. This is a stop-
gap measure to ensure database consistency.
2024-04-12 10:48:40 +02:00
6c28db780f Manager: refuse to delete worker tags without foreign key constraints
Before deleting a Worker Tag, check that foreign key constraints are
active for the current database connection.

Sometimes GORM decides to create a new database connection by itself,
without telling us, and then foreign key constraints are not active on
it. This commit is a workaround to avoid database corruption.
2024-04-12 10:48:40 +02:00
b313a2020d Refactor: Manager, move some test code into a function of its own
Move some of the Worker Tags test code into a function of its own, to have
a clearer separation between 'the test' and 'what needs to happen to do
this part of the test'.

Also it'll make an upcoming change easier to implement.

No functional changes.
2024-04-12 10:48:40 +02:00
05e35a1cc5 Update Changelog 2024-04-11 15:02:24 +02:00
ea68faa577 Web: improve URL handling to allow for TLS/SSL
Explicitly use the `--mode` flag for the webapp development server
(`vite`) to make the web frontend choose the appropriate HTTP and
WebSocket port to communicate with the backend. This also makes sure
that when accessing the frontend via `https://`, the websocket
connection uses `wss://`.

As a side-effect, this also makes port `:8081` usable in production
environments; it would assume it was the development server and try to
access the backend on port `:8080`.

Reviewed-on: #104296
Reviewed-by: Sybren A. Stüvel <sybren@blender.org>
2024-04-11 15:00:48 +02:00
9ee9c07e76 Website: link to 3rd party bug tracker for 3rd party scripts
3rd part job compiler scripts should have their own tracker and handle
their own bug reports.
2024-04-11 10:55:00 +02:00
e83db45192 Website: sillicon builds are included, not "will be" included 2024-04-06 10:19:38 +02:00
7b139be605 Website: remove link to old Flamenco v2 documentation
The documentation itself has disappeared from the website, and it already
was obsolete for a long time anyway.
2024-04-04 20:37:48 +02:00
d799372639 Website: bump experimental version to 3.5-beta1 2024-04-04 11:18:03 +02:00
a0cb8735c9 Manager: add optional description to job types
This description will be shown as a tooltip in the job submission UI.
2024-04-04 11:12:42 +02:00
2bbb7b48d2 OAPI: regenerate code 2024-04-04 11:12:39 +02:00
03a889345d OAPI: add optional description to job types
This description will be shown as a tooltip in the job submission UI.
2024-04-04 11:08:55 +02:00
cfad4e73f9 Bumped version to 3.5-beta1 2024-04-04 10:54:53 +02:00
3b21437776 Website: bump experimental version to 3.5-beta0 2024-04-04 10:47:23 +02:00
f757deee6a Security: Bump Go version 1.22.2 and golang.org/x/net package
Fix a bunch of security issues by upgrading to Go 1.22.2 and bumping
a few packages to their secure versions.

- [Incorrect forwarding of sensitive headers and cookies on HTTP redirect in net/http](https://pkg.go.dev/vuln/GO-2024-2600)
- [Memory exhaustion in multipart form parsing in net/textproto and net/http](https://pkg.go.dev/vuln/GO-2024-2599)
- [Verify panics on certificates with an unknown public key algorithm in crypto/x509](https://pkg.go.dev/vuln/GO-2024-2600)
- [HTTP/2 CONTINUATION flood in net/http](https://pkg.go.dev/vuln/GO-2024-2687)
2024-04-04 10:44:57 +02:00
bce84bf175 Bumped version to 3.5-beta0 2024-04-04 10:38:42 +02:00
43a452e940 Website: Update the required Go version to Latest 2024-03-25 15:10:36 +01:00
c3b8707390 Worker: reduce log level of 'Blender could not be found' to info level
There's still some confusion that this is a thing to solve, whereas it can
usually safely be ignored. Reduced the log level from Warn to Info to make
the message look more innocent.
2024-03-18 22:39:55 +01:00
00dfbc10b6 Worker: link to FAQ entry when the worker cannot be found 2024-03-18 22:38:20 +01:00
63a578688e Make make test fail fast
Pass `-failfast` to the `go test` command, so that it immediately stops
on test failure. This prevents the need to scroll back to see the actual
error, at the expense of only seeing one failure at a time.
2024-03-16 12:14:41 +01:00
b219f9b1c2 Manager tests: replace assert.NoError() with require.NoError()
Back in the days when I wrote the code, I didn't know about the
`require` package yet. Using `require.NoError()` makes the test code
more straight-forward.

No functional changes, except that when tests fail, they now fail
without panicking.
2024-03-16 12:14:39 +01:00
3f4a9025fe Manager tests: replace assert.NoError() with require.NoError()
Back in the days when I wrote the code, I didn't know about the
`require` package yet. Using `require.NoError()` makes the test code
more straight-forward.

No functional changes, except that when tests fail, they now fail
without panicking.
2024-03-16 11:09:18 +01:00
1fee086cef Website: add MQTT prefix to table of MQTT topics 2024-03-13 19:16:40 +01:00
bd3dd90303 Update CHANGELOG.md
Add recent add-on improvements.
2024-03-13 19:13:37 +01:00
041f792594 Website: document Flamenco's MQTT support 2024-03-13 19:10:51 +01:00
5c60561ddb Website: allow separate page title for TOC
You can now set a page title and a separate title for the table of
contents with:

```
---
title: "Manager Configuration: MQTT"
titleTOC: MQTT
---
```
2024-03-13 18:42:52 +01:00
a23c95a738 Update CHANGELOG.md 2024-03-13 18:29:35 +01:00
91890541a2 Webapp: change Tabulator layout mode to fitDataFill
Change the Tabulator layout mode from `fitData` to `fitDataFill`. The new
value adjusts the layout when the data has changed.
2024-03-13 18:23:41 +01:00
3aec8b15ef Add-on: avoid console spam about the scene's job type property
Avoid these warnings on the console:

```
WARN (bpy.rna): source/blender/python/intern/bpy_rna.cc:1339
  pyrna_enum_to_py: current value '0' matches no enum in 'Scene', 'Scene',
  'flamenco_job_type'
```

The solution was two-fold:
- Use a non-empty string as the identifier for the 'Select a Job Type'
  choice.
- Give the property a default value.
2024-03-11 14:32:29 +01:00
d1fbe8b9f9 Manager: set default MQTT topic prefix to 'flamenco'
Set the default MQTT topic prefix to 'flamenco'. It can still be overridden
by the config in the YAML file, but it's nice to have a sensible default
when people don't configure this.
2024-03-08 16:44:39 +01:00
b476e31c0c Manager: remove unused configuration defaults
Remove commented-out sections in the configuration defaults. They're a
leftover from Flamenco v2.
2024-03-08 16:41:40 +01:00
cbafacdff6 Manager: don't forward task log updates to MQTT
Task log updates are big and frequent, and should not be sent via MQTT.
At least not until we have a practical reason to do so.
2024-03-07 15:22:44 +01:00
358efe7ae0 Manager: perform a database vacuum after migrations
Just to make sure the DB is properly cleaned up after a big migration
happened.
2024-03-06 11:59:17 +01:00
7277286391 Manager: increase 'database open' timeout to 1 minute
Increase the 'database open' timeout from 5 seconds to 1 minute. This
timeout also covers database migrations, and the recently added one that
adds a bunch of `NOT NULL` clauses could time out with the old 5 sec
limit.

The reason this takes long, is that SQLite doesn't directly support
adding `NOT NULL` clauses to columns. The only way to do this is to
create a new table with the desired schema, copy all data over, then
drop the old table. And with a big enough database, this takes time.
2024-03-06 11:50:14 +01:00
16114ee529 Worker: fix Go scheduling issue in sleep command test
Add a 1ms delay in the test loop, so that other goroutines can be scheduled
as well. This should fix #104288.
2024-03-04 14:18:08 +01:00
27cbb2ed0f Manager: increase timeout for database integrity check
With a fuller database, 2 seconds is apparently not always long enough,
so increase the timeout to 10 seconds.
2024-03-04 14:04:59 +01:00
63a2bce600 run go mod tidy
No functional changes.
2024-03-04 13:53:24 +01:00
3b4da656c9 Add-on: cache manager info in a JSON file
Instead of storing the cached manager info in the Blender preferences,
store the info in a JSON file. The file is located in the user prefs
folder (`~/.config/blender/{version}/config` on Linux).

This also reduces the number of 'refresh' operators to a single one, which
then fetches all necessary info from the Manager.

This fixes an issue (reported via chat) where worker tags were sometimes
not retained across file saves.
2024-03-04 13:08:53 +01:00
a4e5eef83e Manager: fix database migration 0004
Fix the database migration that adds `NOT NULL` clauses. It used
`INSERT INTO temp_x SELECT * from x;`, and the `*` returns the fields in
the order they are defined on the table. Since this might be different from
the order that the `INSERT INTO temp_x` expects, strange problems can
happen where columns get swapped (or constraints can fail on columns that
they should not fail for, because they got fed data from a different
column).
2024-03-04 13:06:09 +01:00
f9bb7afcc6 Add-on: fix mypy warnings
No functional changes.
2024-03-04 12:46:38 +01:00
7b72d0ca43 Refactor: move jobs-related queries to queries_jobs.sql
This makes it easier to later also create `query_workesr.sql`,
`query_meta.sql` etc. so that the sqlc-generated code can follow the
same subdivision as the persistence service code itself.

No functional changes.
2024-03-03 23:27:55 +01:00
b102b73a1f Refactor: convert more job functions to sqlc
No functional changes.
2024-03-03 23:23:51 +01:00
1ac796d0d8 Refactor: Manager: remove unused query from queries.sql
No functional changes.
2024-03-03 22:42:37 +01:00
3fbb3cde34 Manager: SQLC rename Uuid to UUID
No functional changes.
2024-03-03 20:54:43 +01:00
c046094880 Manager: start replacing GORM with SQLC
GORM has certain downsides:

- Code-first approach, where queries have to be translated to the Go code
  required to execute them.
- GORM comes with its own SQLite implementation, which doesn't provide an
  on-connect callback. This means that new connections cannot correctly
  enable foreign key constraints, causing database consistency issues.

[SQLC](https://sqlc.dev/) solves these issues for us.

This commit doesn't fully replace GORM with SQLC, but introduces it for
a few queries. Once all queries have been converted, GORM can be removed
completely.
2024-03-03 20:15:39 +01:00
7b31eba8d7 Webapp: improve positioning of the farm status indicator
Split the header into two or three parts, depending on the number of
columns shown. The farm status indicator will be above the middle column
(in 3 col mode) or at the right edge of the left column (in 2 col mode).

Also I reverted the hiding of the farm status when SocketIO has
disconnected, as that disconnect happens when navigation between tabs.
That created a too 'blinky' interface, so now it just shows the last-known
farm status.
2024-03-02 23:11:33 +01:00
7bf121e93e Webapp: show explanation in farm status tooltip 2024-03-02 22:32:02 +01:00
63e3c8de37 Webapp: make SocketIO reconnect faster
The exponential backoff was getting a bit too long, making the webapp
sometimes very slow to reconnect. This is now limited to max 3 seconds.
2024-03-02 22:20:42 +01:00
c0c70758c6 Webapp: hide farm status when SOcketIO is disconnected
Better to not show the farm status if the connection is lost.
2024-03-02 22:20:01 +01:00
1e7c059d12 Manager: check the farm status quickly after startup
The database is polled every 30 seconds to determine the farm status; at
startup the first poll is done after 1 second to get a faster status.

Note that when jobs and workers change their status, the farm status is
always updated.
2024-03-02 22:09:53 +01:00
7eb5eb68a3 Manager: ensure foreign keys are enabled in periodic integrity check
There are still issues with foreign keys getting disabled, so enable them
in the periodic database consistency check.

A more permanent solution is likely to drop GORM and switch to something
else that gives us an on-connect-callback, which can then be used to
turn on foreign key constraints for every connection made.
2024-03-01 23:42:04 +01:00
f763d12cd8 Update CHANGELOG 2024-03-01 22:40:21 +01:00
59bf389018 Webapp: show farm status in header bar
Show the farm status in the webapp header bar, and respond to farm status
events to update it when necessary.
2024-03-01 22:36:38 +01:00
c1a9b1e877 Manager: force a poll of the farm status when a job/worker changes state
This introduces the concept of 'event listener', which is now used by
the farm status service to respond to events on the event bus.

This makes it possible to reduce the regular poll period from 5 to 30
seconds. That's now only necessary as backup, just in case events are
missed or otherwise things change without the event bus logic noticing.
2024-03-01 22:36:38 +01:00
9bfb53a7f6 Manager: log error when an event doesn't have a SocketIO event type
SocketIO has 'rooms' and 'event types'. The 'event type' is set via
reflection of the OpenAPI type of the event payload. This has to be set
up in a mapping, though, and if that mapping is incomplete, an error will
now be logged.
2024-03-01 22:36:26 +01:00
ee7af29748 Manager: fix unit test for farm status events 2024-03-01 22:36:26 +01:00
54f7878045 Manager: add farm status events to the event bus
Send an event to the event bus whenever the farm status changes. The event
contains a farm status report (like `{status: "active"}`), and is sent to
the `/status` topic.

Note that at this moment the status is only polled every X seconds, and
thus may lag behind other events.
2024-03-01 08:41:35 +01:00
8cf19876c0 OAPI: regenerate code 2024-03-01 08:36:40 +01:00
3b1421c227 OAPI: add farm status events to the event bus 2024-03-01 08:36:19 +01:00
61cc8ff04d Manager: implement API operation to get the farm status
Add a new API operation to get the overall farm status. This is based on
the jobs and workers, and their status.

The statuses are:

- `active`: Actively working on jobs.
- `idle`: Farm could be active, but has no work to do.
- `waiting`: Work has been queued, but all workers are asleep.
- `asleep`: Farm is idle, and all workers are asleep.
- `inoperative`: Cannot work: no workers, or all are offline/error.
- `starting`: Farm is starting up.
- `unknown`: Unexpected configuration of worker and job statuses.
2024-02-29 20:42:28 +01:00
d9ffe8a1b6 OAPI: regenerate code 2024-02-29 20:38:38 +01:00
0c7bf3bf3b OAPI: add farm status service
Add an OpenAPI operation to fetch the overall farm status from the Manager.
2024-02-29 20:27:44 +01:00