Commit Graph

1694 Commits

Author SHA1 Message Date
45cd62b558 Fix warning in PyCharm by explicitly declaring abc.ABC as superclass
This fixes a warning that `AbstractSubprocessCommand` doesn't implement all
abstract methods.
2018-11-22 16:09:57 +01:00
feace5409f Added MyPy runner to unit tests 2018-11-22 16:09:57 +01:00
d004d961fa Fixed last MyPy error 2018-11-21 11:30:19 +01:00
421ec4b658 Only stop task if task-to-stop is the same as currently-executing-task
The 'may-i-run' endpoint may say 'no' to us running task X, but when the
response comes in and is handled the worker may already be working on task
Y. This case is now recognised, and the 'no' will be ignored.
2018-11-16 15:45:39 +01:00
4161328115 Less verbose debug log when POSTing to Manager
The POSTed logs can get huge, so we limit to the first 80 characters now.
2018-11-16 10:35:59 +01:00
dda04b98aa Some more debug log reduction 2018-11-15 15:28:20 +01:00
9911e9e6c2 Use pid=%d instead of PID %r for consistency with other PID logging 2018-11-15 14:44:15 +01:00
e7d12ccff8 Master → Manager 2018-11-15 14:22:54 +01:00
29c46feaf2 Log fetch_task stuff in a separate logger 2018-11-15 14:14:32 +01:00
b4c38f0e26 Log pre-task sanity checking in sub-logger
This allows us to hide those specific logs from the debug log.
2018-11-15 14:09:57 +01:00
9bde5d0e93 Splice '.line' into logger name for debug log
Now the log is sent to
`flamenco_worker.commands.blender_render.line.(task_id=xxx, command_idx=0)`
instead of
`flamenco_worker.commands.blender_render.(task_id=xxx, command_idx=0).line`

This makes configuration of log levels easier.
2018-11-15 13:53:11 +01:00
9978f280bf Include proper PIDs in unit tests 2018-11-15 13:52:15 +01:00
a350ed2e5a Absolute imports for tests to make PyCharm happy 2018-11-15 13:52:05 +01:00
55d0ce87ea Only send render status updates every 30 seconds
Previously it was done for every line of Blender output that contained
render time info, which was way too often.
2018-11-15 12:48:14 +01:00
6acbc2cc25 Fix logging failed command 2018-11-15 12:43:22 +01:00
49b53b71f3 Send subprocess read-lines to separate logger
This allows the logging config to mask those out if required.
2018-11-15 12:38:42 +01:00
e5205687e3 Don't log 'unable to find remaining time', it's too spammy 2018-11-15 12:35:40 +01:00
d3aab505b3 Better logging when a command fails 2018-11-15 12:23:29 +01:00
252af665fa Bumped version to 2.2-dev6 2018-11-15 12:14:25 +01:00
3a218da3cb Fixed race condition in subprocess PID file handling
This doesn't fix the root cause for multiple subprocesses running, but it
does kill better when it finds another subprocess in a race condition.
2018-11-15 12:13:51 +01:00
cb6fa7e893 Upgraded from Python 3.5 → 3.7 2018-11-15 12:13:51 +01:00
8c6039f65f Added TODO 2018-11-15 11:43:37 +01:00
ac7a5aefd0 More type annotations 2018-11-15 11:43:37 +01:00
826f53b345 Changed from 'task' to 'future'
This makes a clearer distinction between Flamenco tasks ('task') and
AsyncIO tasks ('future'). The AsyncIO Task class is a subclass of its
Future class.
2018-11-15 11:43:37 +01:00
73439de819 Ignore more flamenco-worker logs 2018-11-15 10:44:24 +01:00
518cf77b8c Include mypy in development packages + solved lots of warnings
Haven't written the mypy-running unit test yet, that'll come when all
warnings are solved. This commit doesn't solve all of them, just some
simple ones.
2018-11-15 10:43:59 +01:00
fe901ffd7a Log more process IDs
Also changed 'PID=...' to 'pid=...' in the logs sent to Flamenco Manager,
so that all our logs use lower-case 'pid'.
2018-11-15 10:42:04 +01:00
04c611b122 Fixed bug in deploy script 2018-11-13 17:26:04 +01:00
72aa77e637 Added automated deployment script for Blender Animation Studio 2018-11-13 10:26:55 +01:00
6e1b1b585d Updated changelog 2018-11-13 10:13:24 +01:00
9e55ccd6af Moved from requirements.txt to Pipfile
This also means that we need a `tests/__init__.py` file; this file makes
py.test undertand what is our sources directory, and thus allows them to
`import flamenco_worker`. As a result, the test imports from tests/*.py
need to change to relative imports.
2018-11-13 10:00:58 +01:00
4492405aaf Don't late-import psutil and other modules
Early detection of missing dependencies trumps a few ms of startup time.
2018-11-13 09:46:45 +01:00
17709f1c08 Bumped version to 2.2-dev5 2018-11-12 15:11:05 +01:00
0b456b95be Include the process PID in the log lines
This will help debugging the crazy double-Blender-running we're seeing
in the studio.
2018-11-12 15:09:04 +01:00
c1bc3fc167 Write PID file for subprocesses
This prevents multiple subprocess commands running at once. They shouldn't
be doing that in the first place, but we have a strange bug where multiple
Blenders seem to be running on the same machine.
2018-11-12 15:09:04 +01:00
cb0aa251f9 Added missing unit tests 2018-11-12 13:34:01 +01:00
af32227a2d Typo 2018-11-12 13:26:43 +01:00
2ce7358cf9 Fixed sanity check for Python 3.5 2018-11-12 13:24:03 +01:00
bc16faccff Bumped version to 2.2-dev4 2018-11-12 12:45:49 +01:00
b9a25c2f3b Added pre-task sanity check
This check is performed every time before trying to fetch a task. It checks
for readability or writability of certain filesystem paths, to prevent
tasks from failing when the shared storage hasn't been mounted. Instead,
the worker will go to `error` status and sleep for 10 minutes before trying
again.
2018-11-12 12:43:20 +01:00
3386072774 Bumped attrs (16.3.0 → 18.2.0) and requests (2.12.4 → 2.20.1) libraries 2018-11-12 12:12:02 +01:00
bd67aa90bd Formatting 2018-11-12 10:02:05 +01:00
155d91c562 Split Worker.fetch_task() into fetch_task() and execute_task()
This makes the naming more consistent, and makes it easier to add more
functionality without growing an already-big function.
2018-11-12 09:33:48 +01:00
53dc8189f9 Bumped version to 2.2-dev3 2018-09-12 10:45:17 +02:00
f60cc8c57e Added hack to avoid clogging the logs
Cycles and other render engines produce a line of log for each object in
the scene. In the Spring project there are so many objects (leaves, twigs,
pebbles) that it chokes the Manager with logs.

For now we have some custom code to swallow those lines, in lieu of a
logging system that can handle those volumes properly.
2018-09-12 10:31:50 +02:00
453fd6e4f8 Bumped version to 2.2-dev2 2018-09-11 18:24:44 +02:00
3a3bf547e9 Fixed infinite task update loop when Manager responds with 404 Not Found
Task updates would be sent in an infinite loop when the Manager didn't
know the task, blocking all other task updates.
2018-09-11 18:24:15 +02:00
8cefc5c579 Bumped version to 2.2-dev1 2018-06-15 14:20:04 +02:00
19a969aa5e Updated changelog 2018-06-15 14:19:49 +02:00
99f871dcc3 Added log_a_lot command for debugging purposes
It was quite useful in finding the recent memory leak.
2018-06-15 14:19:06 +02:00