Commit Graph

40 Commits

Author SHA1 Message Date
2a0790617c Pass FFmpeg binary as ffmpeg_cmd and split the command
This is for consistency with the `blender_cmd` setting of Blender render
commands. The splitting is done so that extra CLI commands can be given
in the Manager config.
2018-11-23 16:12:38 +01:00
d2e3791cbe Added command 'create_video' to run FFmpeg
This requires FFmpeg to be installed (not just for the command, but also
for running the accompanying unit test). Because of this external
dependency, the worker has to declare task_type='video-encoding' in its
configuration file before it gets such tasks. This is not enabled by
default.
2018-11-22 16:09:57 +01:00
feace5409f Added MyPy runner to unit tests 2018-11-22 16:09:57 +01:00
421ec4b658 Only stop task if task-to-stop is the same as currently-executing-task
The 'may-i-run' endpoint may say 'no' to us running task X, but when the
response comes in and is handled the worker may already be working on task
Y. This case is now recognised, and the 'no' will be ignored.
2018-11-16 15:45:39 +01:00
9978f280bf Include proper PIDs in unit tests 2018-11-15 13:52:15 +01:00
a350ed2e5a Absolute imports for tests to make PyCharm happy 2018-11-15 13:52:05 +01:00
6acbc2cc25 Fix logging failed command 2018-11-15 12:43:22 +01:00
3a218da3cb Fixed race condition in subprocess PID file handling
This doesn't fix the root cause for multiple subprocesses running, but it
does kill better when it finds another subprocess in a race condition.
2018-11-15 12:13:51 +01:00
826f53b345 Changed from 'task' to 'future'
This makes a clearer distinction between Flamenco tasks ('task') and
AsyncIO tasks ('future'). The AsyncIO Task class is a subclass of its
Future class.
2018-11-15 11:43:37 +01:00
fe901ffd7a Log more process IDs
Also changed 'PID=...' to 'pid=...' in the logs sent to Flamenco Manager,
so that all our logs use lower-case 'pid'.
2018-11-15 10:42:04 +01:00
9e55ccd6af Moved from requirements.txt to Pipfile
This also means that we need a `tests/__init__.py` file; this file makes
py.test undertand what is our sources directory, and thus allows them to
`import flamenco_worker`. As a result, the test imports from tests/*.py
need to change to relative imports.
2018-11-13 10:00:58 +01:00
0b456b95be Include the process PID in the log lines
This will help debugging the crazy double-Blender-running we're seeing
in the studio.
2018-11-12 15:09:04 +01:00
c1bc3fc167 Write PID file for subprocesses
This prevents multiple subprocess commands running at once. They shouldn't
be doing that in the first place, but we have a strange bug where multiple
Blenders seem to be running on the same machine.
2018-11-12 15:09:04 +01:00
cb0aa251f9 Added missing unit tests 2018-11-12 13:34:01 +01:00
155d91c562 Split Worker.fetch_task() into fetch_task() and execute_task()
This makes the naming more consistent, and makes it easier to add more
functionality without growing an already-big function.
2018-11-12 09:33:48 +01:00
f60cc8c57e Added hack to avoid clogging the logs
Cycles and other render engines produce a line of log for each object in
the scene. In the Spring project there are so many objects (leaves, twigs,
pebbles) that it chokes the Manager with logs.

For now we have some custom code to swallow those lines, in lieu of a
logging system that can handle those volumes properly.
2018-09-12 10:31:50 +02:00
3a3bf547e9 Fixed infinite task update loop when Manager responds with 404 Not Found
Task updates would be sent in an infinite loop when the Manager didn't
know the task, blocking all other task updates.
2018-09-11 18:24:15 +02:00
e8a4093104 Fix memory leak in task update queue
I'm guessing the do_db_push() async function execution was starved by
other asyncio tasks, causing Python to remember all the to-be-queued
payloads in memory. By making the function synchronous this doesn't happen.
2018-06-15 14:18:24 +02:00
a4bb3718aa Quick slap hack: refuse to fetch tasks when outgoing queue is full
This should prevent an accumulation of task updates, when the updates
are generated faster than they can be sent to the Manager.
2018-04-18 10:30:27 +02:00
7be0ae2e8a Fix unittest 2017-12-15 17:17:18 +01:00
3200929bde Added support for passing Python scripts to Blender in task definitions. 2017-12-15 16:47:50 +01:00
ea49ddc264 Implemented Manager-dictated state changes.
The Manager can now dictate that the Worker moves to an 'asleep' state.
2017-10-19 18:57:06 +02:00
3838f56b12 Don't push task status 'canceled' when we are no longer allowed to run it.
When the "may I run" system says "no", we shouldn't push a "canceled"
status to the Manager, as this could overwrite another valid status that's
there for some other reason.
2017-09-29 14:45:58 +02:00
1c16cf37c4 Include response text when Manager refuses task update.
This refusal can be because another worker is working on the task, but
can also be for other reasons (like the task no longer being runnable).
2017-09-29 14:45:58 +02:00
c07105e04e Renamed flush_for_shutdown() to flush_and_report()
It'll be called in non-shutdown situations too.
2017-09-29 12:32:01 +02:00
9c8d6e01fb Properly fixed futures not being waited for. 2017-09-06 17:14:49 +02:00
3cd5b0ff6e Removed --factory-startup from the default Blender command
This can be passed to the Worker using the {blender} variable definition
instead.
2017-05-31 11:24:39 +02:00
337e77b64e Sign on at Manager when starting Worker
This passes the current hostname and list of supported task types to the
manager.
2017-03-31 17:24:46 +02:00
95b4f13741 Replaced job_types → task_types
This allows a finer granularity of task selection for a given worker.
2017-03-31 16:04:18 +02:00
5ac585241b Fixed unittest 2017-03-31 15:14:40 +02:00
b623143327 Explicitly clean up temporary directory after tests. 2017-03-16 15:32:54 +01:00
dce19396ef Implemented commands copy_file and remove_tree 2017-03-16 15:32:54 +01:00
162653383a Renamed test_runner_*.py to test_commands_*.py 2017-03-16 14:22:55 +01:00
f4ef19ae9b Moved command implementations from runner.py to commands.py 2017-03-16 12:46:39 +01:00
f5505b9a35 Round timestamped paths to entire seconds.
Round away the milliseconds, as those aren't all that interesting.
Uniqueness is ensured by calling _unique_path().
2017-03-15 18:13:58 +01:00
f652582579 Added move_to_final command to move an intermediate render dir to the final dir 2017-03-15 18:06:32 +01:00
c058754768 Move-out-of-way command now uses nicer filenames
The colons in the timestamp are gone, and numerical suffixes (to ensure
new directories are unique) are prefixed with a ~.
2017-03-15 18:06:10 +01:00
515d5b7d2d Fixed some test issues with CoroMock() 2017-03-15 17:53:47 +01:00
3d121eeef4 Removed CoroMock, when it fails it's really cryptic what's going on. 2017-03-15 17:25:08 +01:00
4d9819c92f Removed Flamenco Server and Manager, and moved Worker to top level
This is a clone of the Flamenco repository from back in the days when
Server, Manager and Worker shared the same Git repository. This is the
commit where that ended, and they went their separate ways.
2017-03-03 16:13:09 +01:00