Released 1.7.1

The biggest issue was the construction of an asyncio.Semaphore() while the
default loop is alive, and then creating a new loop on win32.

I've also taken the opportunity to explicitly pass our loop to some calls,
rather than expecting them to use the correct one automagically, and added
some more explicit timeout handling to the semaphore usage.

CHANGELOG.md

@@ -0,0 +1,52 @@
+# Blender Cloud changelog
+
+
+## Version 1.7.1 (2017-06-13)
+
+- Fixed asyncio issues on Windows
+
+
+## Version 1.7.0 (2017-06-09)
+
+- Fixed reloading after upgrading from 1.4.4 (our last public release).
+- Fixed bug handling a symlinked project path.
+- Added support for Manager-defined path replacement variables.
+
+
+## Version 1.6.4 (2017-04-21)
+
+- Added file exclusion filter for Flamenco. A filter like `*.abc;*.mkv;*.mov` can be
+  used to prevent certain files from being copied to the job storage directory.
+  Requires a Blender that is bundled with BAM 1.1.7 or newer.
+
+
+## Version 1.6.3 (2017-03-21)
+
+- Fixed bug where local project path wasn't shown for projects only set up for Flamenco
+  (and not Attract).
+- Added this CHANGELOG.md file, which will contain user-relevant changes.
+
+
+## Version 1.6.2 (2017-03-17)
+
+- Flamenco: when opening non-existing file path, open parent instead
+- Fix T50954: Improve Blender Cloud add-on project selector
+
+
+## Version 1.6.1 (2017-03-07)
+
+- Show error in GUI when Blender Cloud is unreachable
+- Fixed sample count when using branched path tracing
+
+
+## Version 1.6.0 (2017-02-14)
+
+- Default to frame chunk size of 1 (instead of 10).
+- Turn off "use overwrite" and "use placeholder" for Flamenco blend files.
+- Fixed bugs when blendfile is outside the project directory
+
+
+## Older versions
+
+For the history of older versions, please refer to the
+[Git history](https://developer.blender.org/diffusion/BCA/)

README-flamenco.md

@@ -0,0 +1,57 @@
+# Flamenco
+
+The Blender Cloud add-on has preliminary support for [Flamenco](https://flamenco.io/).
+It requires a project on the [Blender Cloud](https://cloud.blender.org/) that is set up for
+Flamenco, and it requires you to be logged in as a user with rights to use Flamenco.
+
+
+## Quirks
+
+Flamenco support is unpolished, so it has some quirks.
+
+- Project selection happens through the Attract project selector. As a result, you can only
+  select Attract-enabled projects (even when they are not set up for Flamenco). Be careful
+  which project you select.
+- The top level directory of the project is also set through the Attract properties.
+- Defaults are heavily biased for our use in the Blender Institute.
+- Settings that should be project-specific are not, i.e. are regular add-on preferences.
+- Sending a project to Flamenco will check the "File Extensions" setting in the Output panel,
+  and save the blend file to the current filename.
+
+## Render job file locations
+
+Rendering via Flamenco roughly comprises of two steps:
+
+1. Packing the file to render with its dependencies, and placing them in the "job file path".
+2. Rendering, and placing the output files in the "job output path".
+
+### Job file path
+
+The "job file path" consists of the following path components:
+
+1. The add-on preference "job file path", e.g. `/render/_flamenco/storage`
+2. The current date and time, your Blender Cloud username, and the name of the current blend file.
+3. The name of the current blend file.
+
+For example:
+
+`/render/_flamenco/storage/2017-01-18-104841.931387-sybren-03_02_A.layout/03_02_A.layout.blend`
+
+### Job output path
+
+The file path of output files consists of the following path components:
+
+1. The add-on preference "job file path", e.g. `/render/agent327/frames`
+2. The path of the current blend file, relative to the project directory. The first N components
+   of this path can be stripped; when N=1 it turns `scenes/03-searching/03_02_A-snooping/` into
+   `03-searching/03_02_A-snooping/`.
+3. The name of the current blend file, without `.blend`.
+4. The file name depends on the type of output:
+   - When rendering to image files: A 5-digit frame number with the required file extension.
+   - When rendering to a video file: The frame range with the required file extension.
+
+For example:
+
+`/render/agent327/frames/03-searching/03_02_A-snooping/03_02_A.layout/00441.exr`
+
+`/render/agent327/frames/03-searching/03_02_A-snooping/03_02_A.layout/14-51,60-133.mkv`

blender_cloud/__init__.py

@@ -21,7 +21,7 @@
 bl_info = {
     'name': 'Blender Cloud',
     "author": "Sybren A. Stüvel, Francesco Siddi, Inês Almeida, Antony Riakiotakis",
-    'version': (1, 5, 2),
+    'version': (1, 7, 1),
     'blender': (2, 77, 0),
     'location': 'Addon Preferences panel, and Ctrl+Shift+Alt+A anywhere for texture browser',
     'description': 'Texture library browser and Blender Sync. Requires the Blender ID addon '
@@ -65,32 +65,43 @@ def register():
 
         def reload_mod(name):
             modname = '%s.%s' % (__name__, name)
-            module = importlib.reload(sys.modules[modname])
-            sys.modules[modname] = module
-            return module
+            try:
+                old_module = sys.modules[modname]
+            except KeyError:
+                # Wasn't loaded before -- can happen after an upgrade.
+                new_module = importlib.import_module(modname)
+            else:
+                new_module = importlib.reload(old_module)
+
+            sys.modules[modname] = new_module
+            return new_module
 
         reload_mod('blendfile')
         reload_mod('home_project')
         reload_mod('utils')
 
-        blender = reload_mod('blender')
         async_loop = reload_mod('async_loop')
+        flamenco = reload_mod('flamenco')
+        attract = reload_mod('attract')
         texture_browser = reload_mod('texture_browser')
         settings_sync = reload_mod('settings_sync')
         image_sharing = reload_mod('image_sharing')
-        attract = reload_mod('attract')
+        blender = reload_mod('blender')
     else:
         from . import (blender, texture_browser, async_loop, settings_sync, blendfile, home_project,
-                       image_sharing, attract)
+                       image_sharing, attract, flamenco)
 
     async_loop.setup_asyncio_executor()
     async_loop.register()
 
+    flamenco.register()
+    attract.register()
     texture_browser.register()
-    blender.register()
     settings_sync.register()
     image_sharing.register()
-    attract.register()
+    blender.register()
+
+    blender.handle_project_update()
 
 
 def _monkey_patch_requests():
@@ -111,7 +122,8 @@ def _monkey_patch_requests():
 
 
 def unregister():
-    from . import blender, texture_browser, async_loop, settings_sync, image_sharing, attract
+    from . import (blender, texture_browser, async_loop, settings_sync, image_sharing, attract,
+                   flamenco)
 
     image_sharing.unregister()
     attract.unregister()
@@ -119,3 +131,4 @@ def unregister():
     blender.unregister()
     texture_browser.unregister()
     async_loop.unregister()
+    flamenco.unregister()

blender_cloud/async_loop.py

@@ -33,18 +33,28 @@ _loop_kicking_operator_running = False
 
 
 def setup_asyncio_executor():
-    """Sets up AsyncIO to run on a single thread.
+    """Sets up AsyncIO to run properly on each platform."""
 
-    This ensures that only one Pillar HTTP call is performed at the same time. Other
-    calls that could be performed in parallel are queued, and thus we can
-    reliably cancel them.
-    """
+    import sys
 
-    executor = concurrent.futures.ThreadPoolExecutor()
-    loop = asyncio.get_event_loop()
+    if sys.platform == 'win32':
+        asyncio.get_event_loop().close()
+        # On Windows, the default event loop is SelectorEventLoop, which does
+        # not support subprocesses. ProactorEventLoop should be used instead.
+        # Source: https://docs.python.org/3/library/asyncio-subprocess.html
+        loop = asyncio.ProactorEventLoop()
+        asyncio.set_event_loop(loop)
+    else:
+        loop = asyncio.get_event_loop()
+
+    executor = concurrent.futures.ThreadPoolExecutor(max_workers=10)
     loop.set_default_executor(executor)
     # loop.set_debug(True)
 
+    from . import pillar
+    # No more than this many Pillar calls should be made simultaneously
+    pillar.pillar_semaphore = asyncio.Semaphore(3, loop=loop)
+
 
 def kick_async_loop(*args) -> bool:
     """Performs a single iteration of the asyncio event loop.

blender_cloud/attract/__init__.py

@@ -60,6 +60,9 @@ from bpy.types import Operator, Panel, AddonPreferences
 
 log = logging.getLogger(__name__)
 
+# Global flag used to determine whether panels etc. can be drawn.
+attract_is_active = False
+
 
 def active_strip(context):
     try:
@@ -139,6 +142,9 @@ def shot_id_use(strips):
 def compute_strip_conflicts(scene):
     """Sets the strip property atc_object_id_conflict for each strip."""
 
+    if not attract_is_active:
+        return
+
     if not scene or not scene.sequence_editor or not scene.sequence_editor.sequences_all:
         return
 
@@ -161,7 +167,13 @@ def scene_update_post_handler(scene):
     compute_strip_conflicts(scene)
 
 
-class ToolsPanel(Panel):
+class AttractPollMixin:
+    @classmethod
+    def poll(cls, context):
+        return attract_is_active
+
+
+class ToolsPanel(AttractPollMixin, Panel):
     bl_label = 'Attract'
     bl_space_type = 'SEQUENCE_EDITOR'
     bl_region_type = 'UI'
@@ -238,7 +250,7 @@ class ToolsPanel(Panel):
             layout.operator(ATTRACT_OT_submit_all.bl_idname)
 
 
-class AttractOperatorMixin:
+class AttractOperatorMixin(AttractPollMixin):
     """Mix-in class for all Attract operators."""
 
     def _project_needs_setup_error(self):
@@ -261,7 +273,7 @@ class AttractOperatorMixin:
         from .. import pillar, blender
 
         prefs = blender.preferences()
-        project = self.find_project(prefs.attract_project.project)
+        project = self.find_project(prefs.project.project)
 
         # FIXME: Eve doesn't seem to handle the $elemMatch projection properly,
         # even though it works fine in MongoDB itself. As a result, we have to
@@ -295,7 +307,7 @@ class AttractOperatorMixin:
                                'cut_in_timeline_in_frames': strip.frame_final_start},
                 'order': 0,
                 'node_type': 'attract_shot',
-                'project': blender.preferences().attract_project.project,
+                'project': blender.preferences().project.project,
                 'user': user_uuid}
 
         # Create a Node item with the attract API
@@ -373,7 +385,7 @@ class AttractShotFetchUpdate(AttractOperatorMixin, Operator):
 
     @classmethod
     def poll(cls, context):
-        return any(selected_shots(context))
+        return AttractOperatorMixin.poll(context) and any(selected_shots(context))
 
     def execute(self, context):
         for strip in selected_shots(context):
@@ -393,6 +405,9 @@ class AttractShotRelink(AttractShotFetchUpdate):
 
     @classmethod
     def poll(cls, context):
+        if not AttractOperatorMixin.poll(context):
+            return False
+
         strip = active_strip(context)
         return strip is not None and not getattr(strip, 'atc_object_id', None)
 
@@ -433,7 +448,8 @@ class ATTRACT_OT_shot_open_in_browser(AttractOperatorMixin, Operator):
 
     @classmethod
     def poll(cls, context):
-        return bool(context.selected_sequences and active_strip(context))
+        return AttractOperatorMixin.poll(context) and \
+               bool(context.selected_sequences and active_strip(context))
 
     def execute(self, context):
         from ..blender import PILLAR_WEB_SERVER_URL
@@ -459,7 +475,8 @@ class AttractShotDelete(AttractOperatorMixin, Operator):
 
     @classmethod
     def poll(cls, context):
-        return bool(context.selected_sequences)
+        return AttractOperatorMixin.poll(context) and \
+               bool(context.selected_sequences)
 
     def execute(self, context):
         from .. import pillar
@@ -511,7 +528,8 @@ class AttractStripUnlink(AttractOperatorMixin, Operator):
 
     @classmethod
     def poll(cls, context):
-        return bool(context.selected_sequences)
+        return AttractOperatorMixin.poll(context) and \
+               bool(context.selected_sequences)
 
     def execute(self, context):
         unlinked_ids = set()
@@ -554,7 +572,8 @@ class AttractShotSubmitSelected(AttractOperatorMixin, Operator):
 
     @classmethod
     def poll(cls, context):
-        return bool(context.selected_sequences)
+        return AttractOperatorMixin.poll(context) and \
+               bool(context.selected_sequences)
 
     def execute(self, context):
         # Check that the project is set up for Attract.
@@ -610,7 +629,8 @@ class ATTRACT_OT_open_meta_blendfile(AttractOperatorMixin, Operator):
 
     @classmethod
     def poll(cls, context):
-        return bool(any(cls.filename_from_metadata(s) for s in context.selected_sequences))
+        return AttractOperatorMixin.poll(context) and \
+               bool(any(cls.filename_from_metadata(s) for s in context.selected_sequences))
 
     @staticmethod
     def filename_from_metadata(strip):
@@ -677,7 +697,7 @@ class ATTRACT_OT_make_shot_thumbnail(AttractOperatorMixin,
 
     @classmethod
     def poll(cls, context):
-        return bool(context.selected_sequences)
+        return AttractOperatorMixin.poll(context) and bool(context.selected_sequences)
 
     @contextlib.contextmanager
     def thumbnail_render_settings(self, context, thumbnail_width=512):
@@ -843,7 +863,7 @@ class ATTRACT_OT_make_shot_thumbnail(AttractOperatorMixin,
         from .. import blender
 
         prefs = blender.preferences()
-        project = self.find_project(prefs.attract_project.project)
+        project = self.find_project(prefs.project.project)
 
         self.log.info('Uploading file %s', filename)
         resp = await pillar.pillar_call(
@@ -873,7 +893,8 @@ class ATTRACT_OT_copy_id_to_clipboard(AttractOperatorMixin, Operator):
 
     @classmethod
     def poll(cls, context):
-        return bool(context.selected_sequences and active_strip(context))
+        return AttractOperatorMixin.poll(context) and \
+               bool(context.selected_sequences and active_strip(context))
 
     def execute(self, context):
         strip = active_strip(context)
@@ -905,6 +926,29 @@ def draw_strip_movie_meta(self, context):
     box.label('Original Frame Range: %s-%s' % (sfra, efra))
 
 
+def activate():
+    global attract_is_active
+
+    log.info('Activating Attract')
+    attract_is_active = True
+    bpy.app.handlers.scene_update_post.append(scene_update_post_handler)
+    draw.callback_enable()
+
+
+def deactivate():
+    global attract_is_active
+
+    log.info('Deactivating Attract')
+    attract_is_active = False
+    draw.callback_disable()
+
+    try:
+        bpy.app.handlers.scene_update_post.remove(scene_update_post_handler)
+    except ValueError:
+        # This is thrown when scene_update_post_handler does not exist in the handler list.
+        pass
+
+
 def register():
     bpy.types.Sequence.atc_is_synced = bpy.props.BoolProperty(name="Is Synced")
     bpy.types.Sequence.atc_object_id = bpy.props.StringProperty(name="Attract Object ID")
@@ -942,13 +986,9 @@ def register():
     bpy.utils.register_class(ATTRACT_OT_make_shot_thumbnail)
     bpy.utils.register_class(ATTRACT_OT_copy_id_to_clipboard)
 
-    bpy.app.handlers.scene_update_post.append(scene_update_post_handler)
-    draw.callback_enable()
-
 
 def unregister():
-    draw.callback_disable()
-    bpy.app.handlers.scene_update_post.remove(scene_update_post_handler)
+    deactivate()
     bpy.utils.unregister_module(__name__)
     del bpy.types.Sequence.atc_is_synced
     del bpy.types.Sequence.atc_object_id

blender_cloud/attract/draw.py

@@ -172,6 +172,11 @@ def callback_disable():
     if not cb_handle:
         return
 
-    bpy.types.SpaceSequenceEditor.draw_handler_remove(cb_handle[0], 'WINDOW')
+    try:
+        bpy.types.SpaceSequenceEditor.draw_handler_remove(cb_handle[0], 'WINDOW')
+    except ValueError:
+        # Thrown when already removed.
+        pass
+    cb_handle.clear()
 
     tag_redraw_all_sequencer_editors()

blender_cloud/blender.py

@@ -20,16 +20,17 @@
 
 Separated from __init__.py so that we can import & run from non-Blender environments.
 """
-
+import functools
 import logging
 import os.path
 
 import bpy
 from bpy.types import AddonPreferences, Operator, WindowManager, Scene, PropertyGroup
-from bpy.props import StringProperty, EnumProperty, PointerProperty, BoolProperty
+from bpy.props import StringProperty, EnumProperty, PointerProperty, BoolProperty, IntProperty
 import rna_prop_ui
 
-from . import pillar, async_loop
+from . import pillar, async_loop, flamenco
+from .utils import pyside_cache, redraw
 
 PILLAR_WEB_SERVER_URL = 'https://cloud.blender.org/'
 # PILLAR_WEB_SERVER_URL = 'http://pillar-web:5001/'
@@ -41,39 +42,6 @@ log = logging.getLogger(__name__)
 icons = None
 
 
-def redraw(self, context):
-    context.area.tag_redraw()
-
-
-def pyside_cache(propname):
-
-    if callable(propname):
-        raise TypeError('Usage: pyside_cache("property_name")')
-
-    def decorator(wrapped):
-        """Stores the result of the callable in Python-managed memory.
-
-        This is to work around the warning at
-        https://www.blender.org/api/blender_python_api_master/bpy.props.html#bpy.props.EnumProperty
-        """
-
-        import functools
-
-        @functools.wraps(wrapped)
-        # We can't use (*args, **kwargs), because EnumProperty explicitly checks
-        # for the number of fixed positional arguments.
-        def wrapper(self, context):
-            result = None
-            try:
-                result = wrapped(self, context)
-                return result
-            finally:
-                rna_type, rna_info = getattr(self.bl_rna, propname)
-                rna_info['_cached_result'] = result
-        return wrapper
-    return decorator
-
-
 @pyside_cache('version')
 def blender_syncable_versions(self, context):
     """Returns the list of items used by SyncStatusProperties.version EnumProperty."""
@@ -141,13 +109,61 @@ class SyncStatusProperties(PropertyGroup):
 def bcloud_available_projects(self, context):
     """Returns the list of items used by BlenderCloudProjectGroup.project EnumProperty."""
 
-    attr_proj = preferences().attract_project
-    projs = attr_proj.available_projects
+    projs = preferences().project.available_projects
     if not projs:
         return [('', 'No projects available in your Blender Cloud', '')]
     return [(p['_id'], p['name'], '') for p in projs]
 
 
+@functools.lru_cache(1)
+def project_extensions(project_id) -> set:
+    """Returns the extensions the project is enabled for.
+
+    At the moment of writing these are 'attract' and 'flamenco'.
+    """
+
+    log.debug('Finding extensions for project %s', project_id)
+
+    # We can't use our @property, since the preferences may be loaded from a
+    # preferences blend file, in which case it is not constructed from Python code.
+    available_projects = preferences().project.get('available_projects', [])
+    if not available_projects:
+        log.debug('No projects available.')
+        return set()
+
+    proj = next((p for p in available_projects
+                 if p['_id'] == project_id), None)
+    if proj is None:
+        log.debug('Project %s not found in available projects.', project_id)
+        return set()
+
+    return set(proj.get('enabled_for', ()))
+
+
+def handle_project_update(_=None, _2=None):
+    """Handles changing projects, which may cause extensions to be disabled/enabled.
+
+    Ignores arguments so that it can be used as property update callback.
+    """
+
+    project_id = preferences().project.project
+    log.info('Updating internal state to reflect extensions enabled on current project %s.',
+             project_id)
+
+    project_extensions.cache_clear()
+
+    from blender_cloud import attract, flamenco
+    attract.deactivate()
+    flamenco.deactivate()
+
+    enabled_for = project_extensions(project_id)
+    log.info('Project extensions: %s', enabled_for)
+    if 'attract' in enabled_for:
+        attract.activate()
+    if 'flamenco' in enabled_for:
+        flamenco.activate()
+
+
 class BlenderCloudProjectGroup(PropertyGroup):
     status = EnumProperty(
         items=[
@@ -161,7 +177,9 @@ class BlenderCloudProjectGroup(PropertyGroup):
     project = EnumProperty(
         items=bcloud_available_projects,
         name='Cloud project',
-        description='Which Blender Cloud project to work with')
+        description='Which Blender Cloud project to work with',
+        update=handle_project_update
+    )
 
     # List of projects is stored in 'available_projects' ID property,
     # because I don't know how to store a variable list of strings in a proper RNA property.
@@ -172,6 +190,7 @@ class BlenderCloudProjectGroup(PropertyGroup):
     @available_projects.setter
     def available_projects(self, new_projects):
         self['available_projects'] = new_projects
+        handle_project_update()
 
 
 class BlenderCloudPreferences(AddonPreferences):
@@ -187,26 +206,63 @@ class BlenderCloudPreferences(AddonPreferences):
     )
 
     local_texture_dir = StringProperty(
-        name='Default Blender Cloud texture storage directory',
+        name='Default Blender Cloud Texture Storage Directory',
         subtype='DIR_PATH',
         default='//textures')
 
     open_browser_after_share = BoolProperty(
-        name='Open browser after sharing file',
+        name='Open Browser after Sharing File',
         description='When enabled, Blender will open a webbrowser',
         default=True
     )
 
-    # TODO: store local path with the Attract project, so that people
-    # can switch projects and the local path switches with it.
-    attract_project = PointerProperty(type=BlenderCloudProjectGroup)
-    attract_project_local_path = StringProperty(
-        name='Local project path',
+    # TODO: store project-dependent properties with the project, so that people
+    # can switch projects and the Attract and Flamenco properties switch with it.
+    project = PointerProperty(type=BlenderCloudProjectGroup)
+
+    cloud_project_local_path = StringProperty(
+        name='Local Project Path',
         description='Local path of your Attract project, used to search for blend files; '
                     'usually best to set to an absolute path',
         subtype='DIR_PATH',
         default='//../')
 
+    flamenco_manager = PointerProperty(type=flamenco.FlamencoManagerGroup)
+    flamenco_exclude_filter = StringProperty(
+        name='File Exclude Filter',
+        description='Filter like "*.abc;*.mkv" to prevent certain files to be packed '
+                    'into the output directory',
+        default='')
+    # TODO: before making Flamenco public, change the defaults to something less Institute-specific.
+    # NOTE: The assumption is that the workers can also find the files in the same path.
+    #       This assumption is true for the Blender Institute.
+    flamenco_job_file_path = StringProperty(
+        name='Job File Path',
+        description='Path where to store job files, should be accesible for Workers too',
+        subtype='DIR_PATH',
+        default='/render/_flamenco/storage')
+
+    # TODO: before making Flamenco public, change the defaults to something less Institute-specific.
+    flamenco_job_output_path = StringProperty(
+        name='Job Output Path',
+        description='Path where to store output files, should be accessible for Workers',
+        subtype='DIR_PATH',
+        default='/render/_flamenco/output')
+    flamenco_job_output_strip_components = IntProperty(
+        name='Job Output Path Strip Components',
+        description='The final output path comprises of the job output path, and the blend file '
+                    'path relative to the project with this many path components stripped off '
+                    'the front',
+        min=0,
+        default=0,
+        soft_max=4,
+    )
+    flamenco_open_browser_after_submit = BoolProperty(
+        name='Open Browser after Submitting Job',
+        description='When enabled, Blender will open a webbrowser',
+        default=True
+    )
+
     def draw(self, context):
         import textwrap
 
@@ -220,8 +276,8 @@ class BlenderCloudPreferences(AddonPreferences):
             blender_id_profile = None
         else:
             blender_id_profile = blender_id.get_active_profile()
-
         if blender_id is None:
+
             msg_icon = 'ERROR'
             text = 'This add-on requires Blender ID'
             help_text = 'Make sure that the Blender ID add-on is installed and activated'
@@ -287,9 +343,17 @@ class BlenderCloudPreferences(AddonPreferences):
         share_box.label('Image Sharing on Blender Cloud', icon_value=icon('CLOUD'))
         share_box.prop(self, 'open_browser_after_share')
 
-        # Attract stuff
-        attract_box = layout.box()
-        self.draw_attract_buttons(attract_box, self.attract_project)
+        # Project selector
+        project_box = layout.box()
+        project_box.enabled = self.project.status in {'NONE', 'IDLE'}
+
+        self.draw_project_selector(project_box, self.project)
+        extensions = project_extensions(self.project.project)
+
+        # Flamenco stuff
+        if 'flamenco' in extensions:
+            flamenco_box = project_box.column()
+            self.draw_flamenco_buttons(flamenco_box, self.flamenco_manager, context)
 
     def draw_subscribe_button(self, layout):
         layout.operator('pillar.subscribe', icon='WORLD')
@@ -325,12 +389,11 @@ class BlenderCloudPreferences(AddonPreferences):
         else:
             row_pull.label('Cloud Sync is running.')
 
-    def draw_attract_buttons(self, attract_box, bcp: BlenderCloudProjectGroup):
-        attract_row = attract_box.row(align=True)
-        attract_row.label('Attract', icon_value=icon('CLOUD'))
+    def draw_project_selector(self, project_box, bcp: BlenderCloudProjectGroup):
+        project_row = project_box.row(align=True)
+        project_row.label('Project settings', icon_value=icon('CLOUD'))
 
-        attract_row.enabled = bcp.status in {'NONE', 'IDLE'}
-        row_buttons = attract_row.row(align=True)
+        row_buttons = project_row.row(align=True)
 
         projects = bcp.available_projects
         project = bcp.project
@@ -347,7 +410,86 @@ class BlenderCloudPreferences(AddonPreferences):
         else:
             row_buttons.label('Fetching available projects.')
 
-        attract_box.prop(self, 'attract_project_local_path')
+        enabled_for = project_extensions(project)
+        if not project:
+            return
+
+        if not enabled_for:
+            project_box.label('This project is not set up for Attract or Flamenco')
+            return
+
+        project_box.label('This project is set up for: %s' %
+                          ', '.join(sorted(enabled_for)))
+
+        # This is only needed when the project is set up for either Attract or Flamenco.
+        project_box.prop(self, 'cloud_project_local_path',
+                         text='Local Cloud Project Path')
+
+    def draw_flamenco_buttons(self, flamenco_box, bcp: flamenco.FlamencoManagerGroup, context):
+        from .flamenco import bam_interface
+
+        header_row = flamenco_box.row(align=True)
+        header_row.label('Flamenco:', icon_value=icon('CLOUD'))
+
+        manager_split = flamenco_box.split(0.32, align=True)
+        manager_split.label('Manager:')
+        manager_box = manager_split.row(align=True)
+
+        if bcp.status in {'NONE', 'IDLE'}:
+            if not bcp.available_managers or not bcp.manager:
+                manager_box.operator('flamenco.managers',
+                                     text='Find Flamenco Managers',
+                                     icon='FILE_REFRESH')
+            else:
+                manager_box.prop(bcp, 'manager', text='')
+                manager_box.operator('flamenco.managers',
+                                     text='',
+                                     icon='FILE_REFRESH')
+        else:
+            manager_box.label('Fetching available managers.')
+
+        path_split = flamenco_box.split(0.32, align=True)
+        path_split.label(text='Job File Path:')
+        path_box = path_split.row(align=True)
+        path_box.prop(self, 'flamenco_job_file_path', text='')
+        props = path_box.operator('flamenco.explore_file_path', text='', icon='DISK_DRIVE')
+        props.path = self.flamenco_job_file_path
+
+        job_output_box = flamenco_box.column(align=True)
+        path_split = job_output_box.split(0.32, align=True)
+        path_split.label(text='Job Output Path:')
+        path_box = path_split.row(align=True)
+        path_box.prop(self, 'flamenco_job_output_path', text='')
+        props = path_box.operator('flamenco.explore_file_path', text='', icon='DISK_DRIVE')
+        props.path = self.flamenco_job_output_path
+
+        show_warning = bool(self.flamenco_exclude_filter and
+                            not bam_interface.bam_supports_exclude_option())
+        job_output_box.alert = show_warning
+        job_output_box.prop(self, 'flamenco_exclude_filter',
+                            icon='ERROR' if show_warning else 'NONE')
+        if show_warning:
+            job_output_box.label(
+                text='Warning, the exclusion filter requires a newer version of Blender!')
+        job_output_box.alert = False
+
+        prop_split = job_output_box.split(0.32, align=True)
+        prop_split.label('Strip Components:')
+        prop_split.prop(self, 'flamenco_job_output_strip_components', text='')
+
+        from .flamenco import render_output_path
+
+        path_box = job_output_box.row(align=True)
+        output_path = render_output_path(context)
+        if output_path:
+            path_box.label(str(output_path))
+            props = path_box.operator('flamenco.explore_file_path', text='', icon='DISK_DRIVE')
+            props.path = str(output_path.parent)
+        else:
+            path_box.label('Blend file is not in your project path, '
+                           'unable to give output path example.')
+
+        flamenco_box.prop(self, 'flamenco_open_browser_after_submit')
 
 
 class PillarCredentialsUpdate(pillar.PillarOperatorMixin,
@@ -414,7 +556,7 @@ class PILLAR_OT_subscribe(Operator):
 
 
 class PILLAR_OT_projects(async_loop.AsyncModalOperatorMixin,
-                         pillar.PillarOperatorMixin,
+                         pillar.AuthenticatedPillarOperatorMixin,
                          Operator):
     """Fetches the projects available to the user"""
     bl_idname = 'pillar.projects'
@@ -424,53 +566,61 @@ class PILLAR_OT_projects(async_loop.AsyncModalOperatorMixin,
     _log = logging.getLogger('bpy.ops.%s' % bl_idname)
 
     async def async_execute(self, context):
+        if not await self.authenticate(context):
+            return
+
         import pillarsdk
         from .pillar import pillar_call
 
-        self._log.info('Checking credentials')
-        try:
-            db_user = await self.check_credentials(context, ())
-        except pillar.UserNotLoggedInError as ex:
-            self._log.info('Not logged in error raised: %s', ex)
-            self.report({'ERROR'}, 'Please log in on Blender ID first.')
-            self.quit()
-            return
+        self.log.info('Going to fetch projects for user %s', self.user_id)
 
-        user_id = db_user['_id']
-        self.log.info('Going to fetch projects for user %s', user_id)
-
-        preferences().attract_project.status = 'FETCHING'
+        preferences().project.status = 'FETCHING'
 
         # Get all projects, except the home project.
         projects_user = await pillar_call(
             pillarsdk.Project.all,
-            {'where': {'user': user_id,
+            {'where': {'user': self.user_id,
                        'category': {'$ne': 'home'}},
              'sort': '-_created',
              'projection': {'_id': True,
-                            'name': True},
+                            'name': True,
+                            'extension_props': True},
              })
 
         projects_shared = await pillar_call(
             pillarsdk.Project.all,
-            {'where': {'user': {'$ne': user_id},
-                       'permissions.groups.group': {'$in': db_user.groups}},
+            {'where': {'user': {'$ne': self.user_id},
+                       'permissions.groups.group': {'$in': self.db_user.groups}},
              'sort': '-_created',
              'projection': {'_id': True,
-                            'name': True},
+                            'name': True,
+                            'extension_props': True},
              })
 
         # We need to convert to regular dicts before storing in ID properties.
         # Also don't store more properties than we need.
-        projects = [{'_id': p['_id'], 'name': p['name']} for p in projects_user['_items']] + \
-                   [{'_id': p['_id'], 'name': p['name']} for p in projects_shared['_items']]
+        def reduce_properties(project_list):
+            for p in project_list:
+                p = p.to_dict()
+                extension_props = p.get('extension_props', {})
+                enabled_for = list(extension_props.keys())
 
-        preferences().attract_project.available_projects = projects
+                self._log.debug('Project %r is enabled for %s', p['name'], enabled_for)
+                yield {
+                    '_id': p['_id'],
+                    'name': p['name'],
+                    'enabled_for': enabled_for,
+                }
+
+        projects = list(reduce_properties(projects_user['_items'])) + \
+                   list(reduce_properties(projects_shared['_items']))
+
+        preferences().project.available_projects = projects
 
         self.quit()
 
     def quit(self):
-        preferences().attract_project.status = 'IDLE'
+        preferences().project.status = 'IDLE'
         super().quit()
 
 

blender_cloud/flamenco/__init__.py

@@ -0,0 +1,716 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
+"""Flamenco interface.
+
+The preferences are managed blender.py, the rest of the Flamenco-specific stuff is here.
+"""
+import functools
+import logging
+from pathlib import Path, PurePath
+import typing
+
+import bpy
+from bpy.types import AddonPreferences, Operator, WindowManager, Scene, PropertyGroup
+from bpy.props import StringProperty, EnumProperty, PointerProperty, BoolProperty, IntProperty
+
+from .. import async_loop, pillar
+from ..utils import pyside_cache, redraw
+
+log = logging.getLogger(__name__)
+
+# Global flag used to determine whether panels etc. can be drawn.
+flamenco_is_active = False
+
+
+@pyside_cache('manager')
+def available_managers(self, context):
+    """Returns the list of items used by a manager-selector EnumProperty."""
+
+    from ..blender import preferences
+
+    mngrs = preferences().flamenco_manager.available_managers
+    if not mngrs:
+        return [('', 'No managers available in your Blender Cloud', '')]
+    return [(p['_id'], p['name'], '') for p in mngrs]
+
+
+class FlamencoManagerGroup(PropertyGroup):
+    manager = EnumProperty(
+        items=available_managers,
+        name='Flamenco Manager',
+        description='Which Flamenco Manager to use for jobs')
+
+    status = EnumProperty(
+        items=[
+            ('NONE', 'NONE', 'We have done nothing at all yet'),
+            ('IDLE', 'IDLE', 'User requested something, which is done, and we are now idle'),
+            ('FETCHING', 'FETCHING', 'Fetching available Flamenco managers from Blender Cloud'),
+        ],
+        name='status',
+        update=redraw)
+
+    # List of managers is stored in 'available_managers' ID property,
+    # because I don't know how to store a variable list of strings in a proper RNA property.
+    @property
+    def available_managers(self) -> list:
+        return self.get('available_managers', [])
+
+    @available_managers.setter
+    def available_managers(self, new_managers):
+        self['available_managers'] = new_managers
+
+
+class FlamencoPollMixin:
+    @classmethod
+    def poll(cls, context):
+        return flamenco_is_active
+
+
+class FLAMENCO_OT_fmanagers(async_loop.AsyncModalOperatorMixin,
+                            pillar.AuthenticatedPillarOperatorMixin,
+                            FlamencoPollMixin,
+                            Operator):
+    """Fetches the Flamenco Managers available to the user"""
+    bl_idname = 'flamenco.managers'
+    bl_label = 'Fetch available Flamenco Managers'
+
+    stop_upon_exception = True
+    log = logging.getLogger('%s.FLAMENCO_OT_fmanagers' % __name__)
+
+    @property
+    def mypref(self) -> FlamencoManagerGroup:
+        from ..blender import preferences
+
+        return preferences().flamenco_manager
+
+    async def async_execute(self, context):
+        if not await self.authenticate(context):
+            return
+
+        from .sdk import Manager
+        from ..pillar import pillar_call
+
+        self.log.info('Going to fetch managers for user %s', self.user_id)
+
+        self.mypref.status = 'FETCHING'
+        managers = await pillar_call(Manager.all)
+
+        # We need to convert to regular dicts before storing in ID properties.
+        # Also don't store more properties than we need.
+        as_list = [{'_id': p['_id'], 'name': p['name']} for p in managers['_items']]
+
+        self.mypref.available_managers = as_list
+        self.quit()
+
+    def quit(self):
+        self.mypref.status = 'IDLE'
+        super().quit()
+
+
+class FLAMENCO_OT_render(async_loop.AsyncModalOperatorMixin,
+                         pillar.AuthenticatedPillarOperatorMixin,
+                         FlamencoPollMixin,
+                         Operator):
+    """Performs a Blender render on Flamenco."""
+    bl_idname = 'flamenco.render'
+    bl_label = 'Render on Flamenco'
+    bl_description = __doc__.rstrip('.')
+
+    stop_upon_exception = True
+    log = logging.getLogger('%s.FLAMENCO_OT_render' % __name__)
+
+    async def async_execute(self, context):
+        if not await self.authenticate(context):
+            return
+
+        import pillarsdk.exceptions
+        from .sdk import Manager
+        from ..pillar import pillar_call
+        from ..blender import preferences
+
+        scene = context.scene
+
+        # Save to a different file, specifically for Flamenco.
+        context.window_manager.flamenco_status = 'PACKING'
+        filepath = await self._save_blendfile(context)
+
+        # Determine where the render output will be stored.
+        render_output = render_output_path(context, filepath)
+        if render_output is None:
+            self.report({'ERROR'}, 'Current file is outside of project path.')
+            self.quit()
+            return
+        self.log.info('Will output render files to %s', render_output)
+
+        # BAM-pack the files to the destination directory.
+        outfile, missing_sources = await self.bam_pack(filepath)
+        if not outfile:
+            return
+
+        # Fetch Manager for doing path replacement.
+        self.log.info('Going to fetch manager %s', self.user_id)
+        prefs = preferences()
+
+        manager_id = prefs.flamenco_manager.manager
+        try:
+            manager = await pillar_call(Manager.find, manager_id)
+        except pillarsdk.exceptions.ResourceNotFound:
+            self.report({'ERROR'}, 'Manager %s not found, refresh your managers in '
+                                   'the Blender Cloud add-on settings.' % manager_id)
+            self.quit()
+            return
+
+        # Create the job at Flamenco Server.
+        context.window_manager.flamenco_status = 'COMMUNICATING'
+        settings = {'blender_cmd': '{blender}',
+                    'chunk_size': scene.flamenco_render_fchunk_size,
+                    'filepath': manager.replace_path(outfile),
+                    'frames': scene.flamenco_render_frame_range,
+                    'render_output': manager.replace_path(render_output),
+                    }
+
+        # Add extra settings specific to the job type
+        if scene.flamenco_render_job_type == 'blender-render-progressive':
+            if scene.cycles.progressive == 'BRANCHED_PATH':
+                samples = scene.cycles.aa_samples
+            else:
+                samples = scene.cycles.samples
+
+            if scene.cycles.use_square_samples:
+                samples **= 2
+
+            settings['cycles_num_chunks'] = scene.flamenco_render_schunk_count
+            settings['cycles_sample_count'] = samples
+            settings['format'] = 'EXR'
+
+        try:
+            job_info = await create_job(self.user_id,
+                                        prefs.project.project,
+                                        manager_id,
+                                        scene.flamenco_render_job_type,
+                                        settings,
+                                        'Render %s' % filepath.name,
+                                        priority=scene.flamenco_render_job_priority)
+        except Exception as ex:
+            self.report({'ERROR'}, 'Error creating Flamenco job: %s' % ex)
+            self.quit()
+            return
+
+        # Store the job ID in a file in the output dir.
+        with open(str(outfile.parent / 'jobinfo.json'), 'w', encoding='utf8') as outfile:
+            import json
+
+            job_info['missing_files'] = [str(mf) for mf in missing_sources]
+            json.dump(job_info, outfile, sort_keys=True, indent=4)
+
+        # We can now remove the local copy we made with bpy.ops.wm.save_as_mainfile().
+        # Strictly speaking we can already remove it after the BAM-pack, but it may come in
+        # handy in case of failures.
+        try:
+            self.log.info('Removing temporary file %s', filepath)
+            filepath.unlink()
+        except Exception as ex:
+            self.report({'ERROR'}, 'Unable to remove file: %s' % ex)
+            self.quit()
+            return
+
+        if prefs.flamenco_open_browser_after_submit:
+            import webbrowser
+            from urllib.parse import urljoin
+            from ..blender import PILLAR_WEB_SERVER_URL
+
+            url = urljoin(PILLAR_WEB_SERVER_URL, '/flamenco/jobs/%s/redir' % job_info['_id'])
+            webbrowser.open_new_tab(url)
+
+        # Do a final report.
+        if missing_sources:
+            names = (ms.name for ms in missing_sources)
+            self.report({'WARNING'}, 'Flamenco job created with missing files: %s' %
+                        '; '.join(names))
+        else:
+            self.report({'INFO'}, 'Flamenco job created.')
+
+        self.quit()
+
+    async def _save_blendfile(self, context):
+        """Save to a different file, specifically for Flamenco.
+
+        We shouldn't overwrite the artist's file.
+        We can compress, since this file won't be managed by SVN and doesn't need diffability.
+        """
+
+        render = context.scene.render
+
+        # Remember settings we need to restore after saving.
+        old_use_file_extension = render.use_file_extension
+        old_use_overwrite = render.use_overwrite
+        old_use_placeholder = render.use_placeholder
+
+        try:
+
+            # The file extension should be determined by the render settings, not necessarily
+            # by the setttings in the output panel.
+            render.use_file_extension = True
+
+            # Rescheduling should not overwrite existing frames.
+            render.use_overwrite = False
+            render.use_placeholder = False
+
+            filepath = Path(context.blend_data.filepath).with_suffix('.flamenco.blend')
+            self.log.info('Saving copy to temporary file %s', filepath)
+            bpy.ops.wm.save_as_mainfile(filepath=str(filepath),
+                                        compress=True,
+                                        copy=True)
+        finally:
+            # Restore the settings we changed, even after an exception.
+            render.use_file_extension = old_use_file_extension
+            render.use_overwrite = old_use_overwrite
+            render.use_placeholder = old_use_placeholder
+
+        return filepath
+
+    def quit(self):
+        super().quit()
+        bpy.context.window_manager.flamenco_status = 'IDLE'
+
+    async def bam_pack(self, filepath: Path) -> (typing.Optional[Path], typing.List[Path]):
+        """BAM-packs the blendfile to the destination directory.
+
+        Returns the path of the destination blend file.
+
+        :param filepath: the blend file to pack (i.e. the current blend file)
+        :returns: the destination blend file, or None if there were errors BAM-packing,
+            and a list of missing paths.
+        """
+
+        from datetime import datetime
+        from ..blender import preferences
+        from . import bam_interface
+
+        prefs = preferences()
+
+        # Create a unique directory that is still more or less identifyable.
+        # This should work better than a random ID.
+        # BAM doesn't like output directories that end in '.blend'.
+        unique_dir = '%s-%s-%s' % (datetime.now().isoformat('-').replace(':', ''),
+                                   self.db_user['username'],
+                                   filepath.stem)
+        outdir = Path(prefs.flamenco_job_file_path) / unique_dir
+        outfile = outdir / filepath.name
+
+        exclusion_filter = prefs.flamenco_exclude_filter or None
+
+        try:
+            outdir.mkdir(parents=True)
+        except Exception as ex:
+            self.log.exception('Unable to create output path %s', outdir)
+            self.report({'ERROR'}, 'Unable to create output path: %s' % ex)
+            self.quit()
+            return None, []
+
+        try:
+            missing_sources = await bam_interface.bam_copy(filepath, outfile, exclusion_filter)
+        except bam_interface.CommandExecutionError as ex:
+            self.log.exception('Unable to execute BAM pack')
+            self.report({'ERROR'}, 'Unable to execute BAM pack: %s' % ex)
+            self.quit()
+            return None, []
+
+        return outfile, missing_sources
+
+
+class FLAMENCO_OT_scene_to_frame_range(FlamencoPollMixin, Operator):
+    """Sets the scene frame range as the Flamenco render frame range."""
+    bl_idname = 'flamenco.scene_to_frame_range'
+    bl_label = 'Sets the scene frame range as the Flamenco render frame range'
+    bl_description = __doc__.rstrip('.')
+
+    def execute(self, context):
+        s = context.scene
+        s.flamenco_render_frame_range = '%i-%i' % (s.frame_start, s.frame_end)
+        return {'FINISHED'}
+
+
+class FLAMENCO_OT_copy_files(Operator,
+                             FlamencoPollMixin,
+                             async_loop.AsyncModalOperatorMixin):
+    """Uses BAM to copy the current blendfile + dependencies to the target directory."""
+    bl_idname = 'flamenco.copy_files'
+    bl_label = 'Copy files to target'
+    bl_description = __doc__.rstrip('.')
+
+    stop_upon_exception = True
+
+    async def async_execute(self, context):
+        from pathlib import Path
+        from . import bam_interface
+        from ..blender import preferences
+
+        context.window_manager.flamenco_status = 'PACKING'
+
+        missing_sources = await bam_interface.bam_copy(
+            Path(context.blend_data.filepath),
+            Path(preferences().flamenco_job_file_path),
+        )
+
+        if missing_sources:
+            names = (ms.name for ms in missing_sources)
+            self.report({'ERROR'}, 'Missing source files: %s' % '; '.join(names))
+
+        self.quit()
+
+    def quit(self):
+        super().quit()
+        bpy.context.window_manager.flamenco_status = 'IDLE'
+
+
+class FLAMENCO_OT_explore_file_path(FlamencoPollMixin,
+                                    Operator):
+    """Opens the Flamenco job storage path in a file explorer.
+
+    If the path cannot be found, this operator tries to open its parent.
+    """
+
+    bl_idname = 'flamenco.explore_file_path'
+    bl_label = 'Open in file explorer'
+    bl_description = __doc__.rstrip('.')
+
+    path = StringProperty(name='Path', description='Path to explore', subtype='DIR_PATH')
+
+    def execute(self, context):
+        import platform
+        import pathlib
+
+        # Possibly open a parent of the path
+        to_open = pathlib.Path(self.path)
+        while to_open.parent != to_open:  # while we're not at the root
+            if to_open.exists():
+                break
+            to_open = to_open.parent
+        else:
+            self.report({'ERROR'}, 'Unable to open %s or any of its parents.' % self.path)
+            return {'CANCELLED'}
+        to_open = str(to_open)
+
+        if platform.system() == "Windows":
+            import os
+            os.startfile(to_open)
+
+        elif platform.system() == "Darwin":
+            import subprocess
+            subprocess.Popen(["open", to_open])
+
+        else:
+            import subprocess
+            subprocess.Popen(["xdg-open", to_open])
+
+        return {'FINISHED'}
+
+
+async def create_job(user_id: str,
+                     project_id: str,
+                     manager_id: str,
+                     job_type: str,
+                     job_settings: dict,
+                     job_name: str = None,
+                     *,
+                     priority: int = 50,
+                     job_description: str = None) -> dict:
+    """Creates a render job at Flamenco Server, returning the job object as dictionary."""
+
+    import json
+    from .sdk import Job
+    from ..pillar import pillar_call
+
+    job_attrs = {
+        'status': 'queued',
+        'priority': priority,
+        'name': job_name,
+        'settings': job_settings,
+        'job_type': job_type,
+        'user': user_id,
+        'manager': manager_id,
+        'project': project_id,
+    }
+    if job_description:
+        job_attrs['description'] = job_description
+
+    log.info('Going to create Flamenco job:\n%s',
+             json.dumps(job_attrs, indent=4, sort_keys=True))
+
+    job = Job(job_attrs)
+    await pillar_call(job.create)
+
+    log.info('Job created succesfully: %s', job._id)
+    return job.to_dict()
+
+
+def is_image_type(render_output_type: str) -> bool:
+    """Determines whether the render output type is an image (True) or video (False)."""
+
+    # This list is taken from rna_scene.c:273, rna_enum_image_type_items.
+    video_types = {'AVI_JPEG', 'AVI_RAW', 'FRAMESERVER', 'FFMPEG', 'QUICKTIME'}
+    return render_output_type not in video_types
+
+
+@functools.lru_cache(1)
+def _render_output_path(
+        local_project_path: str,
+        blend_filepath: Path,
+        flamenco_job_output_strip_components: int,
+        flamenco_job_output_path: str,
+        render_image_format: str,
+        flamenco_render_frame_range: str,
+) -> typing.Optional[PurePath]:
+    """Cached version of render_output_path()
+
+    This ensures that redraws of the Flamenco Render and Add-on preferences panels
+    is fast.
+    """
+
+    try:
+        project_path = Path(bpy.path.abspath(local_project_path)).resolve()
+    except FileNotFoundError:
+        # Path.resolve() will raise a FileNotFoundError if the project path doesn't exist.
+        return None
+
+    try:
+        blend_abspath = blend_filepath.resolve().absolute()
+    except FileNotFoundError:
+        # Path.resolve() will raise a FileNotFoundError if the path doesn't exist.
+        return None
+
+    try:
+        proj_rel = blend_abspath.parent.relative_to(project_path)
+    except ValueError:
+        return None
+
+    rel_parts = proj_rel.parts[flamenco_job_output_strip_components:]
+    output_top = Path(flamenco_job_output_path)
+
+    # Strip off '.flamenco' too; we use 'xxx.flamenco.blend' as job file, but
+    # don't want to have all the output paths ending in '.flamenco'.
+    stem = blend_filepath.stem
+    if stem.endswith('.flamenco'):
+        stem = stem[:-9]
+
+    dir_components = output_top.joinpath(*rel_parts) / stem
+
+    # Blender will have to append the file extensions by itself.
+    if is_image_type(render_image_format):
+        return dir_components / '######'
+    return dir_components / flamenco_render_frame_range
+
+
+def render_output_path(context, filepath: Path = None) -> typing.Optional[PurePath]:
+    """Returns the render output path to be sent to Flamenco.
+
+    :param context: the Blender context (used to find Flamenco preferences etc.)
+    :param filepath: the Path of the blend file to render, or None for the current file.
+
+    Returns None when the current blend file is outside the project path.
+    """
+
+    from ..blender import preferences
+
+    scene = context.scene
+    prefs = preferences()
+
+    if filepath is None:
+        filepath = Path(context.blend_data.filepath)
+
+    return _render_output_path(
+        prefs.cloud_project_local_path,
+        filepath,
+        prefs.flamenco_job_output_strip_components,
+        prefs.flamenco_job_output_path,
+        scene.render.image_settings.file_format,
+        scene.flamenco_render_frame_range,
+    )
+
+
+class FLAMENCO_PT_render(bpy.types.Panel, FlamencoPollMixin):
+    bl_label = "Flamenco Render"
+    bl_space_type = 'PROPERTIES'
+    bl_region_type = 'WINDOW'
+    bl_context = "render"
+    bl_options = {'DEFAULT_CLOSED'}
+
+    def draw(self, context):
+        layout = self.layout
+
+        from ..blender import preferences
+
+        prefs = preferences()
+
+        labeled_row = layout.split(0.25, align=True)
+        labeled_row.label('Job Type:')
+        labeled_row.prop(context.scene, 'flamenco_render_job_type', text='')
+
+        labeled_row = layout.split(0.25, align=True)
+        labeled_row.label('Frame Range:')
+        prop_btn_row = labeled_row.row(align=True)
+        prop_btn_row.prop(context.scene, 'flamenco_render_frame_range', text='')
+        prop_btn_row.operator('flamenco.scene_to_frame_range', text='', icon='ARROW_LEFTRIGHT')
+
+        layout.prop(context.scene, 'flamenco_render_job_priority')
+        layout.prop(context.scene, 'flamenco_render_fchunk_size')
+
+        if getattr(context.scene, 'flamenco_render_job_type', None) == 'blender-render-progressive':
+            layout.prop(context.scene, 'flamenco_render_schunk_count')
+
+        readonly_stuff = layout.column(align=True)
+        labeled_row = readonly_stuff.split(0.25, align=True)
+        labeled_row.label('Storage:')
+        prop_btn_row = labeled_row.row(align=True)
+        prop_btn_row.label(prefs.flamenco_job_file_path)
+        props = prop_btn_row.operator(FLAMENCO_OT_explore_file_path.bl_idname,
+                                      text='', icon='DISK_DRIVE')
+        props.path = prefs.flamenco_job_file_path
+
+        labeled_row = readonly_stuff.split(0.25, align=True)
+        labeled_row.label('Output:')
+        prop_btn_row = labeled_row.row(align=True)
+        render_output = render_output_path(context)
+
+        if render_output is None:
+            prop_btn_row.label('Unable to render with Flamenco, outside of project directory.')
+        else:
+            prop_btn_row.label(str(render_output))
+            props = prop_btn_row.operator(FLAMENCO_OT_explore_file_path.bl_idname,
+                                          text='', icon='DISK_DRIVE')
+            props.path = str(render_output.parent)
+
+            flamenco_status = context.window_manager.flamenco_status
+            if flamenco_status == 'IDLE':
+                layout.operator(FLAMENCO_OT_render.bl_idname,
+                                text='Render on Flamenco',
+                                icon='RENDER_ANIMATION')
+            elif flamenco_status == 'PACKING':
+                layout.label('Flamenco is packing your file + dependencies')
+            elif flamenco_status == 'COMMUNICATING':
+                layout.label('Communicating with Flamenco Server')
+            else:
+                layout.label('Unknown Flamenco status %s' % flamenco_status)
+
+
+def activate():
+    """Activates draw callbacks, menu items etc. for Flamenco."""
+
+    global flamenco_is_active
+    log.info('Activating Flamenco')
+    flamenco_is_active = True
+    _render_output_path.cache_clear()
+
+
+def deactivate():
+    """Deactivates draw callbacks, menu items etc. for Flamenco."""
+
+    global flamenco_is_active
+    log.info('Deactivating Flamenco')
+    flamenco_is_active = False
+    _render_output_path.cache_clear()
+
+
+def register():
+    from ..utils import redraw
+
+    bpy.utils.register_class(FlamencoManagerGroup)
+    bpy.utils.register_class(FLAMENCO_OT_fmanagers)
+    bpy.utils.register_class(FLAMENCO_OT_render)
+    bpy.utils.register_class(FLAMENCO_OT_scene_to_frame_range)
+    bpy.utils.register_class(FLAMENCO_OT_copy_files)
+    bpy.utils.register_class(FLAMENCO_OT_explore_file_path)
+    bpy.utils.register_class(FLAMENCO_PT_render)
+
+    scene = bpy.types.Scene
+    scene.flamenco_render_fchunk_size = IntProperty(
+        name='Frame Chunk Size',
+        description='Maximum number of frames to render per task',
+        min=1,
+        default=1,
+    )
+    scene.flamenco_render_schunk_count = IntProperty(
+        name='Number of Sample Chunks',
+        description='Number of Cycles samples chunks to use per frame',
+        min=2,
+        default=3,
+        soft_max=10,
+    )
+    scene.flamenco_render_frame_range = StringProperty(
+        name='Frame Range',
+        description='Frames to render, in "printer range" notation'
+    )
+    scene.flamenco_render_job_type = EnumProperty(
+        name='Job Type',
+        items=[
+            ('blender-render', 'Simple Render', 'Simple frame-by-frame render'),
+            ('blender-render-progressive', 'Progressive Render',
+             'Each frame is rendered multiple times with different Cycles sample chunks, then combined'),
+        ]
+    )
+
+    scene.flamenco_render_job_priority = IntProperty(
+        name='Job Priority',
+        min=0,
+        default=50,
+        max=100,
+        description='Higher numbers mean higher priority'
+    )
+
+    bpy.types.WindowManager.flamenco_status = EnumProperty(
+        items=[
+            ('IDLE', 'IDLE', 'Not doing anything.'),
+            ('PACKING', 'PACKING', 'BAM-packing all dependencies.'),
+            ('COMMUNICATING', 'COMMUNICATING', 'Communicating with Flamenco Server.'),
+        ],
+        name='flamenco_status',
+        default='IDLE',
+        description='Current status of the Flamenco add-on',
+        update=redraw)
+
+
+def unregister():
+    deactivate()
+    bpy.utils.unregister_module(__name__)
+
+    try:
+        del bpy.types.Scene.flamenco_render_fchunk_size
+    except AttributeError:
+        pass
+    try:
+        del bpy.types.Scene.flamenco_render_schunk_count
+    except AttributeError:
+        pass
+    try:
+        del bpy.types.Scene.flamenco_render_frame_range
+    except AttributeError:
+        pass
+    try:
+        del bpy.types.Scene.flamenco_render_job_type
+    except AttributeError:
+        pass
+    try:
+        del bpy.types.Scene.flamenco_render_job_priority
+    except AttributeError:
+        pass
+    try:
+        del bpy.types.WindowManager.flamenco_status
+    except AttributeError:
+        pass

blender_cloud/flamenco/bam_interface.py

@@ -0,0 +1,175 @@
+"""BAM packing interface for Flamenco."""
+
+import functools
+import logging
+from pathlib import Path
+import typing
+
+# Timeout of the BAM subprocess, in seconds.
+SUBPROC_READLINE_TIMEOUT = 600
+log = logging.getLogger(__name__)
+
+
+class CommandExecutionError(Exception):
+    """Raised when there was an error executing a BAM command."""
+    pass
+
+
+if 'bam_supports_exclude_option' in locals():
+    locals()['bam_supports_exclude_option'].cache_clear()
+
+
+@functools.lru_cache(maxsize=1)
+def bam_supports_exclude_option() -> bool:
+    """Returns True if the version of BAM bundled with Blender supports --exclude.
+
+    This feature was added to BAM 1.1.7, so we can do a simple version check.
+    """
+
+    try:
+        import io_blend_utils
+    except ImportError:
+        # If this happens, BAM won't work at all. However, this function can be called from
+        # the GUI; by being a bit careful while importing, we avoid breaking Blender's GUI.
+        log.exception('Error importing io_blend_utils module.')
+        return False
+
+    return io_blend_utils.bl_info['version'] >= (1, 1, 7)
+
+
+async def bam_copy(base_blendfile: Path, target_blendfile: Path,
+                   exclusion_filter: str) -> typing.List[Path]:
+    """Uses BAM to copy the given file and dependencies to the target blendfile.
+
+    Due to the way blendfile_pack.py is programmed/structured, we cannot import it
+    and call a function; it has to be run in a subprocess.
+
+    :raises: asyncio.CanceledError if the task was cancelled.
+    :raises: asyncio.TimeoutError if reading a line from the BAM process timed out.
+    :raises: CommandExecutionError if the subprocess failed or output invalid UTF-8.
+    :returns: a list of missing sources; hopefully empty.
+    """
+
+    import asyncio
+    import shlex
+    import subprocess
+
+    import bpy
+    import io_blend_utils
+
+    args = [
+        bpy.app.binary_path_python,
+        '-m', 'bam.pack',
+        '--input', str(base_blendfile),
+        '--output', str(target_blendfile),
+        '--mode', 'FILE',
+    ]
+
+    if exclusion_filter:
+        if bam_supports_exclude_option():
+            args.extend(['--exclude', exclusion_filter])
+        else:
+            log.warning('Your version of Blender does not support the exclusion filter, '
+                        'copying all files.')
+
+    cmd_to_log = ' '.join(shlex.quote(s) for s in args)
+    log.info('Executing %s', cmd_to_log)
+
+    proc = await asyncio.create_subprocess_exec(
+        *args,
+        env={'PYTHONPATH': io_blend_utils.pythonpath()},
+        stdin=subprocess.DEVNULL,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+    )
+
+    missing_sources = []
+
+    try:
+        while not proc.stdout.at_eof():
+            line = await asyncio.wait_for(proc.stdout.readline(),
+                                          SUBPROC_READLINE_TIMEOUT)
+            if not line:
+                # EOF received, so let's bail.
+                break
+
+            try:
+                line = line.decode('utf8')
+            except UnicodeDecodeError as ex:
+                raise CommandExecutionError('Command produced non-UTF8 output, '
+                                            'aborting: %s' % ex)
+
+            line = line.rstrip()
+            if 'source missing:' in line:
+                path = parse_missing_source(line)
+                missing_sources.append(path)
+                log.warning('Source is missing: %s', path)
+
+            log.info('  %s', line)
+    finally:
+        if proc.returncode is None:
+            # Always wait for the process, to avoid zombies.
+            try:
+                proc.kill()
+            except ProcessLookupError:
+                # The process is already stopped, so killing is impossible. That's ok.
+                log.debug("The process was already stopped, aborting is impossible. That's ok.")
+            await proc.wait()
+        log.info('The process stopped with status code %i', proc.returncode)
+
+    if proc.returncode:
+        raise CommandExecutionError('Process stopped with status %i' % proc.returncode)
+
+    return missing_sources
+
+
+def parse_missing_source(line: str) -> Path:
+    r"""Parses a "missing source" line into a pathlib.Path.
+
+    >>> parse_missing_source(r"  source missing: b'D\xc3\xaffficult \xc3\x9cTF-8 filename'")
+    PosixPath('Dïfficult ÜTF-8 filename')
+    >>> parse_missing_source(r"  source missing: b'D\xfffficult Win1252 f\xeflen\xe6me'")
+    PosixPath('D<EFBFBD>fficult Win1252 f<>len<65>me')
+    """
+
+    _, missing_source = line.split(': ', 1)
+    missing_source_as_bytes = parse_byte_literal(missing_source.strip())
+
+    # The file could originate from any platform, so UTF-8 and the current platform's
+    # filesystem encodings are just guesses.
+    try:
+        missing_source = missing_source_as_bytes.decode('utf8')
+    except UnicodeDecodeError:
+        import sys
+        try:
+            missing_source = missing_source_as_bytes.decode(sys.getfilesystemencoding())
+        except UnicodeDecodeError:
+            missing_source = missing_source_as_bytes.decode('ascii', errors='replace')
+
+    path = Path(missing_source)
+
+    return path
+
+
+def parse_byte_literal(bytes_literal: str) -> bytes:
+    r"""Parses a repr(bytes) output into a bytes object.
+
+    >>> parse_byte_literal(r"b'D\xc3\xaffficult \xc3\x9cTF-8 filename'")
+    b'D\xc3\xaffficult \xc3\x9cTF-8 filename'
+    >>> parse_byte_literal(r"b'D\xeffficult Win1252 f\xeflen\xe6me'")
+    b'D\xeffficult Win1252 f\xeflen\xe6me'
+    """
+
+    # Some very basic assertions to make sure we have a proper bytes literal.
+    assert bytes_literal[0] == "b"
+    assert bytes_literal[1] in {'"', "'"}
+    assert bytes_literal[-1] == bytes_literal[1]
+
+    import ast
+    return ast.literal_eval(bytes_literal)
+
+
+if __name__ == '__main__':
+    import doctest
+
+    doctest.testmod()

blender_cloud/flamenco/sdk.py

@@ -0,0 +1,55 @@
+import functools
+import pathlib
+
+from pillarsdk.resource import List, Find, Create
+
+
+class Manager(List, Find):
+    """Manager class wrapping the REST nodes endpoint"""
+    path = 'flamenco/managers'
+    PurePlatformPath = pathlib.PurePath
+
+    @functools.lru_cache()
+    def _sorted_path_replacements(self) -> list:
+        import sys
+
+        if self.path_replacement is None:
+            return []
+
+        print('SORTING PATH REPLACEMENTS')
+
+        items = self.path_replacement.to_dict().items()
+
+        def by_length(item):
+            return -len(item[0]), item[0]
+
+        platform = sys.platform
+        return [(varname, platform_replacements[platform])
+                for varname, platform_replacements in sorted(items, key=by_length)]
+
+    def replace_path(self, some_path: pathlib.PurePath) -> str:
+        """Performs path variable replacement.
+
+        Tries to find platform-specific path prefixes, and replaces them with
+        variables.
+        """
+
+        for varname, path in self._sorted_path_replacements():
+            replacement = self.PurePlatformPath(path)
+            try:
+                relpath = some_path.relative_to(replacement)
+            except ValueError:
+                # Not relative to each other, so no replacement possible
+                continue
+
+            replacement_root = self.PurePlatformPath('{%s}' % varname)
+            return (replacement_root / relpath).as_posix()
+
+        return some_path.as_posix()
+
+
+class Job(List, Find, Create):
+    """Job class wrapping the REST nodes endpoint
+    """
+    path = 'flamenco/jobs'
+    ensure_query_projections = {'project': 1}

blender_cloud/pillar.py

@@ -115,6 +115,12 @@ def with_existing_dir(filename: str, open_mode: str, encoding=None):
         yield file_object
 
 
+def _shorten(somestr: str, maxlen=40) -> str:
+    """Shortens strings for logging"""
+
+    return (somestr[:maxlen - 3] + '...') if len(somestr) > maxlen else somestr
+
+
 def save_as_json(pillar_resource, json_filename):
     with with_existing_dir(json_filename, 'w') as outfile:
         log.debug('Saving metadata to %r' % json_filename)
@@ -200,8 +206,11 @@ def pillar_api(pillar_endpoint: str = None, caching=True) -> pillarsdk.Api:
     return _pillar_api[caching]
 
 
-# No more than this many Pillar calls should be made simultaneously
-pillar_semaphore = asyncio.Semaphore(3)
+# This is an asyncio.Semaphore object, which is late-instantiated to be sure
+# the asyncio loop has been created properly. On Windows we create a new one,
+# which can cause this semaphore to still be linked against the old default
+# loop.
+pillar_semaphore = None
 
 
 async def pillar_call(pillar_func, *args, caching=True, **kwargs):
@@ -214,8 +223,21 @@ async def pillar_call(pillar_func, *args, caching=True, **kwargs):
     partial = functools.partial(pillar_func, *args, api=pillar_api(caching=caching), **kwargs)
     loop = asyncio.get_event_loop()
 
-    async with pillar_semaphore:
+    # Use explicit calls to acquire() and release() so that we have more control over
+    # how long we wait and how we handle timeouts.
+    try:
+        await asyncio.wait_for(pillar_semaphore.acquire(), timeout=10, loop=loop)
+    except asyncio.TimeoutError:
+        log.info('Waiting for semaphore to call %s', pillar_func.__name__)
+        try:
+            await asyncio.wait_for(pillar_semaphore.acquire(), timeout=50, loop=loop)
+        except asyncio.TimeoutError:
+            raise RuntimeError('Timeout waiting for Pillar Semaphore!')
+
+    try:
         return await loop.run_in_executor(None, partial)
+    finally:
+        pillar_semaphore.release()
 
 
 def sync_call(pillar_func, *args, caching=True, **kwargs):
@@ -441,9 +463,9 @@ async def download_to_file(url, filename, *,
         log.debug('Downloading was cancelled before doing the GET')
         raise asyncio.CancelledError('Downloading was cancelled')
 
-    log.debug('Performing GET %s', url)
+    log.debug('Performing GET %s', _shorten(url))
     response = await loop.run_in_executor(None, perform_get_request)
-    log.debug('Status %i from GET %s', response.status_code, url)
+    log.debug('Status %i from GET %s', response.status_code, _shorten(url))
     response.raise_for_status()
 
     if response.status_code == 304:
@@ -457,9 +479,9 @@ async def download_to_file(url, filename, *,
         log.debug('Downloading was cancelled before downloading the GET response')
         raise asyncio.CancelledError('Downloading was cancelled')
 
-    log.debug('Downloading response of GET %s', url)
+    log.debug('Downloading response of GET %s', _shorten(url))
     await loop.run_in_executor(None, download_loop)
-    log.debug('Done downloading response of GET %s', url)
+    log.debug('Done downloading response of GET %s', _shorten(url))
 
     # We're done downloading, now we have something cached we can use.
     log.debug('Saving header cache to %s', header_store)
@@ -534,7 +556,8 @@ async def fetch_texture_thumbs(parent_node_uuid: str, desired_size: str,
              for texture_node in texture_nodes)
 
     # raises any exception from failed handle_texture_node() calls.
-    await asyncio.gather(*coros)
+    loop = asyncio.get_event_loop()
+    await asyncio.gather(*coros, loop=loop)
 
     log.info('fetch_texture_thumbs: Done downloading texture thumbnails')
 
@@ -747,7 +770,8 @@ async def download_texture(texture_node,
                                     future=future)
         downloaders.append(dlr)
 
-    return await asyncio.gather(*downloaders, return_exceptions=True)
+    loop = asyncio.get_event_loop()
+    return await asyncio.gather(*downloaders, return_exceptions=True, loop=loop)
 
 
 async def upload_file(project_id: str, file_path: pathlib.Path, *,
@@ -773,9 +797,9 @@ async def upload_file(project_id: str, file_path: pathlib.Path, *,
         log.debug('Uploading was cancelled before doing the POST')
         raise asyncio.CancelledError('Uploading was cancelled')
 
-    log.debug('Performing POST %s', url)
+    log.debug('Performing POST %s', _shorten(url))
     response = await loop.run_in_executor(None, upload)
-    log.debug('Status %i from POST %s', response.status_code, url)
+    log.debug('Status %i from POST %s', response.status_code, _shorten(url))
     response.raise_for_status()
 
     resp = response.json()
@@ -840,6 +864,39 @@ class PillarOperatorMixin:
                     'Please subscribe to the blender cloud at https://cloud.blender.org/join')
 
 
+class AuthenticatedPillarOperatorMixin(PillarOperatorMixin):
+    """Checks credentials, to be used at the start of async_execute().
+
+    Sets self.user_id to the current user's ID, and self.db_user to the user info dict,
+    if authentication was succesful; sets both to None if not.
+    """
+
+    async def authenticate(self, context) -> bool:
+        from . import pillar
+
+        self.log.info('Checking credentials')
+        self.user_id = None
+        self.db_user = None
+        try:
+            self.db_user = await self.check_credentials(context, ())
+        except pillar.UserNotLoggedInError as ex:
+            self.log.info('Not logged in error raised: %s', ex)
+            self.report({'ERROR'}, 'Please log in on Blender ID first.')
+            self.quit()
+            return False
+        except requests.exceptions.ConnectionError:
+            self.log.exception('Error checking pillar credentials.')
+            self.report({'ERROR'}, 'Unable to connect to Blender Cloud, '
+                                   'check your internet connection.')
+            self.quit()
+            return False
+
+
+        self.user_id = self.db_user['_id']
+        return True
+
+
+
 async def find_or_create_node(where: dict,
                               additional_create_props: dict = None,
                               projection: dict = None,

blender_cloud/texture_browser.py

@@ -457,7 +457,7 @@ class BlenderCloudBrowser(pillar.PillarOperatorMixin,
 
         return menu_item
 
-    def update_menu_item(self, node, *args) -> MenuItem:
+    def update_menu_item(self, node, *args):
         node_uuid = node['_id']
 
         # Just make this thread-safe to be on the safe side.
@@ -538,6 +538,7 @@ class BlenderCloudBrowser(pillar.PillarOperatorMixin,
             self.add_menu_item(node, None, 'SPINNER', texture_node['name'])
 
         def thumbnail_loaded(node, file_desc, thumb_path):
+            self.log.debug('Node %s thumbnail loaded', node['_id'])
             self.update_menu_item(node, file_desc, thumb_path)
 
         await pillar.fetch_texture_thumbs(node_uuid, 's', directory,

blender_cloud/utils.py

@@ -62,3 +62,41 @@ def find_in_path(path: pathlib.Path, filename: str) -> pathlib.Path:
                 return subpath
 
     return None
+
+
+def pyside_cache(propname):
+    """Decorator, stores the result of the decorated callable in Python-managed memory.
+
+    This is to work around the warning at
+    https://www.blender.org/api/blender_python_api_master/bpy.props.html#bpy.props.EnumProperty
+    """
+
+    if callable(propname):
+        raise TypeError('Usage: pyside_cache("property_name")')
+
+    def decorator(wrapped):
+        """Stores the result of the callable in Python-managed memory.
+
+        This is to work around the warning at
+        https://www.blender.org/api/blender_python_api_master/bpy.props.html#bpy.props.EnumProperty
+        """
+
+        import functools
+
+        @functools.wraps(wrapped)
+        # We can't use (*args, **kwargs), because EnumProperty explicitly checks
+        # for the number of fixed positional arguments.
+        def wrapper(self, context):
+            result = None
+            try:
+                result = wrapped(self, context)
+                return result
+            finally:
+                rna_type, rna_info = getattr(self.bl_rna, propname)
+                rna_info['_cached_result'] = result
+        return wrapper
+    return decorator
+
+
+def redraw(self, context):
+    context.area.tag_redraw()

setup.py

@@ -227,11 +227,11 @@ setup(
               'wheels': BuildWheels},
     name='blender_cloud',
     description='The Blender Cloud addon allows browsing the Blender Cloud from Blender.',
-    version='1.5.2',
+    version='1.7.1',
     author='Sybren A. Stüvel',
     author_email='sybren@stuvel.eu',
     packages=find_packages('.'),
-    data_files=[('blender_cloud', ['README.md']),
+    data_files=[('blender_cloud', ['README.md', 'README-flamenco.md', 'CHANGELOG.md']),
                 ('blender_cloud/icons', glob.glob('blender_cloud/icons/*'))],
     scripts=[],
     url='https://developer.blender.org/diffusion/BCA/',

tests/test_path_replacement.py

@@ -0,0 +1,90 @@
+"""Unittests for blender_cloud.utils.
+
+This unittest requires bpy to be importable, so build Blender as a module and install it
+into your virtualenv. See https://stuvel.eu/files/bconf2016/#/10 for notes how.
+"""
+
+import datetime
+import pathlib
+import unittest.mock
+
+import pillarsdk.utils
+
+from blender_cloud.flamenco import sdk
+
+
+class PathReplacementTest(unittest.TestCase):
+    def setUp(self):
+        self.test_manager = sdk.Manager({
+            '_created': datetime.datetime(2017, 5, 31, 15, 12, 32, tzinfo=pillarsdk.utils.utc),
+            '_etag': 'c39942ee4bcc4658adcc21e4bcdfb0ae',
+            '_id': '592edd609837732a2a272c62',
+            '_updated': datetime.datetime(2017, 6, 8, 14, 51, 3, tzinfo=pillarsdk.utils.utc),
+            'description': 'Manager formerly known as "testman"',
+            'job_types': {'sleep': {'vars': {}}},
+            'name': '<script>alert("this is a manager")</script>',
+            'owner': '592edd609837732a2a272c63',
+            'path_replacement': {'job_storage': {'darwin': '/Volume/shared',
+                                                 'linux': '/shared',
+                                                 'windows': 's:/'},
+                                 'render': {'darwin': '/Volume/render/',
+                                            'linux': '/render/',
+                                            'windows': 'r:/'},
+                                 'longrender': {'darwin': '/Volume/render/long',
+                                                'linux': '/render/long',
+                                                'windows': 'r:/long'},
+                                 },
+            'projects': ['58cbdd5698377322d95eb55e'],
+            'service_account': '592edd609837732a2a272c60',
+            'stats': {'nr_of_workers': 3},
+            'url': 'http://192.168.3.101:8083/',
+            'user_groups': ['58cbdd5698377322d95eb55f'],
+            'variables': {'blender': {'darwin': '/opt/myblenderbuild/blender',
+                                      'linux': '/home/sybren/workspace/build_linux/bin/blender '
+                                               '--enable-new-depsgraph --factory-startup',
+                                      'windows': 'c:/temp/blender.exe'}}}
+        )
+
+    def test_linux(self):
+        # (expected result, input)
+        test_paths = [
+            ('/doesnotexistreally', '/doesnotexistreally'),
+            ('{render}/agent327/scenes/A_01_03_B', '/render/agent327/scenes/A_01_03_B'),
+            ('{job_storage}/render/agent327/scenes', '/shared/render/agent327/scenes'),
+            ('{longrender}/agent327/scenes', '/render/long/agent327/scenes'),
+        ]
+
+        self._do_test(test_paths, 'linux', pathlib.PurePosixPath)
+
+    def test_windows(self):
+        # (expected result, input)
+        test_paths = [
+            ('c:/doesnotexistreally', 'c:/doesnotexistreally'),
+            ('c:/some/path', r'c:\some\path'),
+            ('{render}/agent327/scenes/A_01_03_B', r'R:\agent327\scenes\A_01_03_B'),
+            ('{render}/agent327/scenes/A_01_03_B', r'r:\agent327\scenes\A_01_03_B'),
+            ('{render}/agent327/scenes/A_01_03_B', r'r:/agent327/scenes/A_01_03_B'),
+            ('{job_storage}/render/agent327/scenes', 's:/render/agent327/scenes'),
+            ('{longrender}/agent327/scenes', 'r:/long/agent327/scenes'),
+        ]
+
+        self._do_test(test_paths, 'windows', pathlib.PureWindowsPath)
+
+    def test_darwin(self):
+        # (expected result, input)
+        test_paths = [
+            ('/Volume/doesnotexistreally', '/Volume/doesnotexistreally'),
+            ('{render}/agent327/scenes/A_01_03_B', r'/Volume/render/agent327/scenes/A_01_03_B'),
+            ('{job_storage}/render/agent327/scenes', '/Volume/shared/render/agent327/scenes'),
+            ('{longrender}/agent327/scenes', '/Volume/render/long/agent327/scenes'),
+        ]
+
+        self._do_test(test_paths, 'darwin', pathlib.PurePosixPath)
+
+    def _do_test(self, test_paths, platform, pathclass):
+        self.test_manager.PurePlatformPath = pathclass
+        with unittest.mock.patch('sys.platform', platform):
+            for expected_result, input_path in test_paths:
+                self.assertEqual(expected_result,
+                                 self.test_manager.replace_path(pathclass(input_path)),
+                                 'for input %s on platform %s' % (input_path, platform))

update_version.sh

@@ -1,15 +1,19 @@
 #!/bin/bash
 
-if [ -z "$1" ]; then
+VERSION="${1/version-}"
+
+if [ -z "$VERSION" ]; then
     echo "Usage: $0 new-version" >&2
     exit 1
 fi
 
-BL_INFO_VER=$(echo "$1" | sed 's/\./, /g')
+BL_INFO_VER=$(echo "$VERSION" | sed 's/\./, /g')
 
-sed "s/version='[^']*'/version='$1'/" -i setup.py
+sed "s/version='[^']*'/version='$VERSION'/" -i setup.py
 sed "s/'version': ([^)]*)/'version': ($BL_INFO_VER)/" -i blender_cloud/__init__.py
 
 git diff
 echo
-echo "Don't forget to commit!"
+echo "Don't forget to commit and tag:"
+echo git commit -m \'Bumped version to $VERSION\' setup.py blender_cloud/__init__.py
+echo git tag -a version-$VERSION -m \'Tagged version $VERSION\'