Include README-flamenco.md in the distribution ZIP

Lots to do:

- Doesn't call BAM yet to copy files onto the job storage folder (even
  though you can configure that folder).
- Uses the same project as Attract, so you have to select it in an
  unintuitive location. Also, you can only start Flamenco jobs on a project
  that is Attract-enabled (and not necessarily Flamenco-enabled).

.gitignore

@@ -1,9 +1,10 @@
 *.pyc
 *.swp
-*.blend
-*.blend[1-9]
+/*.blend*
 blender_cloud/wheels/*.whl
 /textures*/
 /test_*.py
 /dist/
 /build/
+/addon-bundle/*.zip
+__pycache__

README-flamenco.md

@@ -0,0 +1,57 @@
+# Flamenco
+
+The Blender Cloud add-on has preliminary support for [Flamenco](https://flamenco.io/).
+It requires a project on the [Blender Cloud](https://cloud.blender.org/) that is set up for
+Flamenco, and it requires you to be logged in as a user with rights to use Flamenco.
+
+
+## Quirks
+
+Flamenco support is unpolished, so it has some quirks.
+
+- Project selection happens through the Attract project selector. As a result, you can only
+  select Attract-enabled projects (even when they are not set up for Flamenco). Be careful
+  which project you select.
+- The top level directory of the project is also set through the Attract properties.
+- Defaults are heavily biased for our use in the Blender Institute.
+- Settings that should be project-specific are not, i.e. are regular add-on preferences.
+- Sending a project to Flamenco will check the "File Extensions" setting in the Output panel,
+  and save the blend file to the current filename.
+
+## Render job file locations
+
+Rendering via Flamenco roughly comprises of two steps:
+
+1. Packing the file to render with its dependencies, and placing them in the "job file path".
+2. Rendering, and placing the output files in the "job output path".
+
+### Job file path
+
+The "job file path" consists of the following path components:
+
+1. The add-on preference "job file path", e.g. `/render/_flamenco/storage`
+2. The current date and time, your Blender Cloud username, and the name of the current blend file.
+3. The name of the current blend file.
+
+For example:
+
+`/render/_flamenco/storage/2017-01-18-104841.931387-sybren-03_02_A.layout/03_02_A.layout.blend`
+
+### Job output path
+
+The file path of output files consists of the following path components:
+
+1. The add-on preference "job file path", e.g. `/render/agent327/frames`
+2. The path of the current blend file, relative to the project directory. The first N components
+   of this path can be stripped; when N=1 it turns `scenes/03-searching/03_02_A-snooping/` into
+   `03-searching/03_02_A-snooping/`.
+3. The name of the current blend file, without `.blend`.
+4. The file name depends on the type of output:
+   - When rendering to image files: A 5-digit frame number with the required file extension.
+   - When rendering to a video file: The frame range with the required file extension.
+
+For example:
+
+`/render/agent327/frames/03-searching/03_02_A-snooping/03_02_A.layout/00441.exr`
+
+`/render/agent327/frames/03-searching/03_02_A-snooping/03_02_A.layout/14-51,60-133.mkv`

README.md

@@ -19,6 +19,28 @@ This addon is a *proof of concept* demonstrating the following features:
 
 {F299745}
 
+Installing the addon
+--------------------
+
+* If you don't have one already, sign up for an account at
+  the [Blender ID site](https://www.blender.org/id/).
+* If you had a previous version of the addon installed, deactivate it
+  and restart Blender.
+* Install and log in with the
+  [Blender ID addon](https://developer.blender.org/diffusion/BIA/).
+* Install the Blender Cloud addon in Blender (User Preferences →
+  Addons → Install from file...) by pointing it to
+  `blender_cloud*.addon.zip`.
+* Enable the addon in User Preferences → Addons → System.
+
+Running the addon
+-----------------
+
+After installing the Blender Cloud addon, press Ctrl+Alt+Shift+A to
+activate it (yes, this needs work). Downloaded textures are loaded into
+image datablocks. The download location can be configured in the addon
+preferences.
+
 
 Building an installable ZIP file
 --------------------------------
@@ -40,38 +62,6 @@ can find them, or be bundled as wheel files in `blender_cloud/wheels`.
 The `python setup.py bdist` command gathers the dependencies and bundles
 them as wheel files.
 
-Installing the addon
---------------------
-
-* To build the addon, run `python setup.py bdist` as described above.
-* If you don't have one already, sign up for an account at
-  the [Blender ID site](https://www.blender.org/id/).
-* As a final step, install and log in with the
-  [Blender ID addon](https://developer.blender.org/diffusion/BIA/).
-* Install the Blender Cloud addon in Blender (User Preferences →
-  Addons → Install from file...) by pointing it to
-  `dist/blender_cloud*.addon.zip`.
-* Enable the addon in User Preferences → Addons → System.
-
-NOTE: The addon requires HTTPS connections, and thus is dependent on
-[D1845](https://developer.blender.org/D1845). You can do either of
-these:
-
-* Build Blender yourself
-* Get a recent copy from the buildbot
-* Copy certificate authority certificate PEM file to
-  `blender/2.77/python/lib/python3.5/site-packages/requests/cacert.pem`.
-  You can use the same file from your local requests installation, or
-  use `/etc/ssl/certs/ca-certificates.crt`.
-
-Running the addon
------------------
-
-After installing the Blender Cloud addon, press Ctrl+Alt+Shift+A to
-activate it (yes, this needs work). Downloaded textures are loaded into
-image datablocks. The download location can be configured in the addon
-preferences.
-
 
 Design
 ------

addon-bundle/README.txt

@@ -0,0 +1,52 @@
+Blender Cloud Addon
+===================
+
+Congratulations on downloading the Blender Cloud addon. For your
+convenience, we have bundled it with the Blender ID addon.
+
+To use the Blender Cloud addon, perform the following steps:
+
+- Use Blender (File, User Preferences, Addons, Install from file)
+  to install blender_id-x.x.x.addon.zip
+
+- If you had a previous version of the Blender Cloud addon installed,
+  restart Blender now.
+
+- Log in with your Blender ID.
+
+- Use Blender to install blender_cloud-x.x.x.addon.zip
+
+    If you don't see the addon in the list, enable the Testing
+    category.
+
+- Press Ctrl+Alt+Shift+A to start the texture browser.
+
+- Visit the User Preferences, Addons panel, to use the Blender Sync
+  feature.
+
+
+Support for Blenders not from blender.org
+-----------------------------------------
+
+Maybe you use Blender from another source than blender.org, such as an
+Ubuntu package. If that is the case, you have to make sure that the
+Python package "requests" is installed. On Ubuntu Linux this can be
+done with the command
+
+    sudo apt-get install python3-requests
+
+On other platforms & distributions this might be different.
+
+Blender uses Python 3.5, so make sure you install the package for the
+correct version of Python.
+
+
+Subscribing to the Blender Cloud
+--------------------------------
+
+The Blender Sync feature is free to use for everybody with a Blender
+ID account. In order to use the Texture Browser you need to have a
+Blender Cloud subscription. If you didn't subscribe yet, go to:
+
+    https://cloud.blender.org/join
+

addon-bundle/bundle.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
+cd $(dirname $(readlink -f $0))
+
+BCLOUD=$(ls ../dist/blender_cloud-*.addon.zip | tail -n 1)
+BID=$(ls ../../../blender-id-addon/dist/blender_id-*.addon.zip | tail -n 1)
+
+[ -z "$BCLOUD" ] && echo "BCloud addon not found" >&2
+[ -z "$BID" ] && echo "B'ID addon not found" >&2
+
+cp -va $BCLOUD $BID .
+
+BUNDLE=$(basename $BCLOUD)
+BUNDLE=${BUNDLE/.addon.zip/-bundle-UNZIP_ME_FIRST.zip}
+
+zip -9 $BUNDLE $(basename $BCLOUD) $(basename $BID) README.txt
+
+dolphin --select $BUNDLE 2>/dev/null >/dev/null & disown
+echo "CREATED: $BUNDLE"

blender_cloud/__init__.py

@@ -19,26 +19,20 @@
 # <pep8 compliant>
 
 bl_info = {
-    "name": "Blender Cloud Texture Browser",
-    "author": "Sybren A. Stüvel and Francesco Siddi",
-    "version": (0, 2, 0),
-    "blender": (2, 77, 0),
-    "location": "TO BE DETERMINED",
-    "description": "Allows downloading of textures from the Blender Cloud. Requires "
-                   "the Blender ID addon.",
-    "wiki_url": "http://wiki.blender.org/index.php/Extensions:2.6/Py/"
-                "Scripts/System/BlenderCloud",
-    "category": "System",
-    "support": "TESTING"
+    'name': 'Blender Cloud',
+    "author": "Sybren A. Stüvel, Francesco Siddi, Inês Almeida, Antony Riakiotakis",
+    'version': (1, 5, 999),
+    'blender': (2, 77, 0),
+    'location': 'Addon Preferences panel, and Ctrl+Shift+Alt+A anywhere for texture browser',
+    'description': 'Texture library browser and Blender Sync. Requires the Blender ID addon '
+                   'and Blender 2.77a or newer.',
+    'wiki_url': 'https://wiki.blender.org/index.php/Extensions:2.6/Py/'
+                'Scripts/System/BlenderCloud',
+    'category': 'System',
 }
 
 import logging
 
-logging.basicConfig(level=logging.INFO,
-                    format='%(asctime)-15s %(levelname)8s %(name)s %(message)s')
-logging.getLogger('cachecontrol').setLevel(logging.DEBUG)
-logging.getLogger(__name__).setLevel(logging.DEBUG)
-
 # Support reloading
 if 'pillar' in locals():
     import importlib
@@ -50,16 +44,21 @@ if 'pillar' in locals():
     cache = importlib.reload(cache)
 else:
     from . import wheels
+
     wheels.load_wheels()
 
     from . import pillar, cache
 
+log = logging.getLogger(__name__)
+
 
 def register():
     """Late-loads and registers the Blender-dependent submodules."""
 
     import sys
 
+    _monkey_patch_requests()
+
     # Support reloading
     if '%s.blender' % __name__ in sys.modules:
         import importlib
@@ -70,25 +69,57 @@ def register():
             sys.modules[modname] = module
             return module
 
+        reload_mod('blendfile')
+        reload_mod('home_project')
+        reload_mod('utils')
+
         blender = reload_mod('blender')
-        gui = reload_mod('gui')
         async_loop = reload_mod('async_loop')
-        asset_engine = reload_mod('asset_engine')
+        texture_browser = reload_mod('texture_browser')
+        settings_sync = reload_mod('settings_sync')
+        image_sharing = reload_mod('image_sharing')
+        attract = reload_mod('attract')
+        flamenco = reload_mod('flamenco')
     else:
-        from . import blender, gui, async_loop, asset_engine
+        from . import (blender, texture_browser, async_loop, settings_sync, blendfile, home_project,
+                       image_sharing, attract, flamenco)
 
     async_loop.setup_asyncio_executor()
     async_loop.register()
 
+    flamenco.register()
+    texture_browser.register()
     blender.register()
-    gui.register()
-    asset_engine.register()
+    settings_sync.register()
+    image_sharing.register()
+    attract.register()
+
+
+def _monkey_patch_requests():
+    """Monkey-patch old versions of Requests.
+
+    This is required for the Mac version of Blender 2.77a.
+    """
+
+    import requests
+
+    if requests.__build__ >= 0x020601:
+        return
+
+    log.info('Monkey-patching requests version %s', requests.__version__)
+    from requests.packages.urllib3.response import HTTPResponse
+    HTTPResponse.chunked = False
+    HTTPResponse.chunk_left = None
 
 
 def unregister():
-    from . import blender, gui, async_loop
+    from . import (blender, texture_browser, async_loop, settings_sync, image_sharing, attract,
+                   flamenco)
 
-    gui.unregister()
+    image_sharing.unregister()
+    attract.unregister()
+    settings_sync.unregister()
     blender.unregister()
+    texture_browser.unregister()
     async_loop.unregister()
-
+    flamenco.unregister()

blender_cloud/asset_engine.py

@@ -1,468 +0,0 @@
-# ##### BEGIN GPL LICENSE BLOCK #####
-#
-#  This program is free software; you can redistribute it and/or
-#  modify it under the terms of the GNU General Public License
-#  as published by the Free Software Foundation; either version 2
-#  of the License, or (at your option) any later version.
-#
-#  This program is distributed in the hope that it will be useful,
-#  but WITHOUT ANY WARRANTY; without even the implied warranty of
-#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#  GNU General Public License for more details.
-#
-#  You should have received a copy of the GNU General Public License
-#  along with this program; if not, write to the Free Software Foundation,
-#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
-#
-# ##### END GPL LICENSE BLOCK #####
-
-# <pep8 compliant>
-
-"""Blender Cloud interface for the Asset Engine."""
-import asyncio
-import logging
-
-import bpy
-import time
-from bpy.types import (AssetEngine, AssetList, FileSelectParams,
-                       AssetUUIDList, AssetUUID,
-                       Panel, PropertyGroup, UIList)
-from bpy.props import (StringProperty,
-                       BoolProperty,
-                       IntProperty,
-                       FloatProperty,
-                       EnumProperty,
-                       CollectionProperty)
-
-from . import async_loop, pillar, cache
-
-ASSET_ENGINE_ID = 0xc0ffee
-
-
-def object_id_to_ea_uuid(pillar_object_id: str) -> tuple:
-    """Turns a ObjectId string from Pillar to a tuple of 4 ints.
-
-    >>> object_id_to_ea_uuid('55f2d0dc2beb33006e43dd7e')
-    (12648430, 1441976540, 736834304, 1849941374)
-    >>> object_id_to_ea_uuid('55f2d0dc2beb33006e43dd7e') == \
-        (ASSET_ENGINE_ID, 0x55f2d0dc, 0x2beb3300, 0x6e43dd7e)
-    True
-
-    The first int is hard-coded to indicate this asset engine.
-    The other three ints are 32 bit each, and are taken from the 12-byte
-    ObjectId (see https://docs.mongodb.org/manual/reference/method/ObjectId/)
-    """
-
-    # Make sure it's a 12-byte number in hex.
-    pillar_object_id = pillar_object_id.rjust(24, '0')
-
-    return (ASSET_ENGINE_ID,
-            int(pillar_object_id[0:8], 16),
-            int(pillar_object_id[8:16], 16),
-            int(pillar_object_id[16:24], 16))
-
-
-class BCloudAssetEngineDirListJob:
-    def __init__(self, job_id: int, path: pillar.CloudPath, future: asyncio.Future = None):
-        self.log = logging.getLogger('%s.%s' % (__name__, BCloudAssetEngineDirListJob.__qualname__))
-        self.log.debug('Starting new dirlist job (id=%i) for path %r', job_id, path)
-
-        self.job_id = job_id
-        self.status = {'INVALID'}
-        self.progress = 0.0
-        self.path = path
-
-        # Start a new asynchronous task.
-        self.signalling_future = future or asyncio.Future()
-        # self.async_task = asyncio.ensure_future(self.async_download_previews())
-        # self.log.debug('Created new task %r', self.async_task)
-        # self.status = {'VALID', 'RUNNING'}
-
-        self.async_task = None
-        self.status = {'VALID'}
-
-    def __repr__(self):
-        return '%s(job_id=%i, path=%s, future=%s)' % (type(self), self.job_id, self.path,
-                                                      self.signalling_future)
-
-    def stop(self):
-        self.log.debug('Stopping async task')
-        if self.async_task is None:
-            self.log.debug('No async task, trivially stopped')
-            return
-
-        # Signal that we want to stop.
-        if not self.signalling_future.done():
-            self.log.info("Signalling that we want to cancel anything that's running.")
-            self.signalling_future.cancel()
-
-        # Wait until the asynchronous task is done.
-        if not self.async_task.done():
-            # TODO: Should we really block? Or let it disappear into the background?
-            self.log.info("blocking until async task is done.")
-            loop = asyncio.get_event_loop()
-            try:
-                loop.run_until_complete(self.async_task)
-            except asyncio.CancelledError:
-                self.log.info('Asynchronous task was cancelled')
-                return
-
-        # noinspection PyBroadException
-        try:
-            self.async_task.result()  # This re-raises any exception of the task.
-        except asyncio.CancelledError:
-            self.log.info('Asynchronous task was cancelled')
-        except Exception:
-            self.log.exception("Exception from asynchronous task")
-
-    async def async_download_previews(self):
-        self.log.info('Asynchronously downloading previews')
-
-        def thumbnail_loading(texture_node):
-            self.log.debug('Thumbnail for node %r loading', texture_node)
-            # self.add_menu_item(node, None, 'SPINNER', texture_node['name'])
-
-        def thumbnail_loaded(texture_node, file_desc, thumb_path):
-            self.log.debug('Thumbnail for node %r loaded, thumb at %r', texture_node, thumb_path)
-            # self.update_menu_item(node, file_desc, thumb_path, file_desc['filename'])
-
-        node_uuid = self.path.node_uuid
-        project_uuid = self.path.project_uuid
-
-        # Download either by group_texture node or project UUID (showing all top-level nodes)
-        if node_uuid:
-            self.log.debug('Getting subnodes for parent node %r', node_uuid)
-            children = await pillar.get_nodes(parent_node_uuid=node_uuid,
-                                              node_type='group_textures')
-        elif project_uuid:
-            self.log.debug('Getting subnodes for project node %r', project_uuid)
-            children = await pillar.get_nodes(project_uuid, '')
-
-        else:
-            # TODO: add "nothing here" icon and trigger re-draw
-            self.log.warning("Not node UUID and no project UUID, I can't do anything!")
-            return
-
-        # Download all child nodes
-        self.log.debug('Iterating over child nodes of %r', node_uuid)
-        for child in children:
-            self.log.debug('  - %(_id)s = %(name)s' % child)
-            # self.add_menu_item(child, None, 'FOLDER', child['name'])
-
-        # There are only sub-nodes at the project level, no texture nodes,
-        # so we won't have to bother looking for textures.
-        if not node_uuid:
-            return
-
-        directory = cache.cache_directory('thumbnails', project_uuid, node_uuid)
-
-        self.log.debug('Fetching texture thumbnails for node %r to %r', node_uuid, directory)
-        await pillar.fetch_texture_thumbs(node_uuid, 's', directory,
-                                          thumbnail_loading=thumbnail_loading,
-                                          thumbnail_loaded=thumbnail_loaded,
-                                          future=self.signalling_future)
-
-    def update(self):
-        self.log.debug('update()')
-        async_loop.kick_async_loop()
-
-        if not self.async_task:
-            return
-
-        if self.async_task.done():
-            self.status = {'VALID'}
-
-        self.status = {'VALID', 'RUNNING'}
-
-
-class BCloudAssetEngine(AssetEngine):
-    bl_label = "Blender Cloud"
-    bl_version = 1
-
-    def __init__(self):
-        self.log = logging.getLogger('%s.%s' % (__name__, BCloudAssetEngine.__qualname__))
-        self.log.debug('Starting %s asset engine', self.bl_label)
-
-        self.jobs = {}
-        self._next_job_id = 1
-        self.path = pillar.CloudPath('/5672beecc0261b2005ed1a33')
-        self.dirs = []
-        self.sortedfiltered = []
-
-    def reset(self):
-        pass
-
-    def _start_dirlist_job(self, path: pillar.CloudPath, job_id: int = None) -> int:
-        if not job_id:
-            job_id = self._next_job_id
-            self._next_job_id += 1
-
-        self.jobs[job_id] = BCloudAssetEngineDirListJob(job_id, path)
-        self.path = path
-
-        return job_id
-
-    ########## PY-API only ##########
-    # UI header
-    def draw_header(self, layout, context):
-        params = context.space_data.params
-        assert isinstance(params, FileSelectParams)
-        # self.log.debug('draw_header: params=%r', params)
-
-        # can be None when save/reload with a file selector open
-        if params is None:
-            return
-
-        is_lib_browser = params.use_library_browsing
-
-        layout.prop(params, "display_type", expand=True, text="")
-        layout.prop(params, "sort_method", expand=True, text="")
-
-        layout.prop(params, "show_hidden", text="", icon='FILE_HIDDEN')
-        layout.prop(params, "use_filter", text="", icon='FILTER')
-
-        row = layout.row(align=True)
-        row.active = params.use_filter
-
-        if params.filter_glob:
-            # if st.active_operator and hasattr(st.active_operator, "filter_glob"):
-            #    row.prop(params, "filter_glob", text="")
-            row.label(params.filter_glob)
-        else:
-            row.prop(params, "use_filter_blender", text="")
-            row.prop(params, "use_filter_backup", text="")
-            row.prop(params, "use_filter_image", text="")
-            row.prop(params, "use_filter_movie", text="")
-            row.prop(params, "use_filter_script", text="")
-            row.prop(params, "use_filter_font", text="")
-            row.prop(params, "use_filter_sound", text="")
-            row.prop(params, "use_filter_text", text="")
-
-        if is_lib_browser:
-            row.prop(params, "use_filter_blendid", text="")
-            if params.use_filter_blendid:
-                row.separator()
-                row.prop(params, "filter_id_category", text="")
-
-        row.separator()
-        row.prop(params, "filter_search", text="", icon='VIEWZOOM')
-
-    ########## C (RNA) API ##########
-    def status(self, job_id: int) -> set:
-        """Returns either {'VALID'}, {'RUNNING'} or empty set."""
-
-        if job_id:
-            job = self.jobs.get(job_id, None)
-            return job.status if job is not None else set()
-        return {'VALID'}
-
-    def progress(self, job_id: int) -> float:
-        if job_id:
-            job = self.jobs.get(job_id, None)
-            return job.progress if job is not None else 0.0
-        progress = 0.0
-        nbr_jobs = 0
-        for job in self.jobs.values():
-            if 'RUNNING' in job.status:
-                nbr_jobs += 1
-                progress += job.progress
-        return progress / nbr_jobs if nbr_jobs else 0.0
-
-    def kill(self, job_id: int):
-        self.log.debug('kill(%i)', job_id)
-        if not job_id:
-            for job_id in self.jobs:
-                self.kill(job_id)
-            return
-
-        job = self.jobs.get(job_id, None)
-        if job is not None:
-            job.stop()
-
-    def list_dir(self, job_id: int, asset_list: AssetList) -> int:
-        """Extends the 'asset_list' object with asset_list for the current dir.
-
-        :param job_id: Job ID of a currently running job (to investigate
-            progress), or zero (0) to start a new job.
-        :param asset_list: AssetList to store directory asset_list in.
-
-        :returns: the job ID, which is the given job ID or a new job ID if a
-            new job was started.
-        """
-
-        self.log.debug('list_dir(%i), %i entries already loaded', job_id, len(asset_list.entries))
-
-        # TODO: set asset_list.nbr_entries to the total number of entries.
-
-        # job = self.jobs.get(job_id, None)
-        #
-        # asset_list_path = pillar.CloudPath(asset_list.root_path)
-        # if job is not None:
-        #     if not isinstance(job, BCloudAssetEngineDirListJob) or job.path != asset_list_path:
-        #         # We moved to another directory, abort what's going on now and start a new job.
-        #         self.reset()
-        #         if not isinstance(job, BCloudAssetEngineDirListJob):
-        #             self.log.warn('Job %r is not a BCloudAssetEngineDirListJob', job)
-        #         else:
-        #             self.log.warn('Job %r is investigating path %r while we want %r', job,
-        #                           job.path, asset_list_path)
-        #         return self._start_dirlist_job(pillar.CloudPath(asset_list_path))
-        #
-        #     # Just asking for an update
-        #     job.update()
-        #     return job_id
-        #
-        # # Moved to another directory, but we haven't started any job yet.
-        # if self.path != asset_list_path:
-        #     self.reset()
-        #     self.log.info('No job yet, and path changed from %r to %r',
-        #                   self.path, asset_list_path)
-        #     return self._start_dirlist_job(asset_list_path)
-        #
-        # self.log.warn('No job (id=%i), no change in path (%r == %r), nothing to do.', job_id,
-        #               self.path, asset_list_path)
-
-        # Just add a fake entry for shits and giggles.
-        if asset_list.nbr_entries == 0:
-            asset_list.nbr_entries = 1
-
-        # import time
-        # time.sleep(1)
-        # The job has been finished; the asset_list is complete.
-        # return job_id
-        return -1
-
-    def load_pre(self, uuids, asset_list: AssetList) -> bool:
-        self.log.debug("load_pre(%r, %r)", uuids, asset_list)
-        return False
-
-    def sort_filter(self, use_sort: bool, use_filter: bool, params: FileSelectParams,
-                    asset_list: AssetList) -> bool:
-        self.log.debug("sort_filter(%s, %s, %r, %i in %r)", use_sort, use_filter, params,
-                       len(asset_list.entries), asset_list)
-        asset_list.nbr_entries_filtered = asset_list.nbr_entries
-        return False
-
-    def entries_block_get(self, start_index: int, end_index: int, asset_list: AssetList):
-        self.log.debug("entries_block_get(%i, %i, %r)", start_index, end_index, asset_list)
-
-        entry = asset_list.entries.add()
-        entry.name = 'je moeder'
-        entry.description = 'hahaha'
-        entry.type = {'DIR'}
-        entry.relpath = 'relative'
-        entry.uuid = (1, 2, 3, 4)
-
-        variant = entry.variants.add()
-        variant.uuid = (2, 3, 4, 5)
-        variant.name = 'Variant van je moeder'
-        variant.description = 'Variant van je omschrijving'
-        entry.variants.active = variant
-
-        revision = variant.revisions.add()
-        revision.uuid = (3, 4, 5, 6)
-        revision.size = 1024
-        revision.timestamp = time.time()
-        variant.revisions.active = revision
-
-        return True
-
-    def entries_uuid_get(self, uuids: AssetUUIDList, asset_list: AssetList):
-        self.log.debug("entries_uuid_get(%r, %r)", uuids, asset_list)
-
-        for uuid in uuids.uuids:
-            self.entry_from_uuid(asset_list, uuid)
-        return True
-
-    def entry_from_uuid(self, asset_list: AssetList, uuid: AssetUUID):
-        """Adds the ID'd entry to the asset list.
-
-        Alternatively, it sets the UUID's 'is_unknown_engine' or
-        'is_asset_missing' properties.
-        """
-
-        uuid_asset = tuple(uuid.uuid_asset)
-        uuid_variant = tuple(uuid.uuid_variant)
-        uuid_revision = tuple(uuid.uuid_revision)
-
-        entry = asset_list.entries.add()
-        entry.name = 'je moeder'
-        entry.description = 'hahaha'
-        entry.type = {'DIR'}
-        entry.relpath = 'relative'
-        entry.uuid = uuid_asset
-
-        variant = entry.variants.add()
-        variant.uuid = uuid_variant
-        variant.name = 'Variant van je moeder'
-        variant.description = 'Variant van je omschrijving'
-        entry.variants.active = variant
-
-        revision = variant.revisions.add()
-        revision.uuid = uuid_revision
-        revision.size = 1024
-        revision.timestamp = time.time()
-        variant.revisions.active = revision
-
-
-class BCloudPanel:
-    @classmethod
-    def poll(cls, context):
-        space = context.space_data
-        if space and space.type == 'FILE_BROWSER':
-            ae = space.asset_engine
-            if ae and space.asset_engine_type == "AssetEngineAmber":
-                return True
-        return False
-
-
-class BCloud_PT_options(Panel, BCloudPanel):
-    bl_space_type = 'FILE_BROWSER'
-    bl_region_type = 'TOOLS'
-    bl_category = "Asset Engine"
-    bl_label = "Blender Cloud Options"
-
-    def draw(self, context):
-        layout = self.layout
-        space = context.space_data
-        ae = space.asset_engine
-
-        row = layout.row()
-
-
-class BCloud_PT_tags(Panel, BCloudPanel):
-    bl_space_type = 'FILE_BROWSER'
-    bl_region_type = 'TOOLS'
-    bl_category = "Filter"
-    bl_label = "Tags"
-
-    def draw(self, context):
-        ae = context.space_data.asset_engine
-
-        # Note: This is *ultra-primitive*!
-        #       A good UI will most likely need new widget option anyway (template).
-        #       Or maybe just some UIList...
-        # ~ self.layout.props_enum(ae, "tags")
-        # self.layout.template_list("AMBER_UL_tags_filter", "", ae, "tags", ae, "active_tag_index")
-
-
-def register():
-    import sys
-    import doctest
-    (failures, tests) = doctest.testmod(sys.modules[__name__])
-
-    log = logging.getLogger(__name__)
-    if failures:
-        log.warning('There were test failures: %i of %i tests failed.', failures, tests)
-    else:
-        log.debug('All %i tests were successful.', tests)
-
-    bpy.utils.register_class(BCloudAssetEngine)
-    bpy.utils.register_class(BCloud_PT_options)
-    bpy.utils.register_class(BCloud_PT_tags)
-
-
-def unregister():
-    bpy.utils.register_class(BCloud_PT_tags)
-    bpy.utils.register_class(BCloud_PT_options)
-    bpy.utils.register_class(BCloudAssetEngine)

blender_cloud/async_loop.py

@@ -1,9 +1,28 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
 """Manages the asyncio loop."""
 
 import asyncio
 import traceback
 import concurrent.futures
 import logging
+import gc
 
 import bpy
 
@@ -20,9 +39,18 @@ def setup_asyncio_executor():
     calls that could be performed in parallel are queued, and thus we can
     reliably cancel them.
     """
+    import sys
 
     executor = concurrent.futures.ThreadPoolExecutor()
-    loop = asyncio.get_event_loop()
+
+    if sys.platform == 'win32':
+        # On Windows, the default event loop is SelectorEventLoop, which does
+        # not support subprocesses. ProactorEventLoop should be used instead.
+        # Source: https://docs.python.org/3/library/asyncio-subprocess.html
+        loop = asyncio.ProactorEventLoop()
+        asyncio.set_event_loop(loop)
+    else:
+        loop = asyncio.get_event_loop()
     loop.set_default_executor(executor)
     # loop.set_debug(True)
 
@@ -53,6 +81,9 @@ def kick_async_loop(*args) -> bool:
                   len(all_tasks))
         stop_after_this_kick = True
 
+        # Clean up circular references between tasks.
+        gc.collect()
+
         for task_idx, task in enumerate(all_tasks):
             if not task.done():
                 continue
@@ -68,6 +99,9 @@ def kick_async_loop(*args) -> bool:
                 print('{}: resulted in exception'.format(task))
                 traceback.print_exc()
 
+            # for ref in gc.get_referrers(task):
+            #     log.debug('      - referred by %s', ref)
+
     loop.stop()
     loop.run_forever()
 
@@ -80,6 +114,15 @@ def ensure_async_loop():
     log.debug('Result of starting modal operator is %r', result)
 
 
+def erase_async_loop():
+    global _loop_kicking_operator_running
+
+    log.debug('Erasing async loop')
+
+    loop = asyncio.get_event_loop()
+    loop.stop()
+
+
 class AsyncLoopModalOperator(bpy.types.Operator):
     bl_idname = 'asyncio.loop'
     bl_label = 'Runs the asyncio main loop'
@@ -87,6 +130,14 @@ class AsyncLoopModalOperator(bpy.types.Operator):
     timer = None
     log = logging.getLogger(__name__ + '.AsyncLoopModalOperator')
 
+    def __del__(self):
+        global _loop_kicking_operator_running
+
+        # This can be required when the operator is running while Blender
+        # (re)loads a file. The operator then doesn't get the chance to
+        # finish the async tasks, hence stop_after_this_kick is never True.
+        _loop_kicking_operator_running = False
+
     def execute(self, context):
         return self.invoke(context, None)
 
@@ -108,6 +159,12 @@ class AsyncLoopModalOperator(bpy.types.Operator):
     def modal(self, context, event):
         global _loop_kicking_operator_running
 
+        # If _loop_kicking_operator_running is set to False, someone called
+        # erase_async_loop(). This is a signal that we really should stop
+        # running.
+        if not _loop_kicking_operator_running:
+            return {'FINISHED'}
+
         if event.type != 'TIMER':
             return {'PASS_THROUGH'}
 
@@ -123,6 +180,108 @@ class AsyncLoopModalOperator(bpy.types.Operator):
         return {'RUNNING_MODAL'}
 
 
+# noinspection PyAttributeOutsideInit
+class AsyncModalOperatorMixin:
+    async_task = None  # asyncio task for fetching thumbnails
+    signalling_future = None  # asyncio future for signalling that we want to cancel everything.
+    log = logging.getLogger('%s.AsyncModalOperatorMixin' % __name__)
+
+    _state = 'INITIALIZING'
+    stop_upon_exception = False
+
+    def invoke(self, context, event):
+        context.window_manager.modal_handler_add(self)
+        self.timer = context.window_manager.event_timer_add(1 / 15, context.window)
+
+        self.log.info('Starting')
+        self._new_async_task(self.async_execute(context))
+
+        return {'RUNNING_MODAL'}
+
+    async def async_execute(self, context):
+        """Entry point of the asynchronous operator.
+
+        Implement in a subclass.
+        """
+        return
+
+    def quit(self):
+        """Signals the state machine to stop this operator from running."""
+        self._state = 'QUIT'
+
+    def execute(self, context):
+        return self.invoke(context, None)
+
+    def modal(self, context, event):
+        task = self.async_task
+
+        if self._state != 'EXCEPTION' and task and task.done() and not task.cancelled():
+            ex = task.exception()
+            if ex is not None:
+                self._state = 'EXCEPTION'
+                self.log.error('Exception while running task: %s', ex)
+                if self.stop_upon_exception:
+                    self.quit()
+                    self._finish(context)
+                    return {'FINISHED'}
+
+                return {'RUNNING_MODAL'}
+
+        if self._state == 'QUIT':
+            self._finish(context)
+            return {'FINISHED'}
+
+        return {'PASS_THROUGH'}
+
+    def _finish(self, context):
+        self._stop_async_task()
+        context.window_manager.event_timer_remove(self.timer)
+
+    def _new_async_task(self, async_task: asyncio.coroutine, future: asyncio.Future = None):
+        """Stops the currently running async task, and starts another one."""
+
+        self.log.debug('Setting up a new task %r, so any existing task must be stopped', async_task)
+        self._stop_async_task()
+
+        # Download the previews asynchronously.
+        self.signalling_future = future or asyncio.Future()
+        self.async_task = asyncio.ensure_future(async_task)
+        self.log.debug('Created new task %r', self.async_task)
+
+        # Start the async manager so everything happens.
+        ensure_async_loop()
+
+    def _stop_async_task(self):
+        self.log.debug('Stopping async task')
+        if self.async_task is None:
+            self.log.debug('No async task, trivially stopped')
+            return
+
+        # Signal that we want to stop.
+        self.async_task.cancel()
+        if not self.signalling_future.done():
+            self.log.info("Signalling that we want to cancel anything that's running.")
+            self.signalling_future.cancel()
+
+        # Wait until the asynchronous task is done.
+        if not self.async_task.done():
+            self.log.info("blocking until async task is done.")
+            loop = asyncio.get_event_loop()
+            try:
+                loop.run_until_complete(self.async_task)
+            except asyncio.CancelledError:
+                self.log.info('Asynchronous task was cancelled')
+                return
+
+        # noinspection PyBroadException
+        try:
+            self.async_task.result()  # This re-raises any exception of the task.
+        except asyncio.CancelledError:
+            self.log.info('Asynchronous task was cancelled')
+        except Exception:
+            self.log.exception("Exception from asynchronous task")
+
+
 def register():
     bpy.utils.register_class(AsyncLoopModalOperator)
 

blender_cloud/attract/__init__.py

@@ -0,0 +1,960 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
+# <pep8 compliant>
+
+# Old info, kept here for reference, so that we can merge wiki pages,
+# descriptions, etc.
+#
+# bl_info = {
+#     "name": "Attract",
+#     "author": "Francesco Siddi, Inês Almeida, Antony Riakiotakis",
+#     "version": (0, 2, 0),
+#     "blender": (2, 76, 0),
+#     "location": "Video Sequence Editor",
+#     "description":
+#         "Blender integration with the Attract task tracking service"
+#         ". *requires the Blender ID add-on",
+#     "wiki_url": "http://wiki.blender.org/index.php/Extensions:2.6/Py/"
+#                 "Scripts/Workflow/Attract",
+#     "category": "Workflow",
+#     "support": "TESTING"
+# }
+
+import contextlib
+import functools
+import logging
+
+if "bpy" in locals():
+    import importlib
+
+    draw = importlib.reload(draw)
+    pillar = importlib.reload(pillar)
+    async_loop = importlib.reload(async_loop)
+else:
+    from . import draw
+    from .. import pillar, async_loop
+
+import bpy
+import pillarsdk
+from pillarsdk.nodes import Node
+from pillarsdk.projects import Project
+from pillarsdk import exceptions as sdk_exceptions
+
+from bpy.types import Operator, Panel, AddonPreferences
+
+log = logging.getLogger(__name__)
+
+
+def active_strip(context):
+    try:
+        return context.scene.sequence_editor.active_strip
+    except AttributeError:
+        return None
+
+
+def selected_shots(context):
+    """Generator, yields selected strips if they are Attract shots."""
+    selected_sequences = context.selected_sequences
+
+    if selected_sequences is None:
+        return
+
+    for strip in selected_sequences:
+        atc_object_id = getattr(strip, 'atc_object_id')
+        if not atc_object_id:
+            continue
+
+        yield strip
+
+
+def all_shots(context):
+    """Generator, yields all strips if they are Attract shots."""
+    sequence_editor = context.scene.sequence_editor
+
+    if sequence_editor is None:
+        # we should throw an exception, but at least this change prevents an error
+        return []
+
+    for strip in context.scene.sequence_editor.sequences_all:
+        atc_object_id = getattr(strip, 'atc_object_id')
+        if not atc_object_id:
+            continue
+
+        yield strip
+
+
+def shown_strips(context):
+    """Returns the strips from the current meta-strip-stack, or top-level strips.
+
+    What is returned depends on what the user is currently editing.
+    """
+
+    if context.scene.sequence_editor.meta_stack:
+        return context.scene.sequence_editor.meta_stack[-1].sequences
+
+    return context.scene.sequence_editor.sequences
+
+
+def remove_atc_props(strip):
+    """Resets the attract custom properties assigned to a VSE strip"""
+
+    strip.atc_name = ""
+    strip.atc_description = ""
+    strip.atc_object_id = ""
+    strip.atc_is_synced = False
+
+
+def shot_id_use(strips):
+    """Returns a mapping from shot Object ID to a list of strips that use it."""
+
+    import collections
+
+    # Count the number of uses per Object ID, so that we can highlight double use.
+    ids_in_use = collections.defaultdict(list)
+    for strip in strips:
+        if not getattr(strip, 'atc_is_synced', False):
+            continue
+
+        ids_in_use[strip.atc_object_id].append(strip)
+
+    return ids_in_use
+
+
+def compute_strip_conflicts(scene):
+    """Sets the strip property atc_object_id_conflict for each strip."""
+
+    if not scene or not scene.sequence_editor or not scene.sequence_editor.sequences_all:
+        return
+
+    tag_redraw = False
+    ids_in_use = shot_id_use(scene.sequence_editor.sequences_all)
+    for strips in ids_in_use.values():
+        is_conflict = len(strips) > 1
+        for strip in strips:
+            if strip.atc_object_id_conflict != is_conflict:
+                tag_redraw = True
+            strip.atc_object_id_conflict = is_conflict
+
+    if tag_redraw:
+        draw.tag_redraw_all_sequencer_editors()
+    return ids_in_use
+
+
+@bpy.app.handlers.persistent
+def scene_update_post_handler(scene):
+    compute_strip_conflicts(scene)
+
+
+class ToolsPanel(Panel):
+    bl_label = 'Attract'
+    bl_space_type = 'SEQUENCE_EDITOR'
+    bl_region_type = 'UI'
+
+    def draw_header(self, context):
+        strip = active_strip(context)
+        if strip and strip.atc_object_id:
+            self.layout.prop(strip, 'atc_is_synced', text='')
+
+    def draw(self, context):
+        strip = active_strip(context)
+        layout = self.layout
+        strip_types = {'MOVIE', 'IMAGE', 'META'}
+
+        selshots = list(selected_shots(context))
+        if strip and strip.type in strip_types and strip.atc_object_id:
+            if len(selshots) > 1:
+                noun = '%i Shots' % len(selshots)
+            else:
+                noun = 'This Shot'
+
+            if strip.atc_object_id_conflict:
+                warnbox = layout.box()
+                warnbox.alert = True
+                warnbox.label('Warning: This shot is linked to multiple sequencer strips.',
+                              icon='ERROR')
+
+            layout.prop(strip, 'atc_name', text='Name')
+            layout.prop(strip, 'atc_status', text='Status')
+
+            # Create a special sub-layout for read-only properties.
+            ro_sub = layout.column(align=True)
+            ro_sub.enabled = False
+            ro_sub.prop(strip, 'atc_description', text='Description')
+            ro_sub.prop(strip, 'atc_notes', text='Notes')
+
+            if strip.atc_is_synced:
+                sub = layout.column(align=True)
+                row = sub.row(align=True)
+                if bpy.ops.attract.submit_selected.poll():
+                    row.operator('attract.submit_selected',
+                                 text='Submit %s' % noun,
+                                 icon='TRIA_UP')
+                else:
+                    row.operator(ATTRACT_OT_submit_all.bl_idname)
+                row.operator(AttractShotFetchUpdate.bl_idname,
+                             text='', icon='FILE_REFRESH')
+                row.operator(ATTRACT_OT_shot_open_in_browser.bl_idname,
+                             text='', icon='WORLD')
+                row.operator(ATTRACT_OT_copy_id_to_clipboard.bl_idname,
+                             text='', icon='COPYDOWN')
+                sub.operator(ATTRACT_OT_make_shot_thumbnail.bl_idname,
+                             text='Render Thumbnail for %s' % noun,
+                             icon='RENDER_STILL')
+
+                # Group more dangerous operations.
+                dangerous_sub = layout.split(0.6, align=True)
+                dangerous_sub.operator('attract.strip_unlink',
+                                       text='Unlink %s' % noun,
+                                       icon='PANEL_CLOSE')
+                dangerous_sub.operator(AttractShotDelete.bl_idname,
+                                       text='Delete %s' % noun,
+                                       icon='CANCEL')
+
+        elif context.selected_sequences:
+            if len(context.selected_sequences) > 1:
+                noun = 'Selected Strips'
+            else:
+                noun = 'This Strip'
+            layout.operator(AttractShotSubmitSelected.bl_idname,
+                            text='Submit %s as New Shot' % noun)
+            layout.operator('attract.shot_relink')
+        else:
+            layout.operator(ATTRACT_OT_submit_all.bl_idname)
+
+
+class AttractOperatorMixin:
+    """Mix-in class for all Attract operators."""
+
+    def _project_needs_setup_error(self):
+        self.report({'ERROR'}, 'Your Blender Cloud project is not set up for Attract.')
+        return {'CANCELLED'}
+
+    @functools.lru_cache()
+    def find_project(self, project_uuid: str) -> Project:
+        """Finds a single project.
+
+        Caches the result in memory to prevent more than one call to Pillar.
+        """
+
+        from .. import pillar
+
+        project = pillar.sync_call(Project.find_one, {'where': {'_id': project_uuid}})
+        return project
+
+    def find_node_type(self, node_type_name: str) -> dict:
+        from .. import pillar, blender
+
+        prefs = blender.preferences()
+        project = self.find_project(prefs.attract_project.project)
+
+        # FIXME: Eve doesn't seem to handle the $elemMatch projection properly,
+        # even though it works fine in MongoDB itself. As a result, we have to
+        # search for the node type.
+        node_type_list = project['node_types']
+        node_type = next((nt for nt in node_type_list if nt['name'] == node_type_name), None)
+
+        if not node_type:
+            return self._project_needs_setup_error()
+
+        return node_type
+
+    def submit_new_strip(self, strip):
+        from .. import pillar, blender
+
+        # Define the shot properties
+        user_uuid = pillar.pillar_user_uuid()
+        if not user_uuid:
+            self.report({'ERROR'}, 'Your Blender Cloud user ID is not known, '
+                                   'update your credentials.')
+            return {'CANCELLED'}
+
+        prop = {'name': strip.name,
+                'description': '',
+                'properties': {'status': 'todo',
+                               'notes': '',
+                               'used_in_edit': True,
+                               'trim_start_in_frames': strip.frame_offset_start,
+                               'trim_end_in_frames': strip.frame_offset_end,
+                               'duration_in_edit_in_frames': strip.frame_final_duration,
+                               'cut_in_timeline_in_frames': strip.frame_final_start},
+                'order': 0,
+                'node_type': 'attract_shot',
+                'project': blender.preferences().attract_project.project,
+                'user': user_uuid}
+
+        # Create a Node item with the attract API
+        node = Node(prop)
+        post = pillar.sync_call(node.create)
+
+        # Populate the strip with the freshly generated ObjectID and info
+        if not post:
+            self.report({'ERROR'}, 'Error creating node! Check the console for now.')
+            return {'CANCELLED'}
+
+        strip.atc_object_id = node['_id']
+        strip.atc_is_synced = True
+        strip.atc_name = node['name']
+        strip.atc_description = node['description']
+        strip.atc_notes = node['properties']['notes']
+        strip.atc_status = node['properties']['status']
+
+        draw.tag_redraw_all_sequencer_editors()
+
+    def submit_update(self, strip):
+        import pillarsdk
+        from .. import pillar
+
+        patch = {
+            'op': 'from-blender',
+            '$set': {
+                'name': strip.atc_name,
+                'properties.trim_start_in_frames': strip.frame_offset_start,
+                'properties.trim_end_in_frames': strip.frame_offset_end,
+                'properties.duration_in_edit_in_frames': strip.frame_final_duration,
+                'properties.cut_in_timeline_in_frames': strip.frame_final_start,
+                'properties.status': strip.atc_status,
+                'properties.used_in_edit': True,
+            }
+        }
+
+        node = pillarsdk.Node({'_id': strip.atc_object_id})
+        result = pillar.sync_call(node.patch, patch)
+        log.info('PATCH result: %s', result)
+
+    def relink(self, strip, atc_object_id, *, refresh=False):
+        from .. import pillar
+
+        # The node may have been deleted, so we need to send a 'relink' before we try
+        # to fetch the node itself.
+        node = Node({'_id': atc_object_id})
+        pillar.sync_call(node.patch, {'op': 'relink'})
+
+        try:
+            node = pillar.sync_call(Node.find, atc_object_id, caching=False)
+        except (sdk_exceptions.ResourceNotFound, sdk_exceptions.MethodNotAllowed):
+            verb = 'refresh' if refresh else 'relink'
+            self.report({'ERROR'}, 'Shot %r not found on the Attract server, unable to %s.'
+                        % (atc_object_id, verb))
+            strip.atc_is_synced = False
+            return {'CANCELLED'}
+
+        strip.atc_is_synced = True
+        if not refresh:
+            strip.atc_name = node.name
+            strip.atc_object_id = node['_id']
+
+        # We do NOT set the position/cuts of the shot, that always has to come from Blender.
+        strip.atc_status = node.properties.status
+        strip.atc_notes = node.properties.notes or ''
+        strip.atc_description = node.description or ''
+        draw.tag_redraw_all_sequencer_editors()
+
+
+class AttractShotFetchUpdate(AttractOperatorMixin, Operator):
+    bl_idname = "attract.shot_fetch_update"
+    bl_label = "Fetch Update From Attract"
+    bl_description = 'Update status, description & notes from Attract'
+
+    @classmethod
+    def poll(cls, context):
+        return any(selected_shots(context))
+
+    def execute(self, context):
+        for strip in selected_shots(context):
+            status = self.relink(strip, strip.atc_object_id, refresh=True)
+            # We don't abort when one strip fails. All selected shots should be
+            # refreshed, even if one can't be found (for example).
+            if not isinstance(status, set):
+                self.report({'INFO'}, "Shot {0} refreshed".format(strip.atc_name))
+        return {'FINISHED'}
+
+
+class AttractShotRelink(AttractShotFetchUpdate):
+    bl_idname = "attract.shot_relink"
+    bl_label = "Relink With Attract"
+
+    strip_atc_object_id = bpy.props.StringProperty()
+
+    @classmethod
+    def poll(cls, context):
+        strip = active_strip(context)
+        return strip is not None and not getattr(strip, 'atc_object_id', None)
+
+    def execute(self, context):
+        strip = active_strip(context)
+
+        status = self.relink(strip, self.strip_atc_object_id)
+        if isinstance(status, set):
+            return status
+
+        strip.atc_object_id = self.strip_atc_object_id
+        self.report({'INFO'}, "Shot {0} relinked".format(strip.atc_name))
+
+        return {'FINISHED'}
+
+    def invoke(self, context, event):
+        maybe_id = context.window_manager.clipboard
+        if len(maybe_id) == 24:
+            try:
+                int(maybe_id, 16)
+            except ValueError:
+                pass
+            else:
+                self.strip_atc_object_id = maybe_id
+
+        return context.window_manager.invoke_props_dialog(self)
+
+    def draw(self, context):
+        layout = self.layout
+        col = layout.column()
+        col.prop(self, 'strip_atc_object_id', text='Shot ID')
+
+
+class ATTRACT_OT_shot_open_in_browser(AttractOperatorMixin, Operator):
+    bl_idname = 'attract.shot_open_in_browser'
+    bl_label = 'Open in Browser'
+    bl_description = 'Opens a webbrowser to show the shot on Attract'
+
+    @classmethod
+    def poll(cls, context):
+        return bool(context.selected_sequences and active_strip(context))
+
+    def execute(self, context):
+        from ..blender import PILLAR_WEB_SERVER_URL
+        import webbrowser
+        import urllib.parse
+
+        strip = active_strip(context)
+
+        url = urllib.parse.urljoin(PILLAR_WEB_SERVER_URL,
+                                   'nodes/%s/redir' % strip.atc_object_id)
+        webbrowser.open_new_tab(url)
+        self.report({'INFO'}, 'Opened a browser at %s' % url)
+
+        return {'FINISHED'}
+
+
+class AttractShotDelete(AttractOperatorMixin, Operator):
+    bl_idname = 'attract.shot_delete'
+    bl_label = 'Delete Shot'
+    bl_description = 'Remove this shot from Attract'
+
+    confirm = bpy.props.BoolProperty(name='confirm')
+
+    @classmethod
+    def poll(cls, context):
+        return bool(context.selected_sequences)
+
+    def execute(self, context):
+        from .. import pillar
+
+        if not self.confirm:
+            self.report({'WARNING'}, 'Delete aborted.')
+            return {'CANCELLED'}
+
+        removed = kept = 0
+        for strip in selected_shots(context):
+            node = pillar.sync_call(Node.find, strip.atc_object_id)
+            if not pillar.sync_call(node.delete):
+                self.report({'ERROR'}, 'Unable to delete shot %s on Attract.' % strip.atc_name)
+                kept += 1
+                continue
+            remove_atc_props(strip)
+            removed += 1
+
+        if kept:
+            self.report({'ERROR'}, 'Removed %i shots, but was unable to remove %i' %
+                        (removed, kept))
+        else:
+            self.report({'INFO'}, 'Removed all %i shots from Attract' % removed)
+
+        draw.tag_redraw_all_sequencer_editors()
+        return {'FINISHED'}
+
+    def invoke(self, context, event):
+        self.confirm = False
+        return context.window_manager.invoke_props_dialog(self)
+
+    def draw(self, context):
+        layout = self.layout
+        col = layout.column()
+
+        selshots = list(selected_shots(context))
+        if len(selshots) > 1:
+            noun = '%i shots' % len(selshots)
+        else:
+            noun = 'this shot'
+
+        col.prop(self, 'confirm', text="I hereby confirm: delete %s from The Edit." % noun)
+
+
+class AttractStripUnlink(AttractOperatorMixin, Operator):
+    bl_idname = 'attract.strip_unlink'
+    bl_label = 'Unlink Shot From This Strip'
+    bl_description = 'Remove Attract props from the selected strip(s)'
+
+    @classmethod
+    def poll(cls, context):
+        return bool(context.selected_sequences)
+
+    def execute(self, context):
+        unlinked_ids = set()
+
+        # First remove the Attract properties from the strips.
+        for strip in context.selected_sequences:
+            atc_object_id = getattr(strip, 'atc_object_id')
+            remove_atc_props(strip)
+
+            if atc_object_id:
+                unlinked_ids.add(atc_object_id)
+
+        # For all Object IDs that are no longer in use in the edit, let Attract know.
+        # This should be done with care, as the shot could have been attached to multiple
+        # strips.
+        id_to_shots = compute_strip_conflicts(context.scene)
+        for oid in unlinked_ids:
+            if len(id_to_shots[oid]):
+                # Still in use
+                continue
+
+            node = Node({'_id': oid})
+            pillar.sync_call(node.patch, {'op': 'unlink'})
+
+        if len(unlinked_ids) == 1:
+            shot_id = unlinked_ids.pop()
+            context.window_manager.clipboard = shot_id
+            self.report({'INFO'}, 'Copied unlinked shot ID %s to clipboard' % shot_id)
+        else:
+            self.report({'INFO'}, '%i shots have been marked as Unused.' % len(unlinked_ids))
+
+        draw.tag_redraw_all_sequencer_editors()
+        return {'FINISHED'}
+
+
+class AttractShotSubmitSelected(AttractOperatorMixin, Operator):
+    bl_idname = 'attract.submit_selected'
+    bl_label = 'Submit All Selected'
+    bl_description = 'Submits all selected strips to Attract'
+
+    @classmethod
+    def poll(cls, context):
+        return bool(context.selected_sequences)
+
+    def execute(self, context):
+        # Check that the project is set up for Attract.
+        maybe_error = self.find_node_type('attract_shot')
+        if isinstance(maybe_error, set):
+            return maybe_error
+
+        for strip in context.selected_sequences:
+            status = self.submit(strip)
+            if isinstance(status, set):
+                return status
+
+        self.report({'INFO'}, 'All selected strips sent to Attract.')
+
+        return {'FINISHED'}
+
+    def submit(self, strip):
+        atc_object_id = getattr(strip, 'atc_object_id', None)
+
+        # Submit as new?
+        if not atc_object_id:
+            return self.submit_new_strip(strip)
+
+        # Or just save to Attract.
+        return self.submit_update(strip)
+
+
+class ATTRACT_OT_submit_all(AttractOperatorMixin, Operator):
+    bl_idname = 'attract.submit_all'
+    bl_label = 'Submit All Shots to Attract'
+    bl_description = 'Updates Attract with the current state of the edit'
+
+    def execute(self, context):
+        # Check that the project is set up for Attract.
+        maybe_error = self.find_node_type('attract_shot')
+        if isinstance(maybe_error, set):
+            return maybe_error
+
+        for strip in all_shots(context):
+            status = self.submit_update(strip)
+            if isinstance(status, set):
+                return status
+
+        self.report({'INFO'}, 'All strips re-sent to Attract.')
+
+        return {'FINISHED'}
+
+
+class ATTRACT_OT_open_meta_blendfile(AttractOperatorMixin, Operator):
+    bl_idname = 'attract.open_meta_blendfile'
+    bl_label = 'Open Blendfile'
+    bl_description = 'Open Blendfile from movie strip metadata'
+
+    @classmethod
+    def poll(cls, context):
+        return bool(any(cls.filename_from_metadata(s) for s in context.selected_sequences))
+
+    @staticmethod
+    def filename_from_metadata(strip):
+        """Returns the blendfile name from the strip metadata, or None."""
+
+        # Metadata is a dict like:
+        # meta = {'END_FRAME': '88',
+        #         'BLEND_FILE': 'metadata-test.blend',
+        #         'SCENE': 'SüperSčene',
+        #         'FRAME_STEP': '1',
+        #         'START_FRAME': '32'}
+
+        meta = strip.get('metadata', None)
+        if not meta:
+            return None
+
+        return meta.get('BLEND_FILE', None) or None
+
+    def execute(self, context):
+        for strip in context.selected_sequences:
+            meta = strip.get('metadata', None)
+            if not meta:
+                continue
+
+            fname = meta.get('BLEND_FILE', None)
+            if not fname: continue
+
+            scene = meta.get('SCENE', None)
+            self.open_in_new_blender(fname, scene)
+
+        return {'FINISHED'}
+
+    def open_in_new_blender(self, fname, scene):
+        """
+        :type fname: str
+        :type scene: str
+        """
+        import subprocess
+        import sys
+
+        cmd = [
+            bpy.app.binary_path,
+            str(fname),
+        ]
+
+        cmd[1:1] = [v for v in sys.argv if v.startswith('--enable-')]
+
+        if scene:
+            cmd.extend(['--python-expr',
+                        'import bpy; bpy.context.screen.scene = bpy.data.scenes["%s"]' % scene])
+            cmd.extend(['--scene', scene])
+
+        subprocess.Popen(cmd)
+
+
+class ATTRACT_OT_make_shot_thumbnail(AttractOperatorMixin,
+                                     async_loop.AsyncModalOperatorMixin,
+                                     Operator):
+    bl_idname = 'attract.make_shot_thumbnail'
+    bl_label = 'Render Shot Thumbnail'
+    bl_description = 'Renders the current frame, and uploads it as thumbnail for the shot'
+
+    stop_upon_exception = True
+
+    @classmethod
+    def poll(cls, context):
+        return bool(context.selected_sequences)
+
+    @contextlib.contextmanager
+    def thumbnail_render_settings(self, context, thumbnail_width=512):
+        # Remember current settings so we can restore them later.
+        orig_res_x = context.scene.render.resolution_x
+        orig_res_y = context.scene.render.resolution_y
+        orig_percentage = context.scene.render.resolution_percentage
+        orig_file_format = context.scene.render.image_settings.file_format
+        orig_quality = context.scene.render.image_settings.quality
+
+        try:
+            # Update the render size to something thumbnaily.
+            factor = orig_res_y / orig_res_x
+            context.scene.render.resolution_x = thumbnail_width
+            context.scene.render.resolution_y = round(thumbnail_width * factor)
+            context.scene.render.resolution_percentage = 100
+            context.scene.render.image_settings.file_format = 'JPEG'
+            context.scene.render.image_settings.quality = 85
+
+            yield
+        finally:
+            # Return the render settings to normal.
+            context.scene.render.resolution_x = orig_res_x
+            context.scene.render.resolution_y = orig_res_y
+            context.scene.render.resolution_percentage = orig_percentage
+            context.scene.render.image_settings.file_format = orig_file_format
+            context.scene.render.image_settings.quality = orig_quality
+
+    @contextlib.contextmanager
+    def temporary_current_frame(self, context):
+        """Allows the context to set the scene current frame, restores it on exit.
+
+        Yields the initial current frame, so it can be used for reference in the context.
+        """
+        current_frame = context.scene.frame_current
+        try:
+            yield current_frame
+        finally:
+            context.scene.frame_current = current_frame
+
+    async def async_execute(self, context):
+        nr_of_strips = len(context.selected_sequences)
+        do_multishot = nr_of_strips > 1
+
+        with self.temporary_current_frame(context) as original_curframe:
+            # The multishot and singleshot branches do pretty much the same thing,
+            # but report differently to the user.
+            if do_multishot:
+                context.window_manager.progress_begin(0, nr_of_strips)
+                try:
+                    self.report({'INFO'}, 'Rendering thumbnails for %i selected shots.' %
+                                nr_of_strips)
+
+                    strips = sorted(context.selected_sequences, key=self.by_frame)
+                    for idx, strip in enumerate(strips):
+                        context.window_manager.progress_update(idx)
+
+                        # Pick the middle frame, except for the strip the original current frame
+                        # marker was over.
+                        if not self.strip_contains(strip, original_curframe):
+                            self.set_middle_frame(context, strip)
+                        else:
+                            context.scene.frame_set(original_curframe)
+                        await self.thumbnail_strip(context, strip)
+
+                        if self._state == 'QUIT':
+                            return
+                    context.window_manager.progress_update(nr_of_strips)
+                finally:
+                    context.window_manager.progress_end()
+
+            else:
+                strip = active_strip(context)
+                if not self.strip_contains(strip, original_curframe):
+                    self.report({'WARNING'}, 'Rendering middle frame as thumbnail for active shot.')
+                    self.set_middle_frame(context, strip)
+                else:
+                    self.report({'INFO'}, 'Rendering current frame as thumbnail for active shot.')
+
+                context.window_manager.progress_begin(0, 1)
+                context.window_manager.progress_update(0)
+                try:
+                    await self.thumbnail_strip(context, strip)
+                finally:
+                    context.window_manager.progress_update(1)
+                    context.window_manager.progress_end()
+
+                if self._state == 'QUIT':
+                    return
+
+        self.report({'INFO'}, 'Thumbnail uploaded to Attract')
+        self.quit()
+
+    @staticmethod
+    def strip_contains(strip, framenr: int) -> bool:
+        """Returns True iff the strip covers the given frame number"""
+        return strip.frame_final_start <= framenr <= strip.frame_final_end
+
+    @staticmethod
+    def set_middle_frame(context, strip):
+        """Sets the current frame to the middle frame of the strip."""
+
+        middle = round((strip.frame_final_start + strip.frame_final_end) / 2)
+        context.scene.frame_set(middle)
+
+    @staticmethod
+    def by_frame(sequence_strip) -> int:
+        """Returns the start frame number of the sequence strip.
+
+        This can be used for sorting strips by time.
+        """
+
+        return sequence_strip.frame_final_start
+
+    async def thumbnail_strip(self, context, strip):
+        atc_object_id = getattr(strip, 'atc_object_id', None)
+        if not atc_object_id:
+            self.report({'ERROR'}, 'Strip %s not set up for Attract' % strip.name)
+            self.quit()
+            return
+
+        with self.thumbnail_render_settings(context):
+            bpy.ops.render.render()
+        file_id = await self.upload_via_tempdir(bpy.data.images['Render Result'],
+                                                'attract_shot_thumbnail.jpg')
+
+        if file_id is None:
+            self.quit()
+            return
+
+        # Update the shot to include this file as the picture.
+        node = pillarsdk.Node({'_id': atc_object_id})
+        await pillar.pillar_call(
+            node.patch,
+            {
+                'op': 'from-blender',
+                '$set': {
+                    'picture': file_id,
+                }
+            })
+
+    async def upload_via_tempdir(self, datablock, filename_on_cloud) -> pillarsdk.Node:
+        """Saves the datablock to file, and uploads it to the cloud.
+
+        Saving is done to a temporary directory, which is removed afterwards.
+
+        Returns the node.
+        """
+        import tempfile
+        import os.path
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            filepath = os.path.join(tmpdir, filename_on_cloud)
+            self.log.debug('Saving %s to %s', datablock, filepath)
+            datablock.save_render(filepath)
+            return await self.upload_file(filepath)
+
+    async def upload_file(self, filename: str, fileobj=None):
+        """Uploads a file to the cloud, attached to the image sharing node.
+
+        Returns the node.
+        """
+        from .. import blender
+
+        prefs = blender.preferences()
+        project = self.find_project(prefs.attract_project.project)
+
+        self.log.info('Uploading file %s', filename)
+        resp = await pillar.pillar_call(
+            pillarsdk.File.upload_to_project,
+            project['_id'],
+            'image/jpeg',
+            filename,
+            fileobj=fileobj)
+
+        self.log.debug('Returned data: %s', resp)
+        try:
+            file_id = resp['file_id']
+        except KeyError:
+            self.log.error('Upload did not succeed, response: %s', resp)
+            self.report({'ERROR'}, 'Unable to upload thumbnail to Attract: %s' % resp)
+            return None
+
+        self.log.info('Created file %s', file_id)
+        self.report({'INFO'}, 'File succesfully uploaded to the cloud!')
+
+        return file_id
+
+
+class ATTRACT_OT_copy_id_to_clipboard(AttractOperatorMixin, Operator):
+    bl_idname = 'attract.copy_id_to_clipboard'
+    bl_label = 'Copy shot ID to clipboard'
+
+    @classmethod
+    def poll(cls, context):
+        return bool(context.selected_sequences and active_strip(context))
+
+    def execute(self, context):
+        strip = active_strip(context)
+
+        context.window_manager.clipboard = strip.atc_object_id
+        self.report({'INFO'}, 'Shot ID %s copied to clipboard' % strip.atc_object_id)
+
+        return {'FINISHED'}
+
+
+def draw_strip_movie_meta(self, context):
+    strip = active_strip(context)
+    if not strip:
+        return
+
+    meta = strip.get('metadata', None)
+    if not meta:
+        return None
+
+    box = self.layout.column(align=True)
+    row = box.row(align=True)
+    fname = meta.get('BLEND_FILE', None) or None
+    if fname:
+        row.label('Original Blendfile: %s' % fname)
+        row.operator(ATTRACT_OT_open_meta_blendfile.bl_idname,
+                     text='', icon='FILE_BLEND')
+    sfra = meta.get('START_FRAME', '?')
+    efra = meta.get('END_FRAME', '?')
+    box.label('Original Frame Range: %s-%s' % (sfra, efra))
+
+
+def register():
+    bpy.types.Sequence.atc_is_synced = bpy.props.BoolProperty(name="Is Synced")
+    bpy.types.Sequence.atc_object_id = bpy.props.StringProperty(name="Attract Object ID")
+    bpy.types.Sequence.atc_object_id_conflict = bpy.props.BoolProperty(
+        name='Object ID Conflict',
+        description='Attract Object ID used multiple times',
+        default=False)
+    bpy.types.Sequence.atc_name = bpy.props.StringProperty(name="Shot Name")
+    bpy.types.Sequence.atc_description = bpy.props.StringProperty(name="Shot Description")
+    bpy.types.Sequence.atc_notes = bpy.props.StringProperty(name="Shot Notes")
+
+    # TODO: get this from the project's node type definition.
+    bpy.types.Sequence.atc_status = bpy.props.EnumProperty(
+        items=[
+            ('on_hold', 'On Hold', 'The shot is on hold'),
+            ('todo', 'Todo', 'Waiting'),
+            ('in_progress', 'In Progress', 'The show has been assigned'),
+            ('review', 'Review', ''),
+            ('final', 'Final', ''),
+        ],
+        name="Status")
+    bpy.types.Sequence.atc_order = bpy.props.IntProperty(name="Order")
+
+    bpy.types.SEQUENCER_PT_edit.append(draw_strip_movie_meta)
+
+    bpy.utils.register_class(ToolsPanel)
+    bpy.utils.register_class(AttractShotRelink)
+    bpy.utils.register_class(AttractShotDelete)
+    bpy.utils.register_class(AttractStripUnlink)
+    bpy.utils.register_class(AttractShotFetchUpdate)
+    bpy.utils.register_class(AttractShotSubmitSelected)
+    bpy.utils.register_class(ATTRACT_OT_submit_all)
+    bpy.utils.register_class(ATTRACT_OT_open_meta_blendfile)
+    bpy.utils.register_class(ATTRACT_OT_shot_open_in_browser)
+    bpy.utils.register_class(ATTRACT_OT_make_shot_thumbnail)
+    bpy.utils.register_class(ATTRACT_OT_copy_id_to_clipboard)
+
+    bpy.app.handlers.scene_update_post.append(scene_update_post_handler)
+    draw.callback_enable()
+
+
+def unregister():
+    draw.callback_disable()
+    bpy.app.handlers.scene_update_post.remove(scene_update_post_handler)
+    bpy.utils.unregister_module(__name__)
+    del bpy.types.Sequence.atc_is_synced
+    del bpy.types.Sequence.atc_object_id
+    del bpy.types.Sequence.atc_object_id_conflict
+    del bpy.types.Sequence.atc_name
+    del bpy.types.Sequence.atc_description
+    del bpy.types.Sequence.atc_notes
+    del bpy.types.Sequence.atc_status
+    del bpy.types.Sequence.atc_order

blender_cloud/attract/draw.py

@@ -0,0 +1,177 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
+# <pep8 compliant>
+
+import bpy
+import logging
+import collections
+
+log = logging.getLogger(__name__)
+
+strip_status_colour = {
+    None: (0.7, 0.7, 0.7),
+    'approved': (0.6392156862745098, 0.8784313725490196, 0.30196078431372547),
+    'final': (0.9058823529411765, 0.9607843137254902, 0.8274509803921568),
+    'in_progress': (1.0, 0.7450980392156863, 0.0),
+    'on_hold': (0.796078431372549, 0.6196078431372549, 0.08235294117647059),
+    'review': (0.8941176470588236, 0.9607843137254902, 0.9764705882352941),
+    'todo': (1.0, 0.5019607843137255, 0.5019607843137255)
+}
+
+CONFLICT_COLOUR = (0.576, 0.118, 0.035)  # RGB tuple
+
+
+def get_strip_rectf(strip):
+    # Get x and y in terms of the grid's frames and channels
+    x1 = strip.frame_final_start
+    x2 = strip.frame_final_end
+    y1 = strip.channel + 0.2
+    y2 = strip.channel - 0.2 + 1
+
+    return x1, y1, x2, y2
+
+
+def draw_underline_in_strip(strip_coords, pixel_size_x, color):
+    from bgl import glColor4f, glRectf, glEnable, glDisable, GL_BLEND
+    import bgl
+
+    context = bpy.context
+
+    # Strip coords
+    s_x1, s_y1, s_x2, s_y2 = strip_coords
+
+    # be careful not to draw over the current frame line
+    cf_x = context.scene.frame_current_final
+
+    bgl.glPushAttrib(bgl.GL_COLOR_BUFFER_BIT | bgl.GL_LINE_BIT)
+
+    glColor4f(*color)
+    glEnable(GL_BLEND)
+    bgl.glLineWidth(2)
+    bgl.glBegin(bgl.GL_LINES)
+
+    bgl.glVertex2f(s_x1, s_y1)
+    if s_x1 < cf_x < s_x2:
+        # Bad luck, the line passes our strip
+        bgl.glVertex2f(cf_x - pixel_size_x, s_y1)
+        bgl.glVertex2f(cf_x + pixel_size_x, s_y1)
+    bgl.glVertex2f(s_x2, s_y1)
+
+    bgl.glEnd()
+    bgl.glPopAttrib()
+
+
+def draw_strip_conflict(strip_coords, pixel_size_x):
+    """Draws conflicting states between strips."""
+
+    import bgl
+
+    s_x1, s_y1, s_x2, s_y2 = strip_coords
+    bgl.glPushAttrib(bgl.GL_COLOR_BUFFER_BIT | bgl.GL_LINE_BIT)
+
+    # Always draw the full rectangle, the conflict should be resolved and thus stand out.
+    bgl.glColor3f(*CONFLICT_COLOUR)
+    bgl.glLineWidth(2)
+
+    bgl.glBegin(bgl.GL_LINE_LOOP)
+    bgl.glVertex2f(s_x1, s_y1)
+    bgl.glVertex2f(s_x2, s_y1)
+    bgl.glVertex2f(s_x2, s_y2)
+    bgl.glVertex2f(s_x1, s_y2)
+    bgl.glEnd()
+
+    bgl.glPopAttrib()
+
+
+def draw_callback_px():
+    context = bpy.context
+
+    if not context.scene.sequence_editor:
+        return
+
+    from . import shown_strips
+
+    region = context.region
+    xwin1, ywin1 = region.view2d.region_to_view(0, 0)
+    xwin2, ywin2 = region.view2d.region_to_view(region.width, region.height)
+    one_pixel_further_x, one_pixel_further_y = region.view2d.region_to_view(1, 1)
+    pixel_size_x = one_pixel_further_x - xwin1
+
+    strips = shown_strips(context)
+
+    for strip in strips:
+        if not strip.atc_object_id:
+            continue
+
+        # Get corners (x1, y1), (x2, y2) of the strip rectangle in px region coords
+        strip_coords = get_strip_rectf(strip)
+
+        # check if any of the coordinates are out of bounds
+        if strip_coords[0] > xwin2 or strip_coords[2] < xwin1 or strip_coords[1] > ywin2 or \
+                        strip_coords[3] < ywin1:
+            continue
+
+        # Draw
+        status = strip.atc_status
+        if status in strip_status_colour:
+            color = strip_status_colour[status]
+        else:
+            color = strip_status_colour[None]
+
+        alpha = 1.0 if strip.atc_is_synced else 0.5
+
+        draw_underline_in_strip(strip_coords, pixel_size_x, color + (alpha,))
+        if strip.atc_is_synced and strip.atc_object_id_conflict:
+            draw_strip_conflict(strip_coords, pixel_size_x)
+
+
+def tag_redraw_all_sequencer_editors():
+    context = bpy.context
+
+    # Py cant access notifiers
+    for window in context.window_manager.windows:
+        for area in window.screen.areas:
+            if area.type == 'SEQUENCE_EDITOR':
+                for region in area.regions:
+                    if region.type == 'WINDOW':
+                        region.tag_redraw()
+
+
+# This is a list so it can be changed instead of set
+# if it is only changed, it does not have to be declared as a global everywhere
+cb_handle = []
+
+
+def callback_enable():
+    if cb_handle:
+        return
+
+    cb_handle[:] = bpy.types.SpaceSequenceEditor.draw_handler_add(
+        draw_callback_px, (), 'WINDOW', 'POST_VIEW'),
+
+    tag_redraw_all_sequencer_editors()
+
+
+def callback_disable():
+    if not cb_handle:
+        return
+
+    bpy.types.SpaceSequenceEditor.draw_handler_remove(cb_handle[0], 'WINDOW')
+
+    tag_redraw_all_sequencer_editors()

blender_cloud/blender.py

@@ -1,26 +1,157 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
 """Blender-specific code.
 
 Separated from __init__.py so that we can import & run from non-Blender environments.
 """
 
+import logging
 import os.path
 
 import bpy
-from bpy.types import AddonPreferences, Operator, WindowManager, Scene
-from bpy.props import StringProperty
+from bpy.types import AddonPreferences, Operator, WindowManager, Scene, PropertyGroup
+from bpy.props import StringProperty, EnumProperty, PointerProperty, BoolProperty, IntProperty
+import rna_prop_ui
 
-from . import pillar, gui
+from . import pillar, async_loop, flamenco
+from .utils import pyside_cache, redraw
+
+PILLAR_WEB_SERVER_URL = 'https://cloud.blender.org/'
+# PILLAR_WEB_SERVER_URL = 'http://pillar-web:5001/'
+PILLAR_SERVER_URL = '%sapi/' % PILLAR_WEB_SERVER_URL
 
 ADDON_NAME = 'blender_cloud'
+log = logging.getLogger(__name__)
+
+icons = None
+
+
+@pyside_cache('version')
+def blender_syncable_versions(self, context):
+    """Returns the list of items used by SyncStatusProperties.version EnumProperty."""
+
+    bss = context.window_manager.blender_sync_status
+    versions = bss.available_blender_versions
+    if not versions:
+        return [('', 'No settings stored in your Blender Cloud', '')]
+    return [(v, v, '') for v in versions]
+
+
+class SyncStatusProperties(PropertyGroup):
+    status = EnumProperty(
+        items=[
+            ('NONE', 'NONE', 'We have done nothing at all yet.'),
+            ('IDLE', 'IDLE', 'User requested something, which is done, and we are now idle.'),
+            ('SYNCING', 'SYNCING', 'Synchronising with Blender Cloud.'),
+        ],
+        name='status',
+        description='Current status of Blender Sync',
+        update=redraw)
+
+    version = EnumProperty(
+        items=blender_syncable_versions,
+        name='Version of Blender from which to pull',
+        description='Version of Blender from which to pull')
+
+    message = StringProperty(name='message', update=redraw)
+    level = EnumProperty(
+        items=[
+            ('INFO', 'INFO', ''),
+            ('WARNING', 'WARNING', ''),
+            ('ERROR', 'ERROR', ''),
+            ('SUBSCRIBE', 'SUBSCRIBE', ''),
+        ],
+        name='level',
+        update=redraw)
+
+    def report(self, level: set, message: str):
+        assert len(level) == 1, 'level should be a set of one string, not %r' % level
+        self.level = level.pop()
+        self.message = message
+
+        # Message can also be empty, just to erase it from the GUI.
+        # No need to actually log those.
+        if message:
+            try:
+                loglevel = logging._nameToLevel[self.level]
+            except KeyError:
+                loglevel = logging.WARNING
+            log.log(loglevel, message)
+
+    # List of syncable versions is stored in 'available_blender_versions' ID property,
+    # because I don't know how to store a variable list of strings in a proper RNA property.
+    @property
+    def available_blender_versions(self) -> list:
+        return self.get('available_blender_versions', [])
+
+    @available_blender_versions.setter
+    def available_blender_versions(self, new_versions):
+        self['available_blender_versions'] = new_versions
+
+
+@pyside_cache('project')
+def bcloud_available_projects(self, context):
+    """Returns the list of items used by BlenderCloudProjectGroup.project EnumProperty."""
+
+    attr_proj = preferences().attract_project
+    projs = attr_proj.available_projects
+    if not projs:
+        return [('', 'No projects available in your Blender Cloud', '')]
+    return [(p['_id'], p['name'], '') for p in projs]
+
+
+class BlenderCloudProjectGroup(PropertyGroup):
+    status = EnumProperty(
+        items=[
+            ('NONE', 'NONE', 'We have done nothing at all yet'),
+            ('IDLE', 'IDLE', 'User requested something, which is done, and we are now idle'),
+            ('FETCHING', 'FETCHING', 'Fetching available projects from Blender Cloud'),
+        ],
+        name='status',
+        update=redraw)
+
+    project = EnumProperty(
+        items=bcloud_available_projects,
+        name='Cloud project',
+        description='Which Blender Cloud project to work with')
+
+    # List of projects is stored in 'available_projects' ID property,
+    # because I don't know how to store a variable list of strings in a proper RNA property.
+    @property
+    def available_projects(self) -> list:
+        return self.get('available_projects', [])
+
+    @available_projects.setter
+    def available_projects(self, new_projects):
+        self['available_projects'] = new_projects
 
 
 class BlenderCloudPreferences(AddonPreferences):
     bl_idname = ADDON_NAME
 
-    pillar_server = bpy.props.StringProperty(
+    # The following two properties are read-only to limit the scope of the
+    # addon and allow for proper testing within this scope.
+    pillar_server = StringProperty(
         name='Blender Cloud Server',
         description='URL of the Blender Cloud backend server',
-        default='https://pillar.blender.org:5000/'
+        default=PILLAR_SERVER_URL,
+        get=lambda self: PILLAR_SERVER_URL
     )
 
     local_texture_dir = StringProperty(
@@ -28,52 +159,250 @@ class BlenderCloudPreferences(AddonPreferences):
         subtype='DIR_PATH',
         default='//textures')
 
+    open_browser_after_share = BoolProperty(
+        name='Open browser after sharing file',
+        description='When enabled, Blender will open a webbrowser',
+        default=True
+    )
+
+    # TODO: store local path with the Attract project, so that people
+    # can switch projects and the local path switches with it.
+    attract_project = PointerProperty(type=BlenderCloudProjectGroup)
+    attract_project_local_path = StringProperty(
+        name='Local project path',
+        description='Local path of your Attract project, used to search for blend files; '
+                    'usually best to set to an absolute path',
+        subtype='DIR_PATH',
+        default='//../')
+
+    flamenco_manager = PointerProperty(type=flamenco.FlamencoManagerGroup)
+    # TODO: before making Flamenco public, change the defaults to something less Institute-specific.
+    # NOTE: The assumption is that the workers can also find the files in the same path.
+    #       This assumption is true for the Blender Institute.
+    flamenco_job_file_path = StringProperty(
+        name='Job file path',
+        description='Path where to store job files, should be accesible for Workers too',
+        subtype='DIR_PATH',
+        default='/render/_flamenco/storage')
+
+    # TODO: before making Flamenco public, change the defaults to something less Institute-specific.
+    flamenco_job_output_path = StringProperty(
+        name='Job output path',
+        description='Path where to store output files, should be accessible for Workers',
+        subtype='DIR_PATH',
+        default='/render/_flamenco/output')
+    flamenco_job_output_strip_components = IntProperty(
+        name='Job output path strip components',
+        description='The final output path comprises of the job output path, and the blend file '
+                    'path relative to the project with this many path components stripped off '
+                    'the front',
+        min=0,
+        default=0,
+        soft_max=4,
+    )
+
     def draw(self, context):
+        import textwrap
+
         layout = self.layout
 
         # Carefully try and import the Blender ID addon
         try:
-            import blender_id.profiles as blender_id_profiles
+            import blender_id
         except ImportError:
-            blender_id_profiles = None
+            blender_id = None
             blender_id_profile = None
         else:
-            blender_id_profile = blender_id_profiles.get_active_profile()
+            blender_id_profile = blender_id.get_active_profile()
 
-        if blender_id_profiles is None:
-            blender_id_icon = 'ERROR'
-            blender_id_text = "This add-on requires Blender ID"
-            blender_id_help = "Make sure that the Blender ID add-on is installed and activated"
+        if blender_id is None:
+            msg_icon = 'ERROR'
+            text = 'This add-on requires Blender ID'
+            help_text = 'Make sure that the Blender ID add-on is installed and activated'
         elif not blender_id_profile:
-            blender_id_icon = 'ERROR'
-            blender_id_text = "You are logged out."
-            blender_id_help = "To login, go to the Blender ID add-on preferences."
+            msg_icon = 'ERROR'
+            text = 'You are logged out.'
+            help_text = 'To login, go to the Blender ID add-on preferences.'
+        elif bpy.app.debug and pillar.SUBCLIENT_ID not in blender_id_profile.subclients:
+            msg_icon = 'QUESTION'
+            text = 'No Blender Cloud credentials.'
+            help_text = ('You are logged in on Blender ID, but your credentials have not '
+                         'been synchronized with Blender Cloud yet. Press the Update '
+                         'Credentials button.')
         else:
-            blender_id_icon = 'WORLD_DATA'
-            blender_id_text = "You are logged in as %s." % blender_id_profile['username']
-            blender_id_help = "To logout or change profile, " \
-                              "go to the Blender ID add-on preferences."
+            msg_icon = 'WORLD_DATA'
+            text = 'You are logged in as %s.' % blender_id_profile.username
+            help_text = ('To logout or change profile, '
+                         'go to the Blender ID add-on preferences.')
 
-        sub = layout.column()
-        sub.label(text=blender_id_text, icon=blender_id_icon)
-        sub.label(text="* " + blender_id_help)
+        # Authentication stuff
+        auth_box = layout.box()
+        auth_box.label(text=text, icon=msg_icon)
 
-        sub = layout.column()
-        sub.label(text='Local directory for downloaded textures')
+        help_lines = textwrap.wrap(help_text, 80)
+        for line in help_lines:
+            auth_box.label(text=line)
+        if bpy.app.debug:
+            auth_box.operator("pillar.credentials_update")
+
+        # Texture browser stuff
+        texture_box = layout.box()
+        texture_box.enabled = msg_icon != 'ERROR'
+        sub = texture_box.column()
+        sub.label(text='Local directory for downloaded textures', icon_value=icon('CLOUD'))
         sub.prop(self, "local_texture_dir", text='Default')
         sub.prop(context.scene, "local_texture_dir", text='Current scene')
 
-        # options for Pillar
-        sub = layout.column()
-        sub.enabled = blender_id_icon != 'ERROR'
-        sub.prop(self, "pillar_server")
-        sub.operator("pillar.credentials_update")
+        # Blender Sync stuff
+        bss = context.window_manager.blender_sync_status
+        bsync_box = layout.box()
+        bsync_box.enabled = msg_icon != 'ERROR'
+        row = bsync_box.row().split(percentage=0.33)
+        row.label('Blender Sync with Blender Cloud', icon_value=icon('CLOUD'))
+
+        icon_for_level = {
+            'INFO': 'NONE',
+            'WARNING': 'INFO',
+            'ERROR': 'ERROR',
+            'SUBSCRIBE': 'ERROR',
+        }
+        msg_icon = icon_for_level[bss.level] if bss.message else 'NONE'
+        message_container = row.row()
+        message_container.label(bss.message, icon=msg_icon)
+
+        sub = bsync_box.column()
+
+        if bss.level == 'SUBSCRIBE':
+            self.draw_subscribe_button(sub)
+        self.draw_sync_buttons(sub, bss)
+
+        # Image Share stuff
+        share_box = layout.box()
+        share_box.label('Image Sharing on Blender Cloud', icon_value=icon('CLOUD'))
+        share_box.prop(self, 'open_browser_after_share')
+
+        # Attract stuff
+        attract_box = layout.box()
+        self.draw_attract_buttons(attract_box, self.attract_project)
+
+        # Flamenco stuff
+        flamenco_box = layout.box()
+        self.draw_flamenco_buttons(flamenco_box, self.flamenco_manager, context)
+
+    def draw_subscribe_button(self, layout):
+        layout.operator('pillar.subscribe', icon='WORLD')
+
+    def draw_sync_buttons(self, layout, bss):
+        layout.enabled = bss.status in {'NONE', 'IDLE'}
+
+        buttons = layout.column()
+        row_buttons = buttons.row().split(percentage=0.5)
+        row_push = row_buttons.row()
+        row_pull = row_buttons.row(align=True)
+
+        row_push.operator('pillar.sync',
+                          text='Save %i.%i settings' % bpy.app.version[:2],
+                          icon='TRIA_UP').action = 'PUSH'
+
+        versions = bss.available_blender_versions
+        version = bss.version
+        if bss.status in {'NONE', 'IDLE'}:
+            if not versions or not version:
+                row_pull.operator('pillar.sync',
+                                  text='Find version to load',
+                                  icon='TRIA_DOWN').action = 'REFRESH'
+            else:
+                props = row_pull.operator('pillar.sync',
+                                          text='Load %s settings' % version,
+                                          icon='TRIA_DOWN')
+                props.action = 'PULL'
+                props.blender_version = version
+                row_pull.operator('pillar.sync',
+                                  text='',
+                                  icon='DOTSDOWN').action = 'SELECT'
+        else:
+            row_pull.label('Cloud Sync is running.')
+
+    def draw_attract_buttons(self, attract_box, bcp: BlenderCloudProjectGroup):
+        attract_row = attract_box.row(align=True)
+        attract_row.label('Attract', icon_value=icon('CLOUD'))
+
+        attract_row.enabled = bcp.status in {'NONE', 'IDLE'}
+        row_buttons = attract_row.row(align=True)
+
+        projects = bcp.available_projects
+        project = bcp.project
+        if bcp.status in {'NONE', 'IDLE'}:
+            if not projects or not project:
+                row_buttons.operator('pillar.projects',
+                                     text='Find project to load',
+                                     icon='FILE_REFRESH')
+            else:
+                row_buttons.prop(bcp, 'project')
+                row_buttons.operator('pillar.projects',
+                                     text='',
+                                     icon='FILE_REFRESH')
+        else:
+            row_buttons.label('Fetching available projects.')
+
+        attract_box.prop(self, 'attract_project_local_path')
+
+    def draw_flamenco_buttons(self, flamenco_box, bcp: flamenco.FlamencoManagerGroup, context):
+        flamenco_row = flamenco_box.row(align=True)
+        flamenco_row.label('Flamenco', icon_value=icon('CLOUD'))
+
+        flamenco_row.enabled = bcp.status in {'NONE', 'IDLE'}
+        row_buttons = flamenco_row.row(align=True)
+
+        if bcp.status in {'NONE', 'IDLE'}:
+            if not bcp.available_managers or not bcp.manager:
+                row_buttons.operator('flamenco.managers',
+                                     text='Find Flamenco Managers',
+                                     icon='FILE_REFRESH')
+            else:
+                row_buttons.prop(bcp, 'manager', text='Manager')
+                row_buttons.operator('flamenco.managers',
+                                     text='',
+                                     icon='FILE_REFRESH')
+        else:
+            row_buttons.label('Fetching available managers.')
+
+        path_box = flamenco_box.row(align=True)
+        path_box.prop(self, 'flamenco_job_file_path')
+        props = path_box.operator('flamenco.explore_file_path', text='', icon='DISK_DRIVE')
+        props.path = self.flamenco_job_file_path
+
+        job_output_box = flamenco_box.column(align=True)
+        path_box = job_output_box.row(align=True)
+        path_box.prop(self, 'flamenco_job_output_path')
+        props = path_box.operator('flamenco.explore_file_path', text='', icon='DISK_DRIVE')
+        props.path = self.flamenco_job_output_path
+
+        job_output_box.prop(self, 'flamenco_job_output_strip_components',
+                            text='Strip components')
+
+        from .flamenco import render_output_path
+
+        path_box = job_output_box.row(align=True)
+        output_path = render_output_path(context)
+        path_box.label(str(output_path))
+        props = path_box.operator('flamenco.explore_file_path', text='', icon='DISK_DRIVE')
+        props.path = str(output_path.parent)
+
+        # TODO: make a reusable way to select projects, and use that for Attract and Flamenco.
+        note_box = flamenco_box.column(align=True)
+        note_box.label('NOTE: For now, Flamenco uses the same project as Attract.')
+        note_box.label('This will change in a future version of the add-on.')
 
 
-class PillarCredentialsUpdate(Operator):
+class PillarCredentialsUpdate(pillar.PillarOperatorMixin,
+                              Operator):
     """Updates the Pillar URL and tests the new URL."""
-    bl_idname = "pillar.credentials_update"
-    bl_label = "Update credentials"
+    bl_idname = 'pillar.credentials_update'
+    bl_label = 'Update credentials'
+    bl_description = 'Resynchronises your Blender ID login with Blender Cloud'
+
+    log = logging.getLogger('bpy.ops.%s' % bl_idname)
 
     @classmethod
     def poll(cls, context):
@@ -82,47 +411,170 @@ class PillarCredentialsUpdate(Operator):
 
     @classmethod
     def is_logged_in(cls, context):
-        active_user_id = getattr(context.window_manager, 'blender_id_active_profile', None)
-        return bool(active_user_id)
+        try:
+            import blender_id
+        except ImportError:
+            return False
+
+        return blender_id.is_logged_in()
 
     def execute(self, context):
+        import blender_id
+        import asyncio
+
         # Only allow activation when the user is actually logged in.
         if not self.is_logged_in(context):
-            self.report({'ERROR'}, "No active profile found")
+            self.report({'ERROR'}, 'No active profile found')
             return {'CANCELLED'}
 
-        # Test the new URL
-        endpoint = bpy.context.user_preferences.addons[ADDON_NAME].preferences.pillar_server
-        pillar._pillar_api = None
         try:
-            pillar.get_project_uuid('textures')  # Just any query will do.
-        except Exception as e:
-            print(e)
-            self.report({'ERROR'}, 'Failed connection to %s' % endpoint)
-            return {'FINISHED'}
+            loop = asyncio.get_event_loop()
+            loop.run_until_complete(self.check_credentials(context, set()))
+        except blender_id.BlenderIdCommError as ex:
+            log.exception('Error sending subclient-specific token to Blender ID')
+            self.report({'ERROR'}, 'Failed to sync Blender ID to Blender Cloud')
+            return {'CANCELLED'}
+        except Exception as ex:
+            log.exception('Error in test call to Pillar')
+            self.report({'ERROR'}, 'Failed test connection to Blender Cloud')
+            return {'CANCELLED'}
 
-        self.report({'INFO'}, 'Updated cloud server address to %s' % endpoint)
+        self.report({'INFO'}, 'Blender Cloud credentials & endpoint URL updated.')
         return {'FINISHED'}
 
 
+class PILLAR_OT_subscribe(Operator):
+    """Opens a browser to subscribe the user to the Cloud."""
+    bl_idname = 'pillar.subscribe'
+    bl_label = 'Subscribe to the Cloud'
+    bl_description = "Opens a page in a web browser to subscribe to the Blender Cloud"
+
+    def execute(self, context):
+        import webbrowser
+
+        webbrowser.open_new_tab('https://cloud.blender.org/join')
+        self.report({'INFO'}, 'We just started a browser for you.')
+
+        return {'FINISHED'}
+
+
+class PILLAR_OT_projects(async_loop.AsyncModalOperatorMixin,
+                         pillar.AuthenticatedPillarOperatorMixin,
+                         Operator):
+    """Fetches the projects available to the user"""
+    bl_idname = 'pillar.projects'
+    bl_label = 'Fetch available projects'
+
+    stop_upon_exception = True
+    _log = logging.getLogger('bpy.ops.%s' % bl_idname)
+
+    async def async_execute(self, context):
+        if not await self.authenticate(context):
+            return
+
+        import pillarsdk
+        from .pillar import pillar_call
+
+        self.log.info('Going to fetch projects for user %s', self.user_id)
+
+        preferences().attract_project.status = 'FETCHING'
+
+        # Get all projects, except the home project.
+        projects_user = await pillar_call(
+            pillarsdk.Project.all,
+            {'where': {'user': self.user_id,
+                       'category': {'$ne': 'home'}},
+             'sort': '-_created',
+             'projection': {'_id': True,
+                            'name': True},
+             })
+
+        projects_shared = await pillar_call(
+            pillarsdk.Project.all,
+            {'where': {'user': {'$ne': self.user_id},
+                       'permissions.groups.group': {'$in': self.db_user.groups}},
+             'sort': '-_created',
+             'projection': {'_id': True,
+                            'name': True},
+             })
+
+        # We need to convert to regular dicts before storing in ID properties.
+        # Also don't store more properties than we need.
+        projects = [{'_id': p['_id'], 'name': p['name']} for p in projects_user['_items']] + \
+                   [{'_id': p['_id'], 'name': p['name']} for p in projects_shared['_items']]
+
+        preferences().attract_project.available_projects = projects
+
+        self.quit()
+
+    def quit(self):
+        preferences().attract_project.status = 'IDLE'
+        super().quit()
+
+
+class PILLAR_PT_image_custom_properties(rna_prop_ui.PropertyPanel, bpy.types.Panel):
+    """Shows custom properties in the image editor."""
+
+    bl_space_type = 'IMAGE_EDITOR'
+    bl_region_type = 'UI'
+    bl_label = 'Custom Properties'
+
+    _context_path = 'edit_image'
+    _property_type = bpy.types.Image
+
+
 def preferences() -> BlenderCloudPreferences:
     return bpy.context.user_preferences.addons[ADDON_NAME].preferences
 
 
+def load_custom_icons():
+    global icons
+
+    if icons is not None:
+        # Already loaded
+        return
+
+    import bpy.utils.previews
+    icons = bpy.utils.previews.new()
+    my_icons_dir = os.path.join(os.path.dirname(__file__), 'icons')
+    icons.load('CLOUD', os.path.join(my_icons_dir, 'icon-cloud.png'), 'IMAGE')
+
+
+def unload_custom_icons():
+    global icons
+
+    if icons is None:
+        # Already unloaded
+        return
+
+    bpy.utils.previews.remove(icons)
+    icons = None
+
+
+def icon(icon_name: str) -> int:
+    """Returns the icon ID for the named icon.
+
+    Use with layout.operator('pillar.image_share', icon_value=icon('CLOUD'))
+    """
+
+    return icons[icon_name].icon_id
+
+
 def register():
+    bpy.utils.register_class(BlenderCloudProjectGroup)
     bpy.utils.register_class(BlenderCloudPreferences)
     bpy.utils.register_class(PillarCredentialsUpdate)
-
-    WindowManager.blender_cloud_project = StringProperty(
-        name="Blender Cloud project UUID",
-        default='5672beecc0261b2005ed1a33')  # TODO: don't hard-code this
-
-    WindowManager.blender_cloud_node = StringProperty(
-        name="Blender Cloud node UUID",
-        default='')  # empty == top-level of project
+    bpy.utils.register_class(SyncStatusProperties)
+    bpy.utils.register_class(PILLAR_OT_subscribe)
+    bpy.utils.register_class(PILLAR_OT_projects)
+    bpy.utils.register_class(PILLAR_PT_image_custom_properties)
 
     addon_prefs = preferences()
 
+    WindowManager.last_blender_cloud_location = StringProperty(
+        name="Last Blender Cloud browser location",
+        default="/")
+
     def default_if_empty(scene, context):
         """The scene's local_texture_dir, if empty, reverts to the addon prefs."""
 
@@ -135,13 +587,21 @@ def register():
         default=addon_prefs.local_texture_dir,
         update=default_if_empty)
 
+    WindowManager.blender_sync_status = PointerProperty(type=SyncStatusProperties)
+
+    load_custom_icons()
+
 
 def unregister():
-    gui.unregister()
+    unload_custom_icons()
 
+    bpy.utils.unregister_class(BlenderCloudProjectGroup)
     bpy.utils.unregister_class(PillarCredentialsUpdate)
     bpy.utils.unregister_class(BlenderCloudPreferences)
+    bpy.utils.unregister_class(SyncStatusProperties)
+    bpy.utils.unregister_class(PILLAR_OT_subscribe)
+    bpy.utils.unregister_class(PILLAR_OT_projects)
+    bpy.utils.unregister_class(PILLAR_PT_image_custom_properties)
 
-    del WindowManager.blender_cloud_project
-    del WindowManager.blender_cloud_node
-    del WindowManager.blender_cloud_thumbnails
+    del WindowManager.last_blender_cloud_location
+    del WindowManager.blender_sync_status

blender_cloud/blendfile.py

@@ -0,0 +1,929 @@
+# ***** BEGIN GPL LICENSE BLOCK *****
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software Foundation,
+# Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
+#
+# ***** END GPL LICENCE BLOCK *****
+#
+# (c) 2009, At Mind B.V. - Jeroen Bakker
+# (c) 2014, Blender Foundation - Campbell Barton
+
+import gzip
+import logging
+import os
+import struct
+import tempfile
+
+log = logging.getLogger("blendfile")
+
+FILE_BUFFER_SIZE = 1024 * 1024
+
+
+# -----------------------------------------------------------------------------
+# module global routines
+#
+# read routines
+# open a filename
+# determine if the file is compressed
+# and returns a handle
+def open_blend(filename, access="rb"):
+    """Opens a blend file for reading or writing pending on the access
+    supports 2 kind of blend files. Uncompressed and compressed.
+    Known issue: does not support packaged blend files
+    """
+    handle = open(filename, access)
+    magic_test = b"BLENDER"
+    magic = handle.read(len(magic_test))
+    if magic == magic_test:
+        log.debug("normal blendfile detected")
+        handle.seek(0, os.SEEK_SET)
+        bfile = BlendFile(handle)
+        bfile.is_compressed = False
+        bfile.filepath_orig = filename
+        return bfile
+    elif magic[:2] == b'\x1f\x8b':
+        log.debug("gzip blendfile detected")
+        handle.close()
+        log.debug("decompressing started")
+        fs = gzip.open(filename, "rb")
+        data = fs.read(FILE_BUFFER_SIZE)
+        magic = data[:len(magic_test)]
+        if magic == magic_test:
+            handle = tempfile.TemporaryFile()
+            while data:
+                handle.write(data)
+                data = fs.read(FILE_BUFFER_SIZE)
+            log.debug("decompressing finished")
+            fs.close()
+            log.debug("resetting decompressed file")
+            handle.seek(os.SEEK_SET, 0)
+            bfile = BlendFile(handle)
+            bfile.is_compressed = True
+            bfile.filepath_orig = filename
+            return bfile
+        else:
+            raise Exception("filetype inside gzip not a blend")
+    else:
+        raise Exception("filetype not a blend or a gzip blend")
+
+
+def pad_up_4(offset):
+    return (offset + 3) & ~3
+
+
+# -----------------------------------------------------------------------------
+# module classes
+
+
+class BlendFile:
+    """
+    Blend file.
+    """
+    __slots__ = (
+        # file (result of open())
+        "handle",
+        # str (original name of the file path)
+        "filepath_orig",
+        # BlendFileHeader
+        "header",
+        # struct.Struct
+        "block_header_struct",
+        # BlendFileBlock
+        "blocks",
+        # [DNAStruct, ...]
+        "structs",
+        # dict {b'StructName': sdna_index}
+        # (where the index is an index into 'structs')
+        "sdna_index_from_id",
+        # dict {addr_old: block}
+        "block_from_offset",
+        # int
+        "code_index",
+        # bool (did we make a change)
+        "is_modified",
+        # bool (is file gzipped)
+        "is_compressed",
+        )
+
+    def __init__(self, handle):
+        log.debug("initializing reading blend-file")
+        self.handle = handle
+        self.header = BlendFileHeader(handle)
+        self.block_header_struct = self.header.create_block_header_struct()
+        self.blocks = []
+        self.code_index = {}
+
+        block = BlendFileBlock(handle, self)
+        while block.code != b'ENDB':
+            if block.code == b'DNA1':
+                (self.structs,
+                 self.sdna_index_from_id,
+                 ) = BlendFile.decode_structs(self.header, block, handle)
+            else:
+                handle.seek(block.size, os.SEEK_CUR)
+
+            self.blocks.append(block)
+            self.code_index.setdefault(block.code, []).append(block)
+
+            block = BlendFileBlock(handle, self)
+        self.is_modified = False
+        self.blocks.append(block)
+
+        # cache (could lazy init, incase we never use?)
+        self.block_from_offset = {block.addr_old: block for block in self.blocks if block.code != b'ENDB'}
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, type, value, traceback):
+        self.close()
+
+    def find_blocks_from_code(self, code):
+        assert(type(code) == bytes)
+        if code not in self.code_index:
+            return []
+        return self.code_index[code]
+
+    def find_block_from_offset(self, offset):
+        # same as looking looping over all blocks,
+        # then checking ``block.addr_old == offset``
+        assert(type(offset) is int)
+        return self.block_from_offset.get(offset)
+
+    def close(self):
+        """
+        Close the blend file
+        writes the blend file to disk if changes has happened
+        """
+        handle = self.handle
+
+        if self.is_modified:
+            if self.is_compressed:
+                log.debug("close compressed blend file")
+                handle.seek(os.SEEK_SET, 0)
+                log.debug("compressing started")
+                fs = gzip.open(self.filepath_orig, "wb")
+                data = handle.read(FILE_BUFFER_SIZE)
+                while data:
+                    fs.write(data)
+                    data = handle.read(FILE_BUFFER_SIZE)
+                fs.close()
+                log.debug("compressing finished")
+
+        handle.close()
+
+    def ensure_subtype_smaller(self, sdna_index_curr, sdna_index_next):
+        # never refine to a smaller type
+        if (self.structs[sdna_index_curr].size >
+            self.structs[sdna_index_next].size):
+
+            raise RuntimeError("cant refine to smaller type (%s -> %s)" %
+                               (self.structs[sdna_index_curr].dna_type_id.decode('ascii'),
+                                self.structs[sdna_index_next].dna_type_id.decode('ascii')))
+
+    @staticmethod
+    def decode_structs(header, block, handle):
+        """
+        DNACatalog is a catalog of all information in the DNA1 file-block
+        """
+        log.debug("building DNA catalog")
+        shortstruct = DNA_IO.USHORT[header.endian_index]
+        shortstruct2 = struct.Struct(header.endian_str + b'HH')
+        intstruct = DNA_IO.UINT[header.endian_index]
+
+        data = handle.read(block.size)
+        types = []
+        names = []
+
+        structs = []
+        sdna_index_from_id = {}
+
+        offset = 8
+        names_len = intstruct.unpack_from(data, offset)[0]
+        offset += 4
+
+        log.debug("building #%d names" % names_len)
+        for i in range(names_len):
+            tName = DNA_IO.read_data0_offset(data, offset)
+            offset = offset + len(tName) + 1
+            names.append(DNAName(tName))
+        del names_len
+
+        offset = pad_up_4(offset)
+        offset += 4
+        types_len = intstruct.unpack_from(data, offset)[0]
+        offset += 4
+        log.debug("building #%d types" % types_len)
+        for i in range(types_len):
+            dna_type_id = DNA_IO.read_data0_offset(data, offset)
+            # None will be replaced by the DNAStruct, below
+            types.append(DNAStruct(dna_type_id))
+            offset += len(dna_type_id) + 1
+
+        offset = pad_up_4(offset)
+        offset += 4
+        log.debug("building #%d type-lengths" % types_len)
+        for i in range(types_len):
+            tLen = shortstruct.unpack_from(data, offset)[0]
+            offset = offset + 2
+            types[i].size = tLen
+        del types_len
+
+        offset = pad_up_4(offset)
+        offset += 4
+
+        structs_len = intstruct.unpack_from(data, offset)[0]
+        offset += 4
+        log.debug("building #%d structures" % structs_len)
+        for sdna_index in range(structs_len):
+            d = shortstruct2.unpack_from(data, offset)
+            struct_type_index = d[0]
+            offset += 4
+            dna_struct = types[struct_type_index]
+            sdna_index_from_id[dna_struct.dna_type_id] = sdna_index
+            structs.append(dna_struct)
+
+            fields_len = d[1]
+            dna_offset = 0
+
+            for field_index in range(fields_len):
+                d2 = shortstruct2.unpack_from(data, offset)
+                field_type_index = d2[0]
+                field_name_index = d2[1]
+                offset += 4
+                dna_type = types[field_type_index]
+                dna_name = names[field_name_index]
+                if dna_name.is_pointer or dna_name.is_method_pointer:
+                    dna_size = header.pointer_size * dna_name.array_size
+                else:
+                    dna_size = dna_type.size * dna_name.array_size
+
+                field = DNAField(dna_type, dna_name, dna_size, dna_offset)
+                dna_struct.fields.append(field)
+                dna_struct.field_from_name[dna_name.name_only] = field
+                dna_offset += dna_size
+
+        return structs, sdna_index_from_id
+
+
+class BlendFileBlock:
+    """
+    Instance of a struct.
+    """
+    __slots__ = (
+        # BlendFile
+        "file",
+        "code",
+        "size",
+        "addr_old",
+        "sdna_index",
+        "count",
+        "file_offset",
+        "user_data",
+        )
+
+    def __str__(self):
+        return ("<%s.%s (%s), size=%d at %s>" %
+                # fields=[%s]
+                (self.__class__.__name__,
+                 self.dna_type.dna_type_id.decode('ascii'),
+                 self.code.decode(),
+                 self.size,
+                 # b", ".join(f.dna_name.name_only for f in self.dna_type.fields).decode('ascii'),
+                 hex(self.addr_old),
+                 ))
+
+    def __init__(self, handle, bfile):
+        OLDBLOCK = struct.Struct(b'4sI')
+
+        self.file = bfile
+        self.user_data = None
+
+        data = handle.read(bfile.block_header_struct.size)
+        # header size can be 8, 20, or 24 bytes long
+        # 8: old blend files ENDB block (exception)
+        # 20: normal headers 32 bit platform
+        # 24: normal headers 64 bit platform
+        if len(data) > 15:
+
+            blockheader = bfile.block_header_struct.unpack(data)
+            self.code = blockheader[0].partition(b'\0')[0]
+            if self.code != b'ENDB':
+                self.size = blockheader[1]
+                self.addr_old = blockheader[2]
+                self.sdna_index = blockheader[3]
+                self.count = blockheader[4]
+                self.file_offset = handle.tell()
+            else:
+                self.size = 0
+                self.addr_old = 0
+                self.sdna_index = 0
+                self.count = 0
+                self.file_offset = 0
+        else:
+            blockheader = OLDBLOCK.unpack(data)
+            self.code = blockheader[0].partition(b'\0')[0]
+            self.code = DNA_IO.read_data0(blockheader[0])
+            self.size = 0
+            self.addr_old = 0
+            self.sdna_index = 0
+            self.count = 0
+            self.file_offset = 0
+
+    @property
+    def dna_type(self):
+        return self.file.structs[self.sdna_index]
+
+    def refine_type_from_index(self, sdna_index_next):
+        assert(type(sdna_index_next) is int)
+        sdna_index_curr = self.sdna_index
+        self.file.ensure_subtype_smaller(sdna_index_curr, sdna_index_next)
+        self.sdna_index = sdna_index_next
+
+    def refine_type(self, dna_type_id):
+        assert(type(dna_type_id) is bytes)
+        self.refine_type_from_index(self.file.sdna_index_from_id[dna_type_id])
+
+    def get_file_offset(self, path,
+            default=...,
+            sdna_index_refine=None,
+            base_index=0,
+            ):
+        """
+        Return (offset, length)
+        """
+        assert(type(path) is bytes)
+
+        ofs = self.file_offset
+        if base_index != 0:
+            assert(base_index < self.count)
+            ofs += (self.size // self.count) * base_index
+        self.file.handle.seek(ofs, os.SEEK_SET)
+
+        if sdna_index_refine is None:
+            sdna_index_refine = self.sdna_index
+        else:
+            self.file.ensure_subtype_smaller(self.sdna_index, sdna_index_refine)
+
+        dna_struct = self.file.structs[sdna_index_refine]
+        field = dna_struct.field_from_path(
+                self.file.header, self.file.handle, path)
+
+        return (self.file.handle.tell(), field.dna_name.array_size)
+
+    def get(self, path,
+            default=...,
+            sdna_index_refine=None,
+            use_nil=True, use_str=True,
+            base_index=0,
+            ):
+
+        ofs = self.file_offset
+        if base_index != 0:
+            assert(base_index < self.count)
+            ofs += (self.size // self.count) * base_index
+        self.file.handle.seek(ofs, os.SEEK_SET)
+
+        if sdna_index_refine is None:
+            sdna_index_refine = self.sdna_index
+        else:
+            self.file.ensure_subtype_smaller(self.sdna_index, sdna_index_refine)
+
+        dna_struct = self.file.structs[sdna_index_refine]
+        return dna_struct.field_get(
+                self.file.header, self.file.handle, path,
+                default=default,
+                use_nil=use_nil, use_str=use_str,
+                )
+
+    def get_recursive_iter(self, path, path_root=b"",
+                           default=...,
+                           sdna_index_refine=None,
+                           use_nil=True, use_str=True,
+                           base_index=0,
+                           ):
+        if path_root:
+            path_full = (
+                    (path_root if type(path_root) is tuple else (path_root, )) +
+                    (path if type(path) is tuple else (path, )))
+        else:
+            path_full = path
+
+        try:
+            yield (path_full, self.get(path_full, default, sdna_index_refine, use_nil, use_str, base_index))
+        except NotImplementedError as ex:
+            msg, dna_name, dna_type = ex.args
+            struct_index = self.file.sdna_index_from_id.get(dna_type.dna_type_id, None)
+            if struct_index is None:
+                yield (path_full, "<%s>" % dna_type.dna_type_id.decode('ascii'))
+            else:
+                struct = self.file.structs[struct_index]
+                for f in struct.fields:
+                    yield from self.get_recursive_iter(
+                            f.dna_name.name_only, path_full, default, None, use_nil, use_str, 0)
+
+    def items_recursive_iter(self):
+        for k in self.keys():
+            yield from self.get_recursive_iter(k, use_str=False)
+
+    def get_data_hash(self):
+        """
+        Generates a 'hash' that can be used instead of addr_old as block id, and that should be 'stable' across .blend
+        file load & save (i.e. it does not changes due to pointer addresses variations).
+        """
+        # TODO This implementation is most likely far from optimal... and CRC32 is not renown as the best hashing
+        #      algo either. But for now does the job!
+        import zlib
+        def _is_pointer(self, k):
+            return self.file.structs[self.sdna_index].field_from_path(
+                    self.file.header, self.file.handle, k).dna_name.is_pointer
+
+        hsh = 1
+        for k, v in self.items_recursive_iter():
+            if not _is_pointer(self, k):
+                hsh = zlib.adler32(str(v).encode(), hsh)
+        return hsh
+
+    def set(self, path, value,
+            sdna_index_refine=None,
+            ):
+
+        if sdna_index_refine is None:
+            sdna_index_refine = self.sdna_index
+        else:
+            self.file.ensure_subtype_smaller(self.sdna_index, sdna_index_refine)
+
+        dna_struct = self.file.structs[sdna_index_refine]
+        self.file.handle.seek(self.file_offset, os.SEEK_SET)
+        self.file.is_modified = True
+        return dna_struct.field_set(
+                self.file.header, self.file.handle, path, value)
+
+    # ---------------
+    # Utility get/set
+    #
+    #   avoid inline pointer casting
+    def get_pointer(
+            self, path,
+            default=...,
+            sdna_index_refine=None,
+            base_index=0,
+            ):
+        if sdna_index_refine is None:
+            sdna_index_refine = self.sdna_index
+        result = self.get(path, default, sdna_index_refine=sdna_index_refine, base_index=base_index)
+
+        # default
+        if type(result) is not int:
+            return result
+
+        assert(self.file.structs[sdna_index_refine].field_from_path(
+                self.file.header, self.file.handle, path).dna_name.is_pointer)
+        if result != 0:
+            # possible (but unlikely)
+            # that this fails and returns None
+            # maybe we want to raise some exception in this case
+            return self.file.find_block_from_offset(result)
+        else:
+            return None
+
+    # ----------------------
+    # Python convenience API
+
+    # dict like access
+    def __getitem__(self, item):
+        return self.get(item, use_str=False)
+
+    def __setitem__(self, item, value):
+        self.set(item, value)
+
+    def keys(self):
+        return (f.dna_name.name_only for f in self.dna_type.fields)
+
+    def values(self):
+        for k in self.keys():
+            try:
+                yield self[k]
+            except NotImplementedError as ex:
+                msg, dna_name, dna_type = ex.args
+                yield "<%s>" % dna_type.dna_type_id.decode('ascii')
+
+    def items(self):
+        for k in self.keys():
+            try:
+                yield (k, self[k])
+            except NotImplementedError as ex:
+                msg, dna_name, dna_type = ex.args
+                yield (k, "<%s>" % dna_type.dna_type_id.decode('ascii'))
+
+
+# -----------------------------------------------------------------------------
+# Read Magic
+#
+# magic = str
+# pointer_size = int
+# is_little_endian = bool
+# version = int
+
+
+class BlendFileHeader:
+    """
+    BlendFileHeader allocates the first 12 bytes of a blend file
+    it contains information about the hardware architecture
+    """
+    __slots__ = (
+        # str
+        "magic",
+        # int 4/8
+        "pointer_size",
+        # bool
+        "is_little_endian",
+        # int
+        "version",
+        # str, used to pass to 'struct'
+        "endian_str",
+        # int, used to index common types
+        "endian_index",
+        )
+
+    def __init__(self, handle):
+        FILEHEADER = struct.Struct(b'7s1s1s3s')
+
+        log.debug("reading blend-file-header")
+        values = FILEHEADER.unpack(handle.read(FILEHEADER.size))
+        self.magic = values[0]
+        pointer_size_id = values[1]
+        if pointer_size_id == b'-':
+            self.pointer_size = 8
+        elif pointer_size_id == b'_':
+            self.pointer_size = 4
+        else:
+            assert(0)
+        endian_id = values[2]
+        if endian_id == b'v':
+            self.is_little_endian = True
+            self.endian_str = b'<'
+            self.endian_index = 0
+        elif endian_id == b'V':
+            self.is_little_endian = False
+            self.endian_index = 1
+            self.endian_str = b'>'
+        else:
+            assert(0)
+
+        version_id = values[3]
+        self.version = int(version_id)
+
+    def create_block_header_struct(self):
+        return struct.Struct(b''.join((
+                self.endian_str,
+                b'4sI',
+                b'I' if self.pointer_size == 4 else b'Q',
+                b'II',
+                )))
+
+
+class DNAName:
+    """
+    DNAName is a C-type name stored in the DNA
+    """
+    __slots__ = (
+        "name_full",
+        "name_only",
+        "is_pointer",
+        "is_method_pointer",
+        "array_size",
+        )
+
+    def __init__(self, name_full):
+        self.name_full = name_full
+        self.name_only = self.calc_name_only()
+        self.is_pointer = self.calc_is_pointer()
+        self.is_method_pointer = self.calc_is_method_pointer()
+        self.array_size = self.calc_array_size()
+
+    def __repr__(self):
+        return '%s(%r)' % (type(self).__qualname__, self.name_full)
+
+    def as_reference(self, parent):
+        if parent is None:
+            result = b''
+        else:
+            result = parent + b'.'
+
+        result = result + self.name_only
+        return result
+
+    def calc_name_only(self):
+        result = self.name_full.strip(b'*()')
+        index = result.find(b'[')
+        if index != -1:
+            result = result[:index]
+        return result
+
+    def calc_is_pointer(self):
+        return (b'*' in self.name_full)
+
+    def calc_is_method_pointer(self):
+        return (b'(*' in self.name_full)
+
+    def calc_array_size(self):
+        result = 1
+        temp = self.name_full
+        index = temp.find(b'[')
+
+        while index != -1:
+            index_2 = temp.find(b']')
+            result *= int(temp[index + 1:index_2])
+            temp = temp[index_2 + 1:]
+            index = temp.find(b'[')
+
+        return result
+
+
+class DNAField:
+    """
+    DNAField is a coupled DNAStruct and DNAName
+    and cache offset for reuse
+    """
+    __slots__ = (
+        # DNAName
+        "dna_name",
+        # tuple of 3 items
+        # [bytes (struct name), int (struct size), DNAStruct]
+        "dna_type",
+        # size on-disk
+        "dna_size",
+        # cached info (avoid looping over fields each time)
+        "dna_offset",
+        )
+
+    def __init__(self, dna_type, dna_name, dna_size, dna_offset):
+        self.dna_type = dna_type
+        self.dna_name = dna_name
+        self.dna_size = dna_size
+        self.dna_offset = dna_offset
+
+
+class DNAStruct:
+    """
+    DNAStruct is a C-type structure stored in the DNA
+    """
+    __slots__ = (
+        "dna_type_id",
+        "size",
+        "fields",
+        "field_from_name",
+        "user_data",
+        )
+
+    def __init__(self, dna_type_id):
+        self.dna_type_id = dna_type_id
+        self.fields = []
+        self.field_from_name = {}
+        self.user_data = None
+
+    def __repr__(self):
+        return '%s(%r)' % (type(self).__qualname__, self.dna_type_id)
+
+    def field_from_path(self, header, handle, path):
+        """
+        Support lookups as bytes or a tuple of bytes and optional index.
+
+        C style 'id.name'   -->  (b'id', b'name')
+        C style 'array[4]'  -->  ('array', 4)
+        """
+        if type(path) is tuple:
+            name = path[0]
+            if len(path) >= 2 and type(path[1]) is not bytes:
+                name_tail = path[2:]
+                index = path[1]
+                assert(type(index) is int)
+            else:
+                name_tail = path[1:]
+                index = 0
+        else:
+            name = path
+            name_tail = None
+            index = 0
+
+        assert(type(name) is bytes)
+
+        field = self.field_from_name.get(name)
+
+        if field is not None:
+            handle.seek(field.dna_offset, os.SEEK_CUR)
+            if index != 0:
+                if field.dna_name.is_pointer:
+                    index_offset = header.pointer_size * index
+                else:
+                    index_offset = field.dna_type.size * index
+                assert(index_offset < field.dna_size)
+                handle.seek(index_offset, os.SEEK_CUR)
+            if not name_tail:  # None or ()
+                return field
+            else:
+                return field.dna_type.field_from_path(header, handle, name_tail)
+
+    def field_get(self, header, handle, path,
+                  default=...,
+                  use_nil=True, use_str=True,
+                  ):
+        field = self.field_from_path(header, handle, path)
+        if field is None:
+            if default is not ...:
+                return default
+            else:
+                raise KeyError("%r not found in %r (%r)" %
+                        (path, [f.dna_name.name_only for f in self.fields], self.dna_type_id))
+
+        dna_type = field.dna_type
+        dna_name = field.dna_name
+
+        if dna_name.is_pointer:
+            return DNA_IO.read_pointer(handle, header)
+        elif dna_type.dna_type_id == b'int':
+            if dna_name.array_size > 1:
+                return [DNA_IO.read_int(handle, header) for i in range(dna_name.array_size)]
+            return DNA_IO.read_int(handle, header)
+        elif dna_type.dna_type_id == b'short':
+            if dna_name.array_size > 1:
+                return [DNA_IO.read_short(handle, header) for i in range(dna_name.array_size)]
+            return DNA_IO.read_short(handle, header)
+        elif dna_type.dna_type_id == b'uint64_t':
+            if dna_name.array_size > 1:
+                return [DNA_IO.read_ulong(handle, header) for i in range(dna_name.array_size)]
+            return DNA_IO.read_ulong(handle, header)
+        elif dna_type.dna_type_id == b'float':
+            if dna_name.array_size > 1:
+                return [DNA_IO.read_float(handle, header) for i in range(dna_name.array_size)]
+            return DNA_IO.read_float(handle, header)
+        elif dna_type.dna_type_id == b'char':
+            if use_str:
+                if use_nil:
+                    return DNA_IO.read_string0(handle, dna_name.array_size)
+                else:
+                    return DNA_IO.read_string(handle, dna_name.array_size)
+            else:
+                if use_nil:
+                    return DNA_IO.read_bytes0(handle, dna_name.array_size)
+                else:
+                    return DNA_IO.read_bytes(handle, dna_name.array_size)
+        else:
+            raise NotImplementedError("%r exists but isn't pointer, can't resolve field %r" %
+                    (path, dna_name.name_only), dna_name, dna_type)
+
+    def field_set(self, header, handle, path, value):
+        assert(type(path) == bytes)
+
+        field = self.field_from_path(header, handle, path)
+        if field is None:
+            raise KeyError("%r not found in %r" %
+                    (path, [f.dna_name.name_only for f in self.fields]))
+
+        dna_type = field.dna_type
+        dna_name = field.dna_name
+
+        if dna_type.dna_type_id == b'char':
+            if type(value) is str:
+                return DNA_IO.write_string(handle, value, dna_name.array_size)
+            else:
+                return DNA_IO.write_bytes(handle, value, dna_name.array_size)
+        elif dna_type.dna_type_id == b'int':
+            DNA_IO.write_int(handle, header, value)
+        else:
+            raise NotImplementedError("Setting %r is not yet supported for %r" %
+                                      (dna_type, dna_name), dna_name, dna_type)
+
+
+class DNA_IO:
+    """
+    Module like class, for read-write utility functions.
+
+    Only stores static methods & constants.
+    """
+
+    __slots__ = ()
+
+    def __new__(cls, *args, **kwargs):
+        raise RuntimeError("%s should not be instantiated" % cls)
+
+    @staticmethod
+    def write_string(handle, astring, fieldlen):
+        assert(isinstance(astring, str))
+        if len(astring) >= fieldlen:
+            stringw = astring[0:fieldlen]
+        else:
+            stringw = astring + '\0'
+        handle.write(stringw.encode('utf-8'))
+
+    @staticmethod
+    def write_bytes(handle, astring, fieldlen):
+        assert(isinstance(astring, (bytes, bytearray)))
+        if len(astring) >= fieldlen:
+            stringw = astring[0:fieldlen]
+        else:
+            stringw = astring + b'\0'
+
+        handle.write(stringw)
+
+    @staticmethod
+    def read_bytes(handle, length):
+        data = handle.read(length)
+        return data
+
+    @staticmethod
+    def read_bytes0(handle, length):
+        data = handle.read(length)
+        return DNA_IO.read_data0(data)
+
+    @staticmethod
+    def read_string(handle, length):
+        return DNA_IO.read_bytes(handle, length).decode('utf-8')
+
+    @staticmethod
+    def read_string0(handle, length):
+        return DNA_IO.read_bytes0(handle, length).decode('utf-8')
+
+    @staticmethod
+    def read_data0_offset(data, offset):
+        add = data.find(b'\0', offset) - offset
+        return data[offset:offset + add]
+
+    @staticmethod
+    def read_data0(data):
+        add = data.find(b'\0')
+        return data[:add]
+
+    USHORT = struct.Struct(b'<H'), struct.Struct(b'>H')
+
+    @staticmethod
+    def read_ushort(handle, fileheader):
+        st = DNA_IO.USHORT[fileheader.endian_index]
+        return st.unpack(handle.read(st.size))[0]
+
+    SSHORT = struct.Struct(b'<h'), struct.Struct(b'>h')
+
+    @staticmethod
+    def read_short(handle, fileheader):
+        st = DNA_IO.SSHORT[fileheader.endian_index]
+        return st.unpack(handle.read(st.size))[0]
+
+    UINT = struct.Struct(b'<I'), struct.Struct(b'>I')
+
+    @staticmethod
+    def read_uint(handle, fileheader):
+        st = DNA_IO.UINT[fileheader.endian_index]
+        return st.unpack(handle.read(st.size))[0]
+
+    SINT = struct.Struct(b'<i'), struct.Struct(b'>i')
+
+    @staticmethod
+    def read_int(handle, fileheader):
+        st = DNA_IO.SINT[fileheader.endian_index]
+        return st.unpack(handle.read(st.size))[0]
+
+    @staticmethod
+    def write_int(handle, fileheader, value):
+        assert isinstance(value, int), 'value must be int, but is %r: %r' % (type(value), value)
+        st = DNA_IO.SINT[fileheader.endian_index]
+        to_write = st.pack(value)
+        handle.write(to_write)
+
+    FLOAT = struct.Struct(b'<f'), struct.Struct(b'>f')
+
+    @staticmethod
+    def read_float(handle, fileheader):
+        st = DNA_IO.FLOAT[fileheader.endian_index]
+        return st.unpack(handle.read(st.size))[0]
+
+    ULONG = struct.Struct(b'<Q'), struct.Struct(b'>Q')
+
+    @staticmethod
+    def read_ulong(handle, fileheader):
+        st = DNA_IO.ULONG[fileheader.endian_index]
+        return st.unpack(handle.read(st.size))[0]
+
+    @staticmethod
+    def read_pointer(handle, header):
+        """
+        reads an pointer from a file handle
+        the pointer size is given by the header (BlendFileHeader)
+        """
+        if header.pointer_size == 4:
+            st = DNA_IO.UINT[header.endian_index]
+            return st.unpack(handle.read(st.size))[0]
+        if header.pointer_size == 8:
+            st = DNA_IO.ULONG[header.endian_index]
+            return st.unpack(handle.read(st.size))[0]

blender_cloud/cache.py

@@ -1,3 +1,21 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
 """HTTP Cache management.
 
 This module configures a cached session for the Requests package.
@@ -34,12 +52,16 @@ def cache_directory(*subdirs) -> str:
 
     from . import pillar
 
-    profile = pillar.blender_id_profile() or {'username': 'anonymous'}
+    profile = pillar.blender_id_profile()
+    if profile:
+        username = profile.username
+    else:
+        username = 'anonymous'
 
     # TODO: use bpy.utils.user_resource('CACHE', ...)
     # once https://developer.blender.org/T47684 is finished.
     user_cache_dir = appdirs.user_cache_dir(appname='Blender', appauthor=False)
-    cache_dir = os.path.join(user_cache_dir, 'blender_cloud', profile['username'], *subdirs)
+    cache_dir = os.path.join(user_cache_dir, 'blender_cloud', username, *subdirs)
 
     os.makedirs(cache_dir, mode=0o700, exist_ok=True)
 

blender_cloud/flamenco/__init__.py

@@ -0,0 +1,511 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
+"""Flamenco interface.
+
+The preferences are managed blender.py, the rest of the Flamenco-specific stuff is here.
+"""
+import functools
+import logging
+from pathlib import Path, PurePath
+import typing
+
+import bpy
+from bpy.types import AddonPreferences, Operator, WindowManager, Scene, PropertyGroup
+from bpy.props import StringProperty, EnumProperty, PointerProperty, BoolProperty, IntProperty
+
+from .. import async_loop, pillar
+from ..utils import pyside_cache, redraw
+
+log = logging.getLogger(__name__)
+
+
+@pyside_cache('manager')
+def available_managers(self, context):
+    """Returns the list of items used by a manager-selector EnumProperty."""
+
+    from ..blender import preferences
+
+    mngrs = preferences().flamenco_manager.available_managers
+    if not mngrs:
+        return [('', 'No managers available in your Blender Cloud', '')]
+    return [(p['_id'], p['name'], '') for p in mngrs]
+
+
+class FlamencoManagerGroup(PropertyGroup):
+    manager = EnumProperty(
+        items=available_managers,
+        name='Flamenco Manager',
+        description='Which Flamenco Manager to use for jobs')
+
+    status = EnumProperty(
+        items=[
+            ('NONE', 'NONE', 'We have done nothing at all yet'),
+            ('IDLE', 'IDLE', 'User requested something, which is done, and we are now idle'),
+            ('FETCHING', 'FETCHING', 'Fetching available Flamenco managers from Blender Cloud'),
+        ],
+        name='status',
+        update=redraw)
+
+    # List of managers is stored in 'available_managers' ID property,
+    # because I don't know how to store a variable list of strings in a proper RNA property.
+    @property
+    def available_managers(self) -> list:
+        return self.get('available_managers', [])
+
+    @available_managers.setter
+    def available_managers(self, new_managers):
+        self['available_managers'] = new_managers
+
+
+class FLAMENCO_OT_fmanagers(async_loop.AsyncModalOperatorMixin,
+                            pillar.AuthenticatedPillarOperatorMixin,
+                            Operator):
+    """Fetches the Flamenco Managers available to the user"""
+    bl_idname = 'flamenco.managers'
+    bl_label = 'Fetch available Flamenco Managers'
+
+    stop_upon_exception = True
+    log = logging.getLogger('%s.FLAMENCO_OT_fmanagers' % __name__)
+
+    @property
+    def mypref(self) -> FlamencoManagerGroup:
+        from ..blender import preferences
+
+        return preferences().flamenco_manager
+
+    async def async_execute(self, context):
+        if not await self.authenticate(context):
+            return
+
+        from .sdk import Manager
+        from ..pillar import pillar_call
+
+        self.log.info('Going to fetch managers for user %s', self.user_id)
+
+        self.mypref.status = 'FETCHING'
+        managers = await pillar_call(Manager.all)
+
+        # We need to convert to regular dicts before storing in ID properties.
+        # Also don't store more properties than we need.
+        as_list = [{'_id': p['_id'], 'name': p['name']} for p in managers['_items']]
+
+        self.mypref.available_managers = as_list
+        self.quit()
+
+    def quit(self):
+        self.mypref.status = 'IDLE'
+        super().quit()
+
+
+class FLAMENCO_OT_render(async_loop.AsyncModalOperatorMixin,
+                         pillar.AuthenticatedPillarOperatorMixin,
+                         Operator):
+    """Performs a Blender render on Flamenco."""
+    bl_idname = 'flamenco.render'
+    bl_label = 'Render on Flamenco'
+    bl_description = __doc__.rstrip('.')
+
+    stop_upon_exception = True
+    log = logging.getLogger('%s.FLAMENCO_OT_render' % __name__)
+
+    async def async_execute(self, context):
+        if not await self.authenticate(context):
+            return
+
+        context.window_manager.progress_begin(0, 4)
+        context.window_manager.progress_update(1)
+
+        from pillarsdk import exceptions as sdk_exceptions
+        from ..blender import preferences
+
+        filepath = Path(context.blend_data.filepath)
+        scene = context.scene
+
+        # The file extension should be determined by the render settings, not necessarily
+        # by the setttings in the output panel.
+        scene.render.use_file_extension = True
+        bpy.ops.wm.save_mainfile()
+
+        # Determine where the render output will be stored.
+        render_output = render_output_path(context)
+        if render_output is None:
+            self.report({'ERROR'}, 'Current file is outside of project path.')
+            self.quit()
+            return
+        self.log.info('Will output render files to %s', render_output)
+
+        # BAM-pack the files to the destination directory.
+        outfile, missing_sources = await self.bam_pack(filepath)
+        if not outfile:
+            return
+
+        context.window_manager.progress_update(3)
+
+        # Create the job at Flamenco Server.
+        prefs = preferences()
+
+        settings = {'blender_cmd': '{blender}',
+                    'chunk_size': scene.flamenco_render_chunk_size,
+                    'filepath': str(outfile),
+                    'frames': scene.flamenco_render_frame_range,
+                    'render_output': str(render_output),
+                    }
+        try:
+            job_info = await create_job(self.user_id,
+                                        prefs.attract_project.project,
+                                        prefs.flamenco_manager.manager,
+                                        scene.flamenco_render_job_type,
+                                        settings,
+                                        'Render %s' % filepath.name,
+                                        priority=scene.flamenco_render_job_priority)
+        except sdk_exceptions.ResourceInvalid as ex:
+            self.report({'ERROR'}, 'Error creating Flamenco job: %s' % ex)
+            self.quit()
+            return
+
+        # Store the job ID in a file in the output dir.
+        with open(str(outfile.parent / 'jobinfo.json'), 'w', encoding='utf8') as outfile:
+            import json
+
+            job_info['missing_files'] = [str(mf) for mf in missing_sources]
+            json.dump(job_info, outfile, sort_keys=True, indent=4)
+
+        # Do a final report.
+        if missing_sources:
+            names = (ms.name for ms in missing_sources)
+            self.report({'WARNING'}, 'Flamenco job created with missing files: %s' %
+                        '; '.join(names))
+        else:
+            self.report({'INFO'}, 'Flamenco job created.')
+        self.quit()
+
+    def quit(self):
+        super().quit()
+        bpy.context.window_manager.progress_end()
+
+    async def bam_pack(self, filepath: Path) -> (typing.Optional[Path], typing.List[Path]):
+        """BAM-packs the blendfile to the destination directory.
+
+        Returns the path of the destination blend file.
+
+        :param filepath: the blend file to pack (i.e. the current blend file)
+        :returns: the destination blend file, or None if there were errors BAM-packing,
+            and a list of missing paths.
+        """
+
+        from datetime import datetime
+        from ..blender import preferences
+        from . import bam_interface
+
+        prefs = preferences()
+
+        # Create a unique directory that is still more or less identifyable.
+        # This should work better than a random ID.
+        # BAM doesn't like output directories that end in '.blend'.
+        unique_dir = '%s-%s-%s' % (datetime.now().isoformat('-').replace(':', ''),
+                                   self.db_user['username'],
+                                   filepath.stem)
+        outdir = Path(prefs.flamenco_job_file_path) / unique_dir
+        outfile = outdir / filepath.name
+
+        try:
+            outdir.mkdir(parents=True)
+        except Exception as ex:
+            self.log.exception('Unable to create output path %s', outdir)
+            self.report({'ERROR'}, 'Unable to create output path: %s' % ex)
+            self.quit()
+            return None, []
+
+        try:
+            missing_sources = await bam_interface.bam_copy(filepath, outfile)
+        except bam_interface.CommandExecutionError as ex:
+            self.log.exception('Unable to execute BAM pack')
+            self.report({'ERROR'}, 'Unable to execute BAM pack: %s' % ex)
+            self.quit()
+            return None, []
+
+        return outfile, missing_sources
+
+
+class FLAMENCO_OT_scene_to_frame_range(Operator):
+    """Sets the scene frame range as the Flamenco render frame range."""
+    bl_idname = 'flamenco.scene_to_frame_range'
+    bl_label = 'Sets the scene frame range as the Flamenco render frame range'
+    bl_description = __doc__.rstrip('.')
+
+    def execute(self, context):
+        s = context.scene
+        s.flamenco_render_frame_range = '%i-%i' % (s.frame_start, s.frame_end)
+        return {'FINISHED'}
+
+
+class FLAMENCO_OT_copy_files(Operator,
+                             async_loop.AsyncModalOperatorMixin):
+    """Uses BAM to copy the current blendfile + dependencies to the target directory."""
+    bl_idname = 'flamenco.copy_files'
+    bl_label = 'Copy files to target'
+    bl_description = __doc__.rstrip('.')
+
+    stop_upon_exception = True
+
+    async def async_execute(self, context):
+        from pathlib import Path
+        from . import bam_interface
+        from ..blender import preferences
+
+        missing_sources = await bam_interface.bam_copy(
+            Path(context.blend_data.filepath),
+            Path(preferences().flamenco_job_file_path),
+        )
+
+        if missing_sources:
+            names = (ms.name for ms in missing_sources)
+            self.report({'ERROR'}, 'Missing source files: %s' % '; '.join(names))
+
+        self.quit()
+
+
+class FLAMENCO_OT_explore_file_path(Operator):
+    """Opens the Flamenco job storage path in a file explorer."""
+    bl_idname = 'flamenco.explore_file_path'
+    bl_label = 'Open in file explorer'
+    bl_description = __doc__.rstrip('.')
+
+    path = StringProperty(name='Path', description='Path to explore', subtype='DIR_PATH')
+
+    def execute(self, context):
+        import platform
+        import subprocess
+        import os
+
+        if platform.system() == "Windows":
+            os.startfile(self.path)
+        elif platform.system() == "Darwin":
+            subprocess.Popen(["open", self.path])
+        else:
+            subprocess.Popen(["xdg-open", self.path])
+
+        return {'FINISHED'}
+
+
+async def create_job(user_id: str,
+                     project_id: str,
+                     manager_id: str,
+                     job_type: str,
+                     job_settings: dict,
+                     job_name: str = None,
+                     *,
+                     priority: int = 50,
+                     job_description: str = None) -> dict:
+    """Creates a render job at Flamenco Server, returning the job object as dictionary."""
+
+    import json
+    from .sdk import Job
+    from ..pillar import pillar_call
+
+    job_attrs = {
+        'status': 'queued',
+        'priority': priority,
+        'name': job_name,
+        'settings': job_settings,
+        'job_type': job_type,
+        'user': user_id,
+        'manager': manager_id,
+        'project': project_id,
+    }
+    if job_description:
+        job_attrs['description'] = job_description
+
+    log.info('Going to create Flamenco job:\n%s',
+             json.dumps(job_attrs, indent=4, sort_keys=True))
+
+    job = Job(job_attrs)
+    await pillar_call(job.create)
+
+    log.info('Job created succesfully: %s', job._id)
+    return job.to_dict()
+
+
+def is_image_type(render_output_type: str) -> bool:
+    """Determines whether the render output type is an image (True) or video (False)."""
+
+    # This list is taken from rna_scene.c:273, rna_enum_image_type_items.
+    video_types = {'AVI_JPEG', 'AVI_RAW', 'FRAMESERVER', 'FFMPEG', 'QUICKTIME'}
+    return render_output_type not in video_types
+
+
+@functools.lru_cache(1)
+def _render_output_path(
+        local_project_path: str,
+        blend_filepath: str,
+        flamenco_job_output_strip_components: int,
+        flamenco_job_output_path: str,
+        render_image_format: str,
+        flamenco_render_frame_range: str,
+) -> typing.Optional[PurePath]:
+    """Cached version of render_output_path()
+
+    This ensures that redraws of the Flamenco Render and Add-on preferences panels
+    is fast.
+    """
+
+    project_path = Path(bpy.path.abspath(local_project_path)).resolve()
+    blendfile = Path(blend_filepath)
+
+    try:
+        proj_rel = blendfile.parent.relative_to(project_path)
+    except ValueError:
+        log.exception('Current file is outside of project path %s', project_path)
+        return None
+
+    rel_parts = proj_rel.parts[flamenco_job_output_strip_components:]
+    output_top = Path(flamenco_job_output_path)
+    dir_components = output_top.joinpath(*rel_parts) / blendfile.stem
+
+    # Blender will have to append the file extensions by itself.
+    if is_image_type(render_image_format):
+        return dir_components / '#####'
+    return dir_components / flamenco_render_frame_range
+
+
+def render_output_path(context) -> typing.Optional[PurePath]:
+    """Returns the render output path to be sent to Flamenco.
+
+    Returns None when the current blend file is outside the project path.
+    """
+
+    from ..blender import preferences
+
+    scene = context.scene
+    prefs = preferences()
+
+    return _render_output_path(
+        prefs.attract_project_local_path,
+        context.blend_data.filepath,
+        prefs.flamenco_job_output_strip_components,
+        prefs.flamenco_job_output_path,
+        scene.render.image_settings.file_format,
+        scene.flamenco_render_frame_range,
+    )
+
+
+class FLAMENCO_PT_render(bpy.types.Panel):
+    bl_label = "Flamenco Render"
+    bl_space_type = 'PROPERTIES'
+    bl_region_type = 'WINDOW'
+    bl_context = "render"
+    bl_options = {'DEFAULT_CLOSED'}
+
+    def draw(self, context):
+        layout = self.layout
+
+        from ..blender import preferences
+
+        prefs = preferences()
+
+        layout.prop(context.scene, 'flamenco_render_job_priority')
+        layout.prop(context.scene, 'flamenco_render_chunk_size')
+
+        labeled_row = layout.split(0.2, align=True)
+        labeled_row.label('Job type:')
+        labeled_row.prop(context.scene, 'flamenco_render_job_type', text='')
+
+        labeled_row = layout.split(0.2, align=True)
+        labeled_row.label('Frame range:')
+        prop_btn_row = labeled_row.row(align=True)
+        prop_btn_row.prop(context.scene, 'flamenco_render_frame_range', text='')
+        prop_btn_row.operator('flamenco.scene_to_frame_range', text='', icon='ARROW_LEFTRIGHT')
+
+        readonly_stuff = layout.column(align=True)
+        labeled_row = readonly_stuff.split(0.2, align=True)
+        labeled_row.label('Storage:')
+        prop_btn_row = labeled_row.row(align=True)
+        prop_btn_row.label(prefs.flamenco_job_file_path)
+        props = prop_btn_row.operator(FLAMENCO_OT_explore_file_path.bl_idname,
+                                      text='', icon='DISK_DRIVE')
+        props.path = prefs.flamenco_job_file_path
+
+        labeled_row = readonly_stuff.split(0.2, align=True)
+        labeled_row.label('Output:')
+        prop_btn_row = labeled_row.row(align=True)
+        render_output = render_output_path(context)
+        prop_btn_row.label(str(render_output))
+        props = prop_btn_row.operator(FLAMENCO_OT_explore_file_path.bl_idname,
+                                      text='', icon='DISK_DRIVE')
+        props.path = str(render_output.parent)
+
+        layout.operator(FLAMENCO_OT_render.bl_idname,
+                        text='Render on Flamenco',
+                        icon='RENDER_ANIMATION')
+
+
+def register():
+    bpy.utils.register_class(FlamencoManagerGroup)
+    bpy.utils.register_class(FLAMENCO_OT_fmanagers)
+    bpy.utils.register_class(FLAMENCO_OT_render)
+    bpy.utils.register_class(FLAMENCO_OT_scene_to_frame_range)
+    bpy.utils.register_class(FLAMENCO_OT_copy_files)
+    bpy.utils.register_class(FLAMENCO_OT_explore_file_path)
+    bpy.utils.register_class(FLAMENCO_PT_render)
+
+    scene = bpy.types.Scene
+    scene.flamenco_render_chunk_size = IntProperty(
+        name='Chunk size',
+        description='Maximum number of frames to render per task',
+        default=10,
+    )
+    scene.flamenco_render_frame_range = StringProperty(
+        name='Frame range',
+        description='Frames to render, in "printer range" notation'
+    )
+    scene.flamenco_render_job_type = EnumProperty(
+        name='Job type',
+        items=[
+            ('blender-render', 'Simple Blender render', 'Not tiled, not resumable, just render'),
+        ],
+        description='Flamenco render job type',
+    )
+    scene.flamenco_render_job_priority = IntProperty(
+        name='Job priority',
+        min=0,
+        default=50,
+        max=100,
+        description='Higher numbers mean higher priority'
+    )
+
+
+def unregister():
+    bpy.utils.unregister_module(__name__)
+
+    try:
+        del bpy.types.Scene.flamenco_render_chunk_size
+    except AttributeError:
+        pass
+    try:
+        del bpy.types.Scene.flamenco_render_frame_range
+    except AttributeError:
+        pass
+    try:
+        del bpy.types.Scene.flamenco_render_job_type
+    except AttributeError:
+        pass
+    try:
+        del bpy.types.Scene.flamenco_render_job_priority
+    except AttributeError:
+        pass

blender_cloud/flamenco/bam_interface.py

@@ -0,0 +1,144 @@
+"""BAM packing interface for Flamenco."""
+
+import logging
+from pathlib import Path
+import typing
+
+# Timeout of the BAM subprocess, in seconds.
+SUBPROC_READLINE_TIMEOUT = 600
+log = logging.getLogger(__name__)
+
+
+class CommandExecutionError(Exception):
+    """Raised when there was an error executing a BAM command."""
+    pass
+
+
+async def bam_copy(base_blendfile: Path, target_blendfile: Path) -> typing.List[Path]:
+    """Uses BAM to copy the given file and dependencies to the target blendfile.
+
+    Due to the way blendfile_pack.py is programmed/structured, we cannot import it
+    and call a function; it has to be run in a subprocess.
+
+    :raises: asyncio.CanceledError if the task was cancelled.
+    :raises: asyncio.TimeoutError if reading a line from the BAM process timed out.
+    :raises: CommandExecutionError if the subprocess failed or output invalid UTF-8.
+    :returns: a list of missing sources; hopefully empty.
+    """
+
+    import asyncio
+    import shlex
+    import subprocess
+
+    import bpy
+    import io_blend_utils
+
+    args = [
+        bpy.app.binary_path_python,
+        '-m', 'bam.pack',
+        '--input', str(base_blendfile),
+        '--output', str(target_blendfile),
+        '--mode', 'FILE',
+    ]
+
+    cmd_to_log = ' '.join(shlex.quote(s) for s in args)
+    log.info('Executing %s', cmd_to_log)
+
+    proc = await asyncio.create_subprocess_exec(
+        *args,
+        env={'PYTHONPATH': io_blend_utils.pythonpath()},
+        stdin=subprocess.DEVNULL,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+    )
+
+    missing_sources = []
+
+    try:
+        while not proc.stdout.at_eof():
+            line = await asyncio.wait_for(proc.stdout.readline(),
+                                          SUBPROC_READLINE_TIMEOUT)
+            if not line:
+                # EOF received, so let's bail.
+                break
+
+            try:
+                line = line.decode('utf8')
+            except UnicodeDecodeError as ex:
+                raise CommandExecutionError('Command produced non-UTF8 output, '
+                                            'aborting: %s' % ex)
+
+            line = line.rstrip()
+            if 'source missing:' in line:
+                path = parse_missing_source(line)
+                missing_sources.append(path)
+                log.warning('Source is missing: %s', path)
+
+            log.info('  %s', line)
+    finally:
+        if proc.returncode is None:
+            # Always wait for the process, to avoid zombies.
+            try:
+                proc.kill()
+            except ProcessLookupError:
+                # The process is already stopped, so killing is impossible. That's ok.
+                log.debug("The process was already stopped, aborting is impossible. That's ok.")
+            await proc.wait()
+        log.info('The process stopped with status code %i', proc.returncode)
+
+    if proc.returncode:
+        raise CommandExecutionError('Process stopped with status %i' % proc.returncode)
+
+    return missing_sources
+
+
+def parse_missing_source(line: str) -> Path:
+    r"""Parses a "missing source" line into a pathlib.Path.
+
+    >>> parse_missing_source(r"  source missing: b'D\xc3\xaffficult \xc3\x9cTF-8 filename'")
+    PosixPath('Dïfficult ÜTF-8 filename')
+    >>> parse_missing_source(r"  source missing: b'D\xfffficult Win1252 f\xeflen\xe6me'")
+    PosixPath('D<EFBFBD>fficult Win1252 f<>len<65>me')
+    """
+
+    _, missing_source = line.split(': ', 1)
+    missing_source_as_bytes = parse_byte_literal(missing_source.strip())
+
+    # The file could originate from any platform, so UTF-8 and the current platform's
+    # filesystem encodings are just guesses.
+    try:
+        missing_source = missing_source_as_bytes.decode('utf8')
+    except UnicodeDecodeError:
+        import sys
+        try:
+            missing_source = missing_source_as_bytes.decode(sys.getfilesystemencoding())
+        except UnicodeDecodeError:
+            missing_source = missing_source_as_bytes.decode('ascii', errors='replace')
+
+    path = Path(missing_source)
+
+    return path
+
+
+def parse_byte_literal(bytes_literal: str) -> bytes:
+    r"""Parses a repr(bytes) output into a bytes object.
+
+    >>> parse_byte_literal(r"b'D\xc3\xaffficult \xc3\x9cTF-8 filename'")
+    b'D\xc3\xaffficult \xc3\x9cTF-8 filename'
+    >>> parse_byte_literal(r"b'D\xeffficult Win1252 f\xeflen\xe6me'")
+    b'D\xeffficult Win1252 f\xeflen\xe6me'
+    """
+
+    # Some very basic assertions to make sure we have a proper bytes literal.
+    assert bytes_literal[0] == "b"
+    assert bytes_literal[1] in {'"', "'"}
+    assert bytes_literal[-1] == bytes_literal[1]
+
+    import ast
+    return ast.literal_eval(bytes_literal)
+
+
+if __name__ == '__main__':
+    import doctest
+
+    doctest.testmod()

blender_cloud/flamenco/sdk.py

@@ -0,0 +1,13 @@
+from pillarsdk.resource import List, Find, Create
+
+
+class Manager(List, Find):
+    """Manager class wrapping the REST nodes endpoint"""
+    path = 'flamenco/managers'
+
+
+class Job(List, Find, Create):
+    """Job class wrapping the REST nodes endpoint
+    """
+    path = 'flamenco/jobs'
+    ensure_query_projections = {'project': 1}

blender_cloud/gui.py

@@ -1,671 +0,0 @@
-# ##### BEGIN GPL LICENSE BLOCK #####
-#
-#  Copyright (C) 2014 Blender Aid
-#  http://www.blendearaid.com
-#  blenderaid@gmail.com
-
-#  This program is free software: you can redistribute it and/or modify
-#  it under the terms of the GNU General Public License as published by
-#  the Free Software Foundation, either version 3 of the License, or
-#  (at your option) any later version.
-
-#  This program is distributed in the hope that it will be useful,
-#  but WITHOUT ANY WARRANTY; without even the implied warranty of
-#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#  GNU General Public License for more details.
-
-#  You should have received a copy of the GNU General Public License
-#  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-#
-# ##### END GPL LICENSE BLOCK #####
-import asyncio
-import logging
-import threading
-
-import bpy
-import bgl
-import blf
-import os
-
-from bpy.types import AddonPreferences
-from bpy.props import (BoolProperty, EnumProperty,
-                       FloatProperty, FloatVectorProperty,
-                       IntProperty, StringProperty)
-
-import pillarsdk
-from . import async_loop, pillar, cache
-
-icon_width = 128
-icon_height = 128
-target_item_width = 400
-target_item_height = 128
-
-library_path = '/tmp'
-library_icons_path = os.path.join(os.path.dirname(__file__), "icons")
-
-
-class UpNode(pillarsdk.Node):
-    def __init__(self):
-        super().__init__()
-        self['_id'] = 'UP'
-        self['node_type'] = 'UP'
-
-
-class MenuItem:
-    """GUI menu item for the 3D View GUI."""
-
-    icon_margin_x = 4
-    icon_margin_y = 4
-    text_margin_x = 6
-
-    text_height = 16
-    text_width = 72
-
-    DEFAULT_ICONS = {
-        'FOLDER': os.path.join(library_icons_path, 'folder.png'),
-        'SPINNER': os.path.join(library_icons_path, 'spinner.png'),
-    }
-
-    SUPPORTED_NODE_TYPES = {'UP', 'group_texture', 'texture'}
-
-    def __init__(self, node, file_desc, thumb_path: str, label_text):
-        if node['node_type'] not in self.SUPPORTED_NODE_TYPES:
-            raise TypeError('Node of type %r not supported; supported are %r.' % (
-                node.group_texture, self.SUPPORTED_NODE_TYPES))
-
-        self.node = node  # pillarsdk.Node, contains 'node_type' key to indicate type
-        self.file_desc = file_desc  # pillarsdk.File object, or None if a 'folder' node.
-        self.label_text = label_text
-        self._thumb_path = ''
-        self.icon = None
-        self._is_folder = node['node_type'] == 'group_texture' or isinstance(node, UpNode)
-
-        self.thumb_path = thumb_path
-
-        # Updated when drawing the image
-        self.x = 0
-        self.y = 0
-        self.width = 0
-        self.height = 0
-
-    @property
-    def thumb_path(self) -> str:
-        return self._thumb_path
-
-    @thumb_path.setter
-    def thumb_path(self, new_thumb_path: str):
-        self._thumb_path = self.DEFAULT_ICONS.get(new_thumb_path, new_thumb_path)
-        if self._thumb_path:
-            self.icon = bpy.data.images.load(filepath=self._thumb_path)
-        else:
-            self.icon = None
-
-    @property
-    def node_uuid(self) -> str:
-        return self.node['_id']
-
-    def update(self, node, file_desc, thumb_path: str, label_text):
-        # We can get updated information about our Node, but a MenuItem should
-        # always represent one node, and it shouldn't be shared between nodes.
-        if self.node_uuid != node['_id']:
-            raise ValueError("Don't change the node ID this MenuItem reflects, "
-                             "just create a new one.")
-        self.node = node
-        self.file_desc = file_desc  # pillarsdk.File object, or None if a 'folder' node.
-        self.thumb_path = thumb_path
-        self.label_text = label_text
-
-    @property
-    def is_folder(self) -> bool:
-        return self._is_folder
-
-    def update_placement(self, x, y, width, height):
-        """Use OpenGL to draw this one menu item."""
-
-        self.x = x
-        self.y = y
-        self.width = width
-        self.height = height
-
-    def draw(self, highlighted: bool):
-        bgl.glEnable(bgl.GL_BLEND)
-        if highlighted:
-            bgl.glColor4f(0.555, 0.555, 0.555, 0.8)
-        else:
-            bgl.glColor4f(0.447, 0.447, 0.447, 0.8)
-
-        bgl.glRectf(self.x, self.y, self.x + self.width, self.y + self.height)
-
-        texture = self.icon
-        err = texture.gl_load(filter=bgl.GL_NEAREST, mag=bgl.GL_NEAREST)
-        assert not err, 'OpenGL error: %i' % err
-
-        bgl.glColor4f(0.0, 0.0, 1.0, 0.5)
-        # bgl.glLineWidth(1.5)
-
-        # ------ TEXTURE ---------#
-        bgl.glBindTexture(bgl.GL_TEXTURE_2D, texture.bindcode[0])
-        bgl.glEnable(bgl.GL_TEXTURE_2D)
-        bgl.glBlendFunc(bgl.GL_SRC_ALPHA, bgl.GL_ONE_MINUS_SRC_ALPHA)
-
-        bgl.glColor4f(1, 1, 1, 1)
-        bgl.glBegin(bgl.GL_QUADS)
-        bgl.glTexCoord2d(0, 0)
-        bgl.glVertex2d(self.x + self.icon_margin_x, self.y)
-        bgl.glTexCoord2d(0, 1)
-        bgl.glVertex2d(self.x + self.icon_margin_x, self.y + icon_height)
-        bgl.glTexCoord2d(1, 1)
-        bgl.glVertex2d(self.x + self.icon_margin_x + icon_width, self.y + icon_height)
-        bgl.glTexCoord2d(1, 0)
-        bgl.glVertex2d(self.x + self.icon_margin_x + icon_width, self.y)
-        bgl.glEnd()
-        bgl.glDisable(bgl.GL_TEXTURE_2D)
-        bgl.glDisable(bgl.GL_BLEND)
-
-        texture.gl_free()
-
-        # draw some text
-        font_id = 0
-        blf.position(font_id,
-                     self.x + self.icon_margin_x + icon_width + self.text_margin_x,
-                     self.y + icon_height * 0.5 - 0.25 * self.text_height, 0)
-        blf.size(font_id, self.text_height, self.text_width)
-        blf.draw(font_id, self.label_text)
-
-    def hits(self, mouse_x: int, mouse_y: int) -> bool:
-        return self.x < mouse_x < self.x + self.width and self.y < mouse_y < self.y + self.height
-
-
-class BlenderCloudBrowser(bpy.types.Operator):
-    bl_idname = 'pillar.browser'
-    bl_label = 'Blender Cloud Texture Browser'
-
-    _draw_handle = None
-
-    _state = 'BROWSING'
-
-    project_uuid = '5672beecc0261b2005ed1a33'  # Blender Cloud project UUID
-    node = None  # The Node object we're currently showing, or None if we're at the project top.
-    node_uuid = ''  # Blender Cloud node UUID we're currently showing, i.e. None-safe self.node['_id']
-
-    # This contains a stack of Node objects that lead up to the currently browsed node.
-    # This allows us to display the "up" item.
-    path_stack = []
-
-    async_task = None  # asyncio task for fetching thumbnails
-    signalling_future = None  # asyncio future for signalling that we want to cancel everything.
-    timer = None
-    log = logging.getLogger('%s.BlenderCloudBrowser' % __name__)
-
-    _menu_item_lock = threading.Lock()
-    current_path = ''
-    current_display_content = []
-    loaded_images = set()
-    thumbnails_cache = ''
-    maximized_area = False
-
-    mouse_x = 0
-    mouse_y = 0
-
-    def invoke(self, context, event):
-        wm = context.window_manager
-        self.project_uuid = wm.blender_cloud_project
-        self.node_uuid = wm.blender_cloud_node
-        self.path_stack = []
-
-        self.thumbnails_cache = cache.cache_directory('thumbnails')
-        self.mouse_x = event.mouse_x
-        self.mouse_y = event.mouse_y
-
-        # See if we have to maximize the current area
-        if not context.screen.show_fullscreen:
-            self.maximized_area = True
-            bpy.ops.screen.screen_full_area(use_hide_panels=True)
-
-        # Add the region OpenGL drawing callback
-        # draw in view space with 'POST_VIEW' and 'PRE_VIEW'
-        self._draw_handle = context.space_data.draw_handler_add(
-            self.draw_menu, (context,), 'WINDOW', 'POST_PIXEL')
-
-        self.current_display_content = []
-        self.loaded_images = set()
-        self.browse_assets()
-
-        context.window_manager.modal_handler_add(self)
-        self.timer = context.window_manager.event_timer_add(1 / 30, context.window)
-
-        return {'RUNNING_MODAL'}
-
-    def modal(self, context, event):
-        task = self.async_task
-        if self._state != 'EXCEPTION' and task.done() and not task.cancelled():
-            ex = task.exception()
-            if ex is not None:
-                self._state = 'EXCEPTION'
-                self.log.error('Exception while running task: %s', ex)
-                return {'RUNNING_MODAL'}
-
-        if self._state == 'QUIT':
-            self._finish(context)
-            return {'FINISHED'}
-
-        if event.type == 'TAB' and event.value == 'RELEASE':
-            self.log.info('Ensuring async loop is running')
-            async_loop.ensure_async_loop()
-
-        if event.type == 'TIMER':
-            context.area.tag_redraw()
-            return {'RUNNING_MODAL'}
-
-        if 'MOUSE' in event.type:
-            context.area.tag_redraw()
-            self.mouse_x = event.mouse_x
-            self.mouse_y = event.mouse_y
-
-        if self._state == 'BROWSING' and event.type == 'LEFTMOUSE' and event.value == 'RELEASE':
-            selected = self.get_clicked()
-
-            if selected is None:
-                # No item clicked, ignore it.
-                return {'RUNNING_MODAL'}
-
-            if selected.is_folder:
-                self.descend_node(selected.node)
-            else:
-                if selected.file_desc is None:
-                    # This can happen when the thumbnail information isn't loaded yet.
-                    # Just ignore the click for now.
-                    # TODO: think of a way to handle this properly.
-                    return {'RUNNING_MODAL'}
-                self.handle_item_selection(context, selected)
-
-        elif event.type in {'RIGHTMOUSE', 'ESC'}:
-            self._finish(context)
-            return {'CANCELLED'}
-
-        return {'RUNNING_MODAL'}
-
-    def descend_node(self, node):
-        """Descends the node hierarchy by visiting this node.
-
-        Also keeps track of the current node, so that we know where the "up" button should go.
-        """
-
-        # Going up or down?
-        if self.path_stack and isinstance(node, UpNode):
-            self.log.debug('Going up, pop the stack; pre-pop stack is %r', self.path_stack)
-            node = self.path_stack.pop()
-
-        else:
-            # Going down, keep track of where we were (project top-level is None)
-            self.path_stack.append(self.node)
-            self.log.debug('Going up, push the stack; post-push stack is %r', self.path_stack)
-
-        # Set 'current' to the given node
-        self.node_uuid = node['_id'] if node else None
-        self.node = node
-        self.browse_assets()
-
-    def _stop_async_task(self):
-        self.log.debug('Stopping async task')
-        if self.async_task is None:
-            self.log.debug('No async task, trivially stopped')
-            return
-
-        # Signal that we want to stop.
-        if not self.signalling_future.done():
-            self.log.info("Signalling that we want to cancel anything that's running.")
-            self.signalling_future.cancel()
-
-        # Wait until the asynchronous task is done.
-        if not self.async_task.done():
-            self.log.info("blocking until async task is done.")
-            loop = asyncio.get_event_loop()
-            try:
-                loop.run_until_complete(self.async_task)
-            except asyncio.CancelledError:
-                self.log.info('Asynchronous task was cancelled')
-                return
-
-        # noinspection PyBroadException
-        try:
-            self.async_task.result()  # This re-raises any exception of the task.
-        except asyncio.CancelledError:
-            self.log.info('Asynchronous task was cancelled')
-        except Exception:
-            self.log.exception("Exception from asynchronous task")
-
-    def _finish(self, context):
-        self.log.debug('Finishing the modal operator')
-        self._stop_async_task()
-        self.clear_images()
-
-        context.space_data.draw_handler_remove(self._draw_handle, 'WINDOW')
-        context.window_manager.event_timer_remove(self.timer)
-
-        if self.maximized_area:
-            bpy.ops.screen.screen_full_area(use_hide_panels=True)
-
-        context.area.tag_redraw()
-        self.log.debug('Modal operator finished')
-
-    def clear_images(self):
-        """Removes all images we loaded from Blender's memory."""
-
-        for image in bpy.data.images:
-            if image.filepath_raw not in self.loaded_images:
-                continue
-
-            image.user_clear()
-            bpy.data.images.remove(image)
-
-        self.loaded_images.clear()
-        self.current_display_content.clear()
-
-    def add_menu_item(self, *args) -> MenuItem:
-        menu_item = MenuItem(*args)
-
-        # Just make this thread-safe to be on the safe side.
-        with self._menu_item_lock:
-            self.current_display_content.append(menu_item)
-            self.loaded_images.add(menu_item.icon.filepath_raw)
-
-        return menu_item
-
-    def update_menu_item(self, node, *args) -> MenuItem:
-        node_uuid = node['_id']
-
-        # Just make this thread-safe to be on the safe side.
-        with self._menu_item_lock:
-            for menu_item in self.current_display_content:
-                if menu_item.node_uuid == node_uuid:
-                    menu_item.update(node, *args)
-                    self.loaded_images.add(menu_item.icon.filepath_raw)
-                    break
-            else:
-                raise ValueError('Unable to find MenuItem(node_uuid=%r)' % node_uuid)
-
-    async def async_download_previews(self, thumbnails_directory):
-        self.log.info('Asynchronously downloading previews to %r', thumbnails_directory)
-        self.clear_images()
-
-        def thumbnail_loading(node, texture_node):
-            self.add_menu_item(node, None, 'SPINNER', texture_node['name'])
-
-        def thumbnail_loaded(node, file_desc, thumb_path):
-            self.update_menu_item(node, file_desc, thumb_path, file_desc['filename'])
-
-        # Download either by group_texture node UUID or by project UUID (which shows all top-level nodes)
-        if self.node_uuid:
-            self.log.debug('Getting subnodes for parent node %r', self.node_uuid)
-            children = await pillar.get_nodes(parent_node_uuid=self.node_uuid,
-                                              node_type='group_textures')
-
-            # Make sure we can go up again.
-            if self.path_stack:
-                self.add_menu_item(UpNode(), None, 'FOLDER', '.. up ..')
-        elif self.project_uuid:
-            self.log.debug('Getting subnodes for project node %r', self.project_uuid)
-            children = await pillar.get_nodes(self.project_uuid, '')
-
-        else:
-            # TODO: add "nothing here" icon and trigger re-draw
-            self.log.warning("Not node UUID and no project UUID, I can't do anything!")
-            return
-
-        # Download all child nodes
-        self.log.debug('Iterating over child nodes of %r', self.node_uuid)
-        for child in children:
-            # print('  - %(_id)s = %(name)s' % child)
-            self.add_menu_item(child, None, 'FOLDER', child['name'])
-
-        # There are only sub-nodes at the project level, no texture nodes,
-        # so we won't have to bother looking for textures.
-        if not self.node_uuid:
-            return
-
-        directory = os.path.join(thumbnails_directory, self.project_uuid, self.node_uuid)
-        os.makedirs(directory, exist_ok=True)
-
-        self.log.debug('Fetching texture thumbnails for node %r', self.node_uuid)
-        await pillar.fetch_texture_thumbs(self.node_uuid, 's', directory,
-                                          thumbnail_loading=thumbnail_loading,
-                                          thumbnail_loaded=thumbnail_loaded,
-                                          future=self.signalling_future)
-
-    def browse_assets(self):
-        self._state = 'BROWSING'
-        self.log.debug('Browsing assets at project %r node %r', self.project_uuid, self.node_uuid)
-        self._new_async_task(self.async_download_previews(self.thumbnails_cache))
-
-    def _new_async_task(self, async_task: asyncio.coroutine, future: asyncio.Future=None):
-        """Stops the currently running async task, and starts another one."""
-
-        self.log.debug('Setting up a new task %r, so any existing task must be stopped', async_task)
-        self._stop_async_task()
-
-        # Download the previews asynchronously.
-        self.signalling_future = future or asyncio.Future()
-        self.async_task = asyncio.ensure_future(async_task)
-        self.log.debug('Created new task %r', self.async_task)
-
-        # Start the async manager so everything happens.
-        async_loop.ensure_async_loop()
-
-    def draw_menu(self, context):
-        """Draws the GUI with OpenGL."""
-
-        drawers = {
-            'BROWSING': self._draw_browser,
-            'DOWNLOADING_TEXTURE': self._draw_downloading,
-            'EXCEPTION': self._draw_exception,
-        }
-
-        if self._state in drawers:
-            drawer = drawers[self._state]
-            drawer(context)
-
-        # For debugging: draw the state
-        font_id = 0
-        bgl.glColor4f(1.0, 1.0, 1.0, 1.0)
-        blf.size(font_id, 20, 72)
-        blf.position(font_id, 5, 5, 0)
-        blf.draw(font_id, self._state)
-        bgl.glDisable(bgl.GL_BLEND)
-
-    @staticmethod
-    def _window_region(context):
-        window_regions = [region
-                          for region in context.area.regions
-                          if region.type == 'WINDOW']
-        return window_regions[0]
-
-    def _draw_browser(self, context):
-        """OpenGL drawing code for the BROWSING state."""
-
-        margin_x = 5
-        margin_y = 5
-        padding_x = 5
-
-        window_region = self._window_region(context)
-        content_width = window_region.width - margin_x * 2
-        content_height = window_region.height - margin_y * 2
-
-        content_x = margin_x
-        content_y = context.area.height - margin_y - target_item_height
-
-        col_count = content_width // target_item_width
-
-        item_width = (content_width - (col_count * padding_x)) / col_count
-        item_height = target_item_height
-
-        block_width = item_width + padding_x
-        block_height = item_height + margin_y
-
-        bgl.glEnable(bgl.GL_BLEND)
-        bgl.glColor4f(0.0, 0.0, 0.0, 0.6)
-        bgl.glRectf(0, 0, window_region.width, window_region.height)
-
-        if self.current_display_content:
-            for item_idx, item in enumerate(self.current_display_content):
-                x = content_x + (item_idx % col_count) * block_width
-                y = content_y - (item_idx // col_count) * block_height
-
-                item.update_placement(x, y, item_width, item_height)
-                item.draw(highlighted=item.hits(self.mouse_x, self.mouse_y))
-        else:
-            font_id = 0
-            text = "Communicating with Blender Cloud"
-            bgl.glColor4f(1.0, 1.0, 1.0, 1.0)
-            blf.size(font_id, 20, 72)
-            text_width, text_height = blf.dimensions(font_id, text)
-            blf.position(font_id,
-                         content_x + content_width * 0.5 - text_width * 0.5,
-                         content_y - content_height * 0.3 + text_height * 0.5, 0)
-            blf.draw(font_id, text)
-
-        bgl.glDisable(bgl.GL_BLEND)
-        # bgl.glColor4f(0.0, 0.0, 0.0, 1.0)
-
-    def _draw_downloading(self, context):
-        """OpenGL drawing code for the DOWNLOADING_TEXTURE state."""
-
-        content_height, content_width = self._window_size(context)
-
-        bgl.glEnable(bgl.GL_BLEND)
-        bgl.glColor4f(0.0, 0.0, 0.2, 0.6)
-        bgl.glRectf(0, 0, content_width, content_height)
-
-        font_id = 0
-        text = "Downloading texture from Blender Cloud"
-        bgl.glColor4f(1.0, 1.0, 1.0, 1.0)
-        blf.size(font_id, 20, 72)
-        text_width, text_height = blf.dimensions(font_id, text)
-
-        blf.position(font_id,
-                     content_width * 0.5 - text_width * 0.5,
-                     content_height * 0.7 + text_height * 0.5, 0)
-        blf.draw(font_id, text)
-        bgl.glDisable(bgl.GL_BLEND)
-
-    def _window_size(self, context):
-        window_region = self._window_region(context)
-        content_width = window_region.width
-        content_height = window_region.height
-        return content_height, content_width
-
-    def _draw_exception(self, context):
-        """OpenGL drawing code for the EXCEPTION state."""
-
-        import textwrap
-
-        content_height, content_width = self._window_size(context)
-
-        bgl.glEnable(bgl.GL_BLEND)
-        bgl.glColor4f(0.2, 0.0, 0.0, 0.6)
-        bgl.glRectf(0, 0, content_width, content_height)
-
-        font_id = 0
-        text = "An error occurred:\n%s" % self.async_task.exception()
-        lines = textwrap.wrap(text)
-
-        bgl.glColor4f(1.0, 1.0, 1.0, 1.0)
-        blf.size(font_id, 20, 72)
-        _, text_height = blf.dimensions(font_id, 'yhBp')
-
-        def position(line_nr):
-            blf.position(font_id,
-                         content_width * 0.1,
-                         content_height * 0.8 - line_nr * text_height, 0)
-
-        for line_idx, line in enumerate(lines):
-            position(line_idx)
-            blf.draw(font_id, line)
-        bgl.glDisable(bgl.GL_BLEND)
-
-    def get_clicked(self) -> MenuItem:
-
-        for item in self.current_display_content:
-            if item.hits(self.mouse_x, self.mouse_y):
-                return item
-
-        return None
-
-    def handle_item_selection(self, context, item: MenuItem):
-        """Called when the user clicks on a menu item that doesn't represent a folder."""
-
-        self.clear_images()
-        self._state = 'DOWNLOADING_TEXTURE'
-
-        node_path_components = [node['name'] for node in self.path_stack if node is not None]
-        local_path_components = [self.project_uuid] + node_path_components + [self.node['name']]
-
-        top_texture_directory = bpy.path.abspath(context.scene.local_texture_dir)
-        local_path = os.path.join(top_texture_directory, *local_path_components)
-        meta_path = os.path.join(top_texture_directory, '.blender_cloud')
-
-        self.log.info('Downloading texture %r to %s', item.node_uuid, local_path)
-        self.log.debug('Metadata will be stored at %s', meta_path)
-
-        file_paths = []
-
-        def texture_downloading(file_path, file_desc, *args):
-            self.log.info('Texture downloading to %s', file_path)
-
-        def texture_downloaded(file_path, file_desc, *args):
-            self.log.info('Texture downloaded to %r.', file_path)
-            image_dblock = bpy.data.images.load(filepath=file_path)
-            image_dblock['bcloud_file_uuid'] = file_desc['_id']
-            image_dblock['bcloud_texture_node_uuid'] = item.node_uuid
-            file_paths.append(file_path)
-
-        def texture_download_completed(_):
-            self.log.info('Texture download complete, inspect:\n%s', '\n'.join(file_paths))
-            self._state = 'QUIT'
-
-        signalling_future = asyncio.Future()
-        self._new_async_task(pillar.download_texture(item.node, local_path,
-                                                     metadata_directory=meta_path,
-                                                     texture_loading=texture_downloading,
-                                                     texture_loaded=texture_downloaded,
-                                                     future=signalling_future))
-        self.async_task.add_done_callback(texture_download_completed)
-
-
-# store keymaps here to access after registration
-addon_keymaps = []
-
-
-def menu_draw(self, context):
-    layout = self.layout
-    layout.separator()
-    layout.operator(BlenderCloudBrowser.bl_idname, icon='MOD_SCREW')
-
-
-def register():
-    bpy.utils.register_class(BlenderCloudBrowser)
-    bpy.types.INFO_MT_mesh_add.append(menu_draw)
-
-    # handle the keymap
-    wm = bpy.context.window_manager
-    kc = wm.keyconfigs.addon
-    if not kc:
-        print('No addon key configuration space found, so no custom hotkeys added.')
-        return
-
-    km = kc.keymaps.new(name='Screen')
-    kmi = km.keymap_items.new('pillar.browser', 'A', 'PRESS', ctrl=True, shift=True, alt=True)
-    addon_keymaps.append((km, kmi))
-
-
-def unregister():
-    bpy.utils.unregister_class(BlenderCloudBrowser)
-
-    # handle the keymap
-    for km, kmi in addon_keymaps:
-        km.keymap_items.remove(kmi)
-    addon_keymaps.clear()
-
-
-if __name__ == "__main__":
-    register()

blender_cloud/home_project.py

@@ -0,0 +1,50 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
+import logging
+
+import pillarsdk
+from pillarsdk import exceptions as sdk_exceptions
+from .pillar import pillar_call
+
+log = logging.getLogger(__name__)
+HOME_PROJECT_ENDPOINT = '/bcloud/home-project'
+
+
+async def get_home_project(params=None) -> pillarsdk.Project:
+    """Returns the home project."""
+
+    log.debug('Getting home project')
+    try:
+        return await pillar_call(pillarsdk.Project.find_from_endpoint,
+                                 HOME_PROJECT_ENDPOINT, params=params)
+    except sdk_exceptions.ForbiddenAccess:
+        log.warning('Access to the home project was denied. '
+                    'Double-check that you are logged in with valid BlenderID credentials.')
+        raise
+    except sdk_exceptions.ResourceNotFound:
+        log.warning('No home project available.')
+        raise
+
+
+async def get_home_project_id() -> str:
+    """Returns just the ID of the home project."""
+
+    home_proj = await get_home_project({'projection': {'_id': 1}})
+    home_proj_id = home_proj['_id']
+    return home_proj_id

blender_cloud/image_sharing.py

@@ -0,0 +1,334 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
+import logging
+import os.path
+import tempfile
+import datetime
+
+import bpy
+import pillarsdk
+from pillarsdk import exceptions as sdk_exceptions
+from .pillar import pillar_call
+from . import async_loop, pillar, home_project, blender
+
+REQUIRES_ROLES_FOR_IMAGE_SHARING = {'subscriber', 'demo'}
+IMAGE_SHARING_GROUP_NODE_NAME = 'Image sharing'
+log = logging.getLogger(__name__)
+
+
+async def find_image_sharing_group_id(home_project_id, user_id):
+    # Find the top-level image sharing group node.
+    try:
+        share_group, created = await pillar.find_or_create_node(
+            where={'project': home_project_id,
+                   'node_type': 'group',
+                   'parent': None,
+                   'name': IMAGE_SHARING_GROUP_NODE_NAME},
+            additional_create_props={
+                'user': user_id,
+                'properties': {},
+            },
+            projection={'_id': 1},
+            may_create=True)
+    except pillar.PillarError:
+        log.exception('Pillar error caught')
+        raise pillar.PillarError('Unable to find image sharing folder on the Cloud')
+
+    return share_group['_id']
+
+
+class PILLAR_OT_image_share(pillar.PillarOperatorMixin,
+                            async_loop.AsyncModalOperatorMixin,
+                            bpy.types.Operator):
+    bl_idname = 'pillar.image_share'
+    bl_label = 'Share an image/screenshot via Blender Cloud'
+    bl_description = 'Uploads an image for sharing via Blender Cloud'
+
+    log = logging.getLogger('bpy.ops.%s' % bl_idname)
+
+    home_project_id = None
+    home_project_url = 'home'
+    share_group_id = None  # top-level share group node ID
+    user_id = None
+
+    target = bpy.props.EnumProperty(
+        items=[
+            ('FILE', 'File', 'Share an image file'),
+            ('DATABLOCK', 'Datablock', 'Share an image datablock'),
+            ('SCREENSHOT', 'Screenshot', 'Share a screenshot'),
+        ],
+        name='target',
+        default='SCREENSHOT')
+
+    name = bpy.props.StringProperty(name='name',
+                                    description='File or datablock name to sync')
+
+    screenshot_show_multiview = bpy.props.BoolProperty(
+        name='screenshot_show_multiview',
+        description='Enable Multi-View',
+        default=False)
+
+    screenshot_use_multiview = bpy.props.BoolProperty(
+        name='screenshot_use_multiview',
+        description='Use Multi-View',
+        default=False)
+
+    screenshot_full = bpy.props.BoolProperty(
+        name='screenshot_full',
+        description='Full Screen, Capture the whole window (otherwise only capture the active area)',
+        default=False)
+
+    def invoke(self, context, event):
+        # Do a quick test on datablock dirtyness. If it's not packed and dirty,
+        # the user should save it first.
+        if self.target == 'DATABLOCK':
+            if not self.name:
+                self.report({'ERROR'}, 'No name given of the datablock to share.')
+                return {'CANCELLED'}
+
+            datablock = bpy.data.images[self.name]
+            if datablock.type == 'IMAGE' and datablock.is_dirty and not datablock.packed_file:
+                self.report({'ERROR'}, 'Datablock is dirty, save it first.')
+                return {'CANCELLED'}
+
+        return async_loop.AsyncModalOperatorMixin.invoke(self, context, event)
+
+    async def async_execute(self, context):
+        """Entry point of the asynchronous operator."""
+
+        self.report({'INFO'}, 'Communicating with Blender Cloud')
+
+        try:
+            # Refresh credentials
+            try:
+                db_user = await self.check_credentials(context, REQUIRES_ROLES_FOR_IMAGE_SHARING)
+                self.user_id = db_user['_id']
+                self.log.debug('Found user ID: %s', self.user_id)
+            except pillar.NotSubscribedToCloudError:
+                self.log.exception('User not subscribed to cloud.')
+                self.report({'ERROR'}, 'Please subscribe to the Blender Cloud.')
+                self._state = 'QUIT'
+                return
+            except pillar.UserNotLoggedInError:
+                self.log.exception('Error checking/refreshing credentials.')
+                self.report({'ERROR'}, 'Please log in on Blender ID first.')
+                self._state = 'QUIT'
+                return
+
+            # Find the home project.
+            try:
+                home_proj = await home_project.get_home_project({
+                    'projection': {'_id': 1, 'url': 1}
+                })
+            except sdk_exceptions.ForbiddenAccess:
+                self.log.exception('Forbidden access to home project.')
+                self.report({'ERROR'}, 'Did not get access to home project.')
+                self._state = 'QUIT'
+                return
+            except sdk_exceptions.ResourceNotFound:
+                self.report({'ERROR'}, 'Home project not found.')
+                self._state = 'QUIT'
+                return
+
+            self.home_project_id = home_proj['_id']
+            self.home_project_url = home_proj['url']
+
+            try:
+                gid = await find_image_sharing_group_id(self.home_project_id,
+                                                        self.user_id)
+                self.share_group_id = gid
+                self.log.debug('Found group node ID: %s', self.share_group_id)
+            except sdk_exceptions.ForbiddenAccess:
+                self.log.exception('Unable to find Group ID')
+                self.report({'ERROR'}, 'Unable to find sync folder.')
+                self._state = 'QUIT'
+                return
+
+            await self.share_image(context)
+        except Exception as ex:
+            self.log.exception('Unexpected exception caught.')
+            self.report({'ERROR'}, 'Unexpected error %s: %s' % (type(ex), ex))
+
+        self._state = 'QUIT'
+
+    async def share_image(self, context):
+        """Sends files to the Pillar server."""
+
+        if self.target == 'FILE':
+            self.report({'INFO'}, "Uploading %s '%s'" % (self.target.lower(), self.name))
+            node = await self.upload_file(self.name)
+        elif self.target == 'SCREENSHOT':
+            node = await self.upload_screenshot(context)
+        else:
+            self.report({'INFO'}, "Uploading %s '%s'" % (self.target.lower(), self.name))
+            node = await self.upload_datablock(context)
+
+        self.report({'INFO'}, 'Upload complete, creating link to share.')
+        share_info = await pillar_call(node.share)
+        url = share_info.get('short_link')
+        context.window_manager.clipboard = url
+        self.report({'INFO'}, 'The link has been copied to your clipboard: %s' % url)
+
+        await self.maybe_open_browser(url)
+
+    async def upload_file(self, filename: str, fileobj=None) -> pillarsdk.Node:
+        """Uploads a file to the cloud, attached to the image sharing node.
+
+        Returns the node.
+        """
+
+        self.log.info('Uploading file %s', filename)
+        node = await pillar_call(pillarsdk.Node.create_asset_from_file,
+                                 self.home_project_id,
+                                 self.share_group_id,
+                                 'image',
+                                 filename,
+                                 extra_where={'user': self.user_id},
+                                 always_create_new_node=True,
+                                 fileobj=fileobj,
+                                 caching=False)
+        node_id = node['_id']
+        self.log.info('Created node %s', node_id)
+        self.report({'INFO'}, 'File succesfully uploaded to the cloud!')
+
+        return node
+
+    async def maybe_open_browser(self, url):
+        prefs = blender.preferences()
+        if not prefs.open_browser_after_share:
+            return
+
+        import webbrowser
+
+        self.log.info('Opening browser at %s', url)
+        webbrowser.open_new_tab(url)
+
+    async def upload_datablock(self, context) -> pillarsdk.Node:
+        """Saves a datablock to file if necessary, then upload.
+
+        Returns the node.
+        """
+
+        self.log.info("Uploading datablock '%s'" % self.name)
+        datablock = bpy.data.images[self.name]
+
+        if datablock.type == 'RENDER_RESULT':
+            # Construct a sensible name for this render.
+            filename = '%s-%s-render%s' % (
+                os.path.splitext(os.path.basename(context.blend_data.filepath))[0],
+                context.scene.name,
+                context.scene.render.file_extension)
+            return await self.upload_via_tempdir(datablock, filename)
+
+        if datablock.packed_file is not None:
+            return await self.upload_packed_file(datablock)
+
+        if datablock.is_dirty:
+            # We can handle dirty datablocks like this if we want.
+            # However, I (Sybren) do NOT think it's a good idea to:
+            # - Share unsaved data to the cloud; users can assume it's saved
+            #   to disk and close blender, losing their file.
+            # - Save unsaved data first; this can overwrite a file a user
+            #   didn't want to overwrite.
+            filename = bpy.path.basename(datablock.filepath)
+            return await self.upload_via_tempdir(datablock, filename)
+
+        filepath = bpy.path.abspath(datablock.filepath)
+        return await self.upload_file(filepath)
+
+    async def upload_via_tempdir(self, datablock, filename_on_cloud) -> pillarsdk.Node:
+        """Saves the datablock to file, and uploads it to the cloud.
+
+        Saving is done to a temporary directory, which is removed afterwards.
+
+        Returns the node.
+        """
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            filepath = os.path.join(tmpdir, filename_on_cloud)
+            self.log.debug('Saving %s to %s', datablock, filepath)
+            datablock.save_render(filepath)
+            return await self.upload_file(filepath)
+
+    async def upload_packed_file(self, datablock) -> pillarsdk.Node:
+        """Uploads a packed file directly from memory.
+
+        Returns the node.
+        """
+
+        import io
+
+        filename = '%s.%s' % (datablock.name, datablock.file_format.lower())
+        fileobj = io.BytesIO(datablock.packed_file.data)
+        fileobj.seek(0)  # ensure PillarSDK reads the file from the beginning.
+        self.log.info('Uploading packed file directly from memory to %r.', filename)
+        return await self.upload_file(filename, fileobj=fileobj)
+
+    async def upload_screenshot(self, context) -> pillarsdk.Node:
+        """Takes a screenshot, saves it to a temp file, and uploads it."""
+
+        self.name = datetime.datetime.now().strftime('Screenshot-%Y-%m-%d-%H%M%S.png')
+        self.report({'INFO'}, "Uploading %s '%s'" % (self.target.lower(), self.name))
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            filepath = os.path.join(tmpdir, self.name)
+            self.log.debug('Saving screenshot to %s', filepath)
+            bpy.ops.screen.screenshot(filepath=filepath,
+                                      show_multiview=self.screenshot_show_multiview,
+                                      use_multiview=self.screenshot_use_multiview,
+                                      full=self.screenshot_full)
+            return await self.upload_file(filepath)
+
+
+def image_editor_menu(self, context):
+    image = context.space_data.image
+
+    box = self.layout.row()
+    if image and image.has_data:
+        text = 'Share on Blender Cloud'
+        if image.type == 'IMAGE' and image.is_dirty and not image.packed_file:
+            box.enabled = False
+            text = 'Save image before sharing on Blender Cloud'
+
+        props = box.operator(PILLAR_OT_image_share.bl_idname, text=text,
+                             icon_value=blender.icon('CLOUD'))
+        props.target = 'DATABLOCK'
+        props.name = image.name
+
+
+def window_menu(self, context):
+    props = self.layout.operator(PILLAR_OT_image_share.bl_idname,
+                                 text='Share screenshot via Blender Cloud',
+                                 icon_value=blender.icon('CLOUD'))
+    props.target = 'SCREENSHOT'
+    props.screenshot_full = True
+
+
+def register():
+    bpy.utils.register_class(PILLAR_OT_image_share)
+
+    bpy.types.IMAGE_MT_image.append(image_editor_menu)
+    bpy.types.INFO_MT_window.append(window_menu)
+
+
+def unregister():
+    bpy.utils.unregister_class(PILLAR_OT_image_share)
+
+    bpy.types.IMAGE_MT_image.remove(image_editor_menu)
+    bpy.types.INFO_MT_window.remove(window_menu)

blender_cloud/pillar.py

@@ -1,9 +1,29 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
 import asyncio
+import datetime
 import json
 import os
 import functools
 import logging
 from contextlib import closing, contextmanager
+import urllib.parse
 import pathlib
 
 import requests
@@ -15,7 +35,12 @@ from pillarsdk.utils import sanitize_filename
 
 from . import cache
 
-_pillar_api = None  # will become a pillarsdk.Api object.
+SUBCLIENT_ID = 'PILLAR'
+TEXTURE_NODE_TYPES = {'texture', 'hdri'}
+
+RFC1123_DATE_FORMAT = '%a, %d %b %Y %H:%M:%S GMT'
+
+_pillar_api = {}  # will become a mapping from bool (cached/non-cached) to pillarsdk.Api objects.
 log = logging.getLogger(__name__)
 uncached_session = requests.session()
 _testing_blender_id_profile = None  # Just for testing, overrides what is returned by blender_id_profile.
@@ -28,6 +53,17 @@ class UserNotLoggedInError(RuntimeError):
     This is basically for every interaction with Pillar.
     """
 
+    def __str__(self):
+        return self.__class__.__name__
+
+
+class CredentialsNotSyncedError(UserNotLoggedInError):
+    """Raised when the user may be logged in on Blender ID, but has no Blender Cloud token."""
+
+
+class NotSubscribedToCloudError(UserNotLoggedInError):
+    """Raised when the user may be logged in on Blender ID, but has no Blender Cloud token."""
+
 
 class PillarError(RuntimeError):
     """Raised when there is some issue with the communication with Pillar.
@@ -50,6 +86,8 @@ class CloudPath(pathlib.PurePosixPath):
     @property
     def project_uuid(self) -> str:
         assert self.parts[0] == '/'
+        if len(self.parts) <= 1:
+            return None
         return self.parts[1]
 
     @property
@@ -59,11 +97,10 @@ class CloudPath(pathlib.PurePosixPath):
 
     @property
     def node_uuid(self) -> str:
-        node_uuids = self.node_uuids
-
-        if not node_uuids:
+        if len(self.parts) <= 2:
             return None
-        return node_uuids[-1]
+
+        return self.parts[-1]
 
 
 @contextmanager
@@ -84,66 +121,183 @@ def save_as_json(pillar_resource, json_filename):
         json.dump(pillar_resource, outfile, sort_keys=True, cls=pillarsdk.utils.PillarJSONEncoder)
 
 
-def blender_id_profile() -> dict:
+def blender_id_profile() -> 'blender_id.BlenderIdProfile':
     """Returns the Blender ID profile of the currently logged in user."""
 
     # Allow overriding before we import the bpy module.
     if _testing_blender_id_profile is not None:
         return _testing_blender_id_profile
 
-    import bpy
-
-    active_user_id = getattr(bpy.context.window_manager, 'blender_id_active_profile', None)
-    if not active_user_id:
-        return None
-
-    import blender_id.profiles
-    return blender_id.profiles.get_active_profile()
+    import blender_id
+    return blender_id.get_active_profile()
 
 
-def pillar_api(pillar_endpoint: str = None) -> pillarsdk.Api:
+def blender_id_subclient() -> dict:
+    """Returns the subclient dict, containing the 'subclient_user_id' and 'token' keys."""
+
+    profile = blender_id_profile()
+    if not profile:
+        raise UserNotLoggedInError()
+
+    subclient = profile.subclients.get(SUBCLIENT_ID)
+    if not subclient:
+        raise CredentialsNotSyncedError()
+
+    return subclient
+
+
+def pillar_user_uuid() -> str:
+    """Returns the UUID of the Pillar user."""
+
+    import blender_id
+    return blender_id.get_subclient_user_id(SUBCLIENT_ID)
+
+
+def pillar_api(pillar_endpoint: str = None, caching=True) -> pillarsdk.Api:
     """Returns the Pillar SDK API object for the current user.
 
     The user must be logged in.
 
     :param pillar_endpoint: URL of the Pillar server, for testing purposes. If not specified,
         it will use the addon preferences.
+    :param caching: whether to return a caching or non-caching API
     """
 
     global _pillar_api
 
     # Only return the Pillar API object if the user is still logged in.
-    profile = blender_id_profile()
-    if not profile:
-        raise UserNotLoggedInError()
+    subclient = blender_id_subclient()
 
-    if _pillar_api is None:
+    if not _pillar_api:
         # Allow overriding the endpoint before importing Blender-specific stuff.
         if pillar_endpoint is None:
             from . import blender
             pillar_endpoint = blender.preferences().pillar_server
 
-        pillarsdk.Api.requests_session = cache.requests_session()
+        _caching_api = pillarsdk.Api(endpoint=pillar_endpoint,
+                                     username=subclient['subclient_user_id'],
+                                     password=SUBCLIENT_ID,
+                                     token=subclient['token'])
+        _caching_api.requests_session = cache.requests_session()
 
-        _pillar_api = pillarsdk.Api(endpoint=pillar_endpoint,
-                                    username=profile['username'],
-                                    password=None,
-                                    token=profile['token'])
+        _noncaching_api = pillarsdk.Api(endpoint=pillar_endpoint,
+                                        username=subclient['subclient_user_id'],
+                                        password=SUBCLIENT_ID,
+                                        token=subclient['token'])
+        _noncaching_api.requests_session = uncached_session
 
-    return _pillar_api
+        # Send the addon version as HTTP header.
+        from blender_cloud import bl_info
+        addon_version = '.'.join(str(v) for v in bl_info['version'])
+        _caching_api.global_headers['Blender-Cloud-Addon'] = addon_version
+        _noncaching_api.global_headers['Blender-Cloud-Addon'] = addon_version
+
+        _pillar_api = {
+            True: _caching_api,
+            False: _noncaching_api,
+        }
+
+    return _pillar_api[caching]
+
+
+# No more than this many Pillar calls should be made simultaneously
+pillar_semaphore = asyncio.Semaphore(3)
+
+
+async def pillar_call(pillar_func, *args, caching=True, **kwargs):
+    """Calls a Pillar function.
+
+    A semaphore is used to ensure that there won't be too many
+    calls to Pillar simultaneously.
+    """
+
+    partial = functools.partial(pillar_func, *args, api=pillar_api(caching=caching), **kwargs)
+    loop = asyncio.get_event_loop()
+
+    async with pillar_semaphore:
+        return await loop.run_in_executor(None, partial)
+
+
+def sync_call(pillar_func, *args, caching=True, **kwargs):
+    """Synchronous call to Pillar, ensures the correct Api object is used."""
+
+    return pillar_func(*args, api=pillar_api(caching=caching), **kwargs)
+
+
+async def check_pillar_credentials(required_roles: set):
+    """Tries to obtain the user at Pillar using the user's credentials.
+
+    :param required_roles: set of roles to require -- having one of those is enough.
+    :raises UserNotLoggedInError: when the user is not logged in on Blender ID.
+    :raises CredentialsNotSyncedError: when the user is logged in on Blender ID but
+        doesn't have a valid subclient token for Pillar.
+    :returns: the Pillar User ID of the current user.
+    """
+
+    profile = blender_id_profile()
+    if not profile:
+        raise UserNotLoggedInError()
+
+    subclient = profile.subclients.get(SUBCLIENT_ID)
+    if not subclient:
+        raise CredentialsNotSyncedError()
+
+    pillar_user_id = subclient['subclient_user_id']
+    if not pillar_user_id:
+        raise CredentialsNotSyncedError()
+
+    try:
+        db_user = await pillar_call(pillarsdk.User.me)
+    except (pillarsdk.UnauthorizedAccess, pillarsdk.ResourceNotFound, pillarsdk.ForbiddenAccess):
+        raise CredentialsNotSyncedError()
+
+    roles = db_user.roles or set()
+    log.debug('User has roles %r', roles)
+    if required_roles and not required_roles.intersection(set(roles)):
+        # Delete the subclient info. This forces a re-check later, which can
+        # then pick up on the user's new status.
+        del profile.subclients[SUBCLIENT_ID]
+        profile.save_json()
+        raise NotSubscribedToCloudError()
+
+    return db_user
+
+
+async def refresh_pillar_credentials(required_roles: set):
+    """Refreshes the authentication token on Pillar.
+
+    :raises blender_id.BlenderIdCommError: when Blender ID refuses to send a token to Pillar.
+    :raises Exception: when the Pillar credential check fails.
+    """
+
+    global _pillar_api
+
+    import blender_id
+
+    from . import blender
+    pillar_endpoint = blender.preferences().pillar_server
+
+    # Create a subclient token and send it to Pillar.
+    # May raise a blender_id.BlenderIdCommError
+    try:
+        blender_id.create_subclient_token(SUBCLIENT_ID, pillar_endpoint)
+    except blender_id.communication.BlenderIdCommError as ex:
+        log.warning("Unable to create authentication token: %s", ex)
+        raise CredentialsNotSyncedError()
+
+    # Test the new URL
+    _pillar_api = None
+    return await check_pillar_credentials(required_roles)
 
 
 async def get_project_uuid(project_url: str) -> str:
     """Returns the UUID for the project, given its '/p/<project_url>' string."""
 
-    find_one = functools.partial(pillarsdk.Project.find_one, {
-        'where': {'url': project_url},
-        'projection': {'permissions': 1},
-    }, api=pillar_api())
-
-    loop = asyncio.get_event_loop()
     try:
-        project = await loop.run_in_executor(None, find_one)
+        project = await pillar_call(pillarsdk.Project.find_one, {
+            'where': {'url': project_url},
+            'projection': {'permissions': 1},
+        })
     except pillarsdk.exceptions.ResourceNotFound:
         log.error('Project with URL %r does not exist', project_url)
         return None
@@ -153,7 +307,7 @@ async def get_project_uuid(project_url: str) -> str:
 
 
 async def get_nodes(project_uuid: str = None, parent_node_uuid: str = None,
-                    node_type: str = None) -> list:
+                    node_type=None, max_results=None) -> list:
     """Gets nodes for either a project or given a parent node.
 
     @param project_uuid: the UUID of the project, or None if only querying by parent_node_uuid.
@@ -178,19 +332,43 @@ async def get_nodes(project_uuid: str = None, parent_node_uuid: str = None,
         where['project'] = project_uuid
 
     if node_type:
-        where['node_type'] = node_type
+        if isinstance(node_type, str):
+            where['node_type'] = node_type
+        else:
+            # Convert set & tuple to list
+            where['node_type'] = {'$in': list(node_type)}
 
-    node_all = functools.partial(pillarsdk.Node.all, {
-        'projection': {'name': 1, 'parent': 1, 'node_type': 1,
-                       'properties.order': 1, 'properties.status': 1,
-                       'properties.files': 1,
-                       'properties.content_type': 1, 'picture': 1},
-        'where': where,
-        'sort': 'properties.order',
-        'embed': ['parent']}, api=pillar_api())
+    params = {'projection': {'name': 1, 'parent': 1, 'node_type': 1, 'properties.order': 1,
+                             'properties.status': 1, 'properties.files': 1,
+                             'properties.content_type': 1, 'picture': 1},
+              'where': where,
+              'embed': ['parent']}
 
-    loop = asyncio.get_event_loop()
-    children = await loop.run_in_executor(None, node_all)
+    # Pagination
+    if max_results:
+        params['max_results'] = int(max_results)
+
+    children = await pillar_call(pillarsdk.Node.all, params)
+
+    return children['_items']
+
+
+async def get_texture_projects(max_results=None) -> list:
+    """Returns project dicts that contain textures."""
+
+    params = {}
+
+    # Pagination
+    if max_results:
+        params['max_results'] = int(max_results)
+
+    try:
+        children = await pillar_call(pillarsdk.Project.all_from_endpoint,
+                                     '/bcloud/texture-libraries',
+                                     params=params)
+    except pillarsdk.ResourceNotFound as ex:
+        log.warning('Unable to find texture projects: %s', ex)
+        raise PillarError('Unable to find texture projects: %s' % ex)
 
     return children['_items']
 
@@ -306,11 +484,7 @@ async def fetch_thumbnail_info(file: pillarsdk.File, directory: str, desired_siz
         finished.
     """
 
-    api = pillar_api()
-
-    loop = asyncio.get_event_loop()
-    thumb_link = await loop.run_in_executor(None, functools.partial(
-        file.thumbnail_file, desired_size, api=api))
+    thumb_link = await pillar_call(file.thumbnail, desired_size)
 
     if thumb_link is None:
         raise ValueError("File {} has no thumbnail of size {}"
@@ -346,30 +520,21 @@ async def fetch_texture_thumbs(parent_node_uuid: str, desired_size: str,
     # Download all texture nodes in parallel.
     log.debug('Getting child nodes of node %r', parent_node_uuid)
     texture_nodes = await get_nodes(parent_node_uuid=parent_node_uuid,
-                                    node_type='texture')
+                                    node_type=TEXTURE_NODE_TYPES)
 
     if is_cancelled(future):
         log.warning('fetch_texture_thumbs: Texture downloading cancelled')
         return
 
-    # We don't want to gather too much in parallel, as it will make cancelling take more time.
-    # This is caused by HTTP requests going out in parallel, and once the socket is open and
-    # the GET request is sent, we can't cancel until the server starts streaming the response.
-    chunk_size = 2
-    for i in range(0, len(texture_nodes), chunk_size):
-        chunk = texture_nodes[i:i + chunk_size]
+    coros = (download_texture_thumbnail(texture_node, desired_size,
+                                        thumbnail_directory,
+                                        thumbnail_loading=thumbnail_loading,
+                                        thumbnail_loaded=thumbnail_loaded,
+                                        future=future)
+             for texture_node in texture_nodes)
 
-        log.debug('fetch_texture_thumbs: Gathering texture[%i:%i] for parent node %r',
-                  i, i + chunk_size, parent_node_uuid)
-        coros = (download_texture_thumbnail(texture_node, desired_size,
-                                            thumbnail_directory,
-                                            thumbnail_loading=thumbnail_loading,
-                                            thumbnail_loaded=thumbnail_loaded,
-                                            future=future)
-                 for texture_node in chunk)
-
-        # raises any exception from failed handle_texture_node() calls.
-        await asyncio.gather(*coros)
+    # raises any exception from failed handle_texture_node() calls.
+    await asyncio.gather(*coros)
 
     log.info('fetch_texture_thumbs: Done downloading texture thumbnails')
 
@@ -381,7 +546,7 @@ async def download_texture_thumbnail(texture_node, desired_size: str,
                                      thumbnail_loaded: callable,
                                      future: asyncio.Future = None):
     # Skip non-texture nodes, as we can't thumbnail them anyway.
-    if texture_node['node_type'] != 'texture':
+    if texture_node['node_type'] not in TEXTURE_NODE_TYPES:
         return
 
     if is_cancelled(future):
@@ -389,17 +554,29 @@ async def download_texture_thumbnail(texture_node, desired_size: str,
                   texture_node['_id'])
         return
 
-    api = pillar_api()
     loop = asyncio.get_event_loop()
 
-    file_find = functools.partial(pillarsdk.File.find, params={
-        'projection': {'filename': 1, 'variations': 1, 'width': 1, 'height': 1},
-    }, api=api)
+    # Find out which file to use for the thumbnail picture.
+    pic_uuid = texture_node.picture
+    if not pic_uuid:
+        # Fall back to the first texture file, if it exists.
+        log.debug('Node %r does not have a picture, falling back to first file.',
+                  texture_node['_id'])
+        files = texture_node.properties and texture_node.properties.files
+        if not files:
+            log.info('Node %r does not have a picture nor files, skipping.', texture_node['_id'])
+            return
+        pic_uuid = files[0].file
+        if not pic_uuid:
+            log.info('Node %r does not have a picture nor files, skipping.', texture_node['_id'])
+            return
 
-    # Find the File that belongs to this texture node
-    pic_uuid = texture_node['picture']
+    # Load the File that belongs to this texture node's picture.
     loop.call_soon_threadsafe(thumbnail_loading, texture_node, texture_node)
-    file_desc = await loop.run_in_executor(None, file_find, pic_uuid)
+    file_desc = await pillar_call(pillarsdk.File.find, pic_uuid, params={
+        'projection': {'filename': 1, 'variations': 1, 'width': 1, 'height': 1,
+                       'length': 1},
+    })
 
     if file_desc is None:
         log.warning('Unable to find file for texture node %s', pic_uuid)
@@ -427,14 +604,80 @@ async def download_texture_thumbnail(texture_node, desired_size: str,
     loop.call_soon_threadsafe(thumbnail_loaded, texture_node, file_desc, thumb_path)
 
 
+async def fetch_node_files(node: pillarsdk.Node,
+                           *,
+                           file_doc_loading: callable,
+                           file_doc_loaded: callable,
+                           future: asyncio.Future = None):
+    """Fetches all files of a texture/hdri node.
+
+    @param node: Node document to fetch all file docs for.
+    @param file_doc_loading: callback function that takes (file_id, ) parameters,
+        which is called before a file document will be downloaded. This allows you to
+        show a "downloading" indicator.
+    @param file_doc_loaded: callback function that takes (file_id, pillarsdk.File object)
+        parameters, which is called for every thumbnail after it's been downloaded.
+    @param future: Future that's inspected; if it is not None and cancelled, texture downloading
+        is aborted.
+    """
+
+    # Download all thumbnails in parallel.
+    if is_cancelled(future):
+        log.warning('fetch_texture_thumbs: Texture downloading cancelled')
+        return
+
+    coros = (download_file_doc(file_ref.file,
+                               file_doc_loading=file_doc_loading,
+                               file_doc_loaded=file_doc_loaded,
+                               future=future)
+             for file_ref in node.properties.files)
+
+    # raises any exception from failed handle_texture_node() calls.
+    await asyncio.gather(*coros)
+
+    log.info('fetch_node_files: Done downloading %i files', len(node.properties.files))
+
+
+async def download_file_doc(file_id,
+                            *,
+                            file_doc_loading: callable,
+                            file_doc_loaded: callable,
+                            future: asyncio.Future = None):
+
+    if is_cancelled(future):
+        log.debug('fetch_texture_thumbs cancelled before finding File for file_id %s', file_id)
+        return
+
+    loop = asyncio.get_event_loop()
+
+    # Load the File that belongs to this texture node's picture.
+    loop.call_soon_threadsafe(file_doc_loading, file_id)
+    file_desc = await pillar_call(pillarsdk.File.find, file_id, params={
+        'projection': {'filename': 1, 'variations': 1, 'width': 1, 'height': 1,
+                       'length': 1},
+    })
+
+    if file_desc is None:
+        log.warning('Unable to find File for file_id %s', file_id)
+
+    loop.call_soon_threadsafe(file_doc_loaded, file_id, file_desc)
+
+
 async def download_file_by_uuid(file_uuid,
                                 target_directory: str,
                                 metadata_directory: str,
                                 *,
-                                map_type: str=None,
-                                file_loading: callable,
-                                file_loaded: callable,
+                                filename: str = None,
+                                map_type: str = None,
+                                file_loading: callable = None,
+                                file_loaded: callable = None,
+                                file_loaded_sync: callable = None,
                                 future: asyncio.Future):
+    """Downloads a file from Pillar by its UUID.
+
+    :param filename: overrules the filename in file_doc['filename'] if given.
+        The extension from file_doc['filename'] is still used, though.
+    """
     if is_cancelled(future):
         log.debug('download_file_by_uuid(%r) cancelled.', file_uuid)
         return
@@ -442,21 +685,28 @@ async def download_file_by_uuid(file_uuid,
     loop = asyncio.get_event_loop()
 
     # Find the File document.
-    api = pillar_api()
-    file_find = functools.partial(pillarsdk.File.find, params={
-        'projection': {'link': 1, 'filename': 1},
-    }, api=api)
-    file_desc = await loop.run_in_executor(None, file_find, file_uuid)
+    file_desc = await pillar_call(pillarsdk.File.find, file_uuid, params={
+        'projection': {'link': 1, 'filename': 1, 'length': 1},
+    })
 
     # Save the file document to disk
     metadata_file = os.path.join(metadata_directory, 'files', '%s.json' % file_uuid)
     save_as_json(file_desc, metadata_file)
 
-    file_path = os.path.join(target_directory,
-                             sanitize_filename('%s-%s' % (map_type, file_desc['filename'])))
+    # Let the caller override the filename root.
+    root, ext = os.path.splitext(file_desc['filename'])
+    if filename:
+        root, _ = os.path.splitext(filename)
+    if not map_type or root.endswith(map_type):
+        target_filename = '%s%s' % (root, ext)
+    else:
+        target_filename = '%s-%s%s' % (root, map_type, ext)
+
+    file_path = os.path.join(target_directory, sanitize_filename(target_filename))
     file_url = file_desc['link']
     # log.debug('Texture %r:\n%s', file_uuid, pprint.pformat(file_desc.to_dict()))
-    loop.call_soon_threadsafe(file_loading, file_path, file_desc)
+    if file_loading is not None:
+        loop.call_soon_threadsafe(file_loading, file_path, file_desc, map_type)
 
     # Cached headers are stored in the project space
     header_store = os.path.join(metadata_directory, 'files',
@@ -464,7 +714,10 @@ async def download_file_by_uuid(file_uuid,
 
     await download_to_file(file_url, file_path, header_store=header_store, future=future)
 
-    loop.call_soon_threadsafe(file_loaded, file_path, file_desc)
+    if file_loaded is not None:
+        loop.call_soon_threadsafe(file_loaded, file_path, file_desc, map_type)
+    if file_loaded_sync is not None:
+        await file_loaded_sync(file_path, file_desc, map_type)
 
 
 async def download_texture(texture_node,
@@ -474,23 +727,214 @@ async def download_texture(texture_node,
                            texture_loading: callable,
                            texture_loaded: callable,
                            future: asyncio.Future):
-    if texture_node['node_type'] != 'texture':
-        raise TypeError("Node type should be 'texture', not %r" % texture_node['node_type'])
+    node_type_name = texture_node['node_type']
+    if node_type_name not in TEXTURE_NODE_TYPES:
+        raise TypeError("Node type should be in %r, not %r" %
+                        (TEXTURE_NODE_TYPES, node_type_name))
+
+    filename = '%s.taken_from_file' % sanitize_filename(texture_node['name'])
 
     # Download every file. Eve doesn't support embedding from a list-of-dicts.
-    downloaders = (download_file_by_uuid(file_info['file'],
-                                         target_directory,
-                                         metadata_directory,
-                                         map_type=file_info['map_type'],
-                                         file_loading=texture_loading,
-                                         file_loaded=texture_loaded,
-                                         future=future)
-                   for file_info in texture_node['properties']['files'])
+    downloaders = []
+    for file_info in texture_node['properties']['files']:
+        dlr = download_file_by_uuid(file_info['file'],
+                                    target_directory,
+                                    metadata_directory,
+                                    filename=filename,
+                                    map_type=file_info.map_type or file_info.resolution,
+                                    file_loading=texture_loading,
+                                    file_loaded=texture_loaded,
+                                    future=future)
+        downloaders.append(dlr)
 
-    return await asyncio.gather(*downloaders)
+    return await asyncio.gather(*downloaders, return_exceptions=True)
+
+
+async def upload_file(project_id: str, file_path: pathlib.Path, *,
+                      future: asyncio.Future) -> str:
+    """Uploads a file to the Blender Cloud, returning a file document ID."""
+
+    from .blender import PILLAR_SERVER_URL
+
+    loop = asyncio.get_event_loop()
+    url = urllib.parse.urljoin(PILLAR_SERVER_URL, '/storage/stream/%s' % project_id)
+
+    # Upload the file in a different thread.
+    def upload():
+        auth_token = blender_id_subclient()['token']
+
+        with file_path.open(mode='rb') as infile:
+            return uncached_session.post(url,
+                                         files={'file': infile},
+                                         auth=(auth_token, SUBCLIENT_ID))
+
+    # Check for cancellation even before we start our POST request
+    if is_cancelled(future):
+        log.debug('Uploading was cancelled before doing the POST')
+        raise asyncio.CancelledError('Uploading was cancelled')
+
+    log.debug('Performing POST %s', url)
+    response = await loop.run_in_executor(None, upload)
+    log.debug('Status %i from POST %s', response.status_code, url)
+    response.raise_for_status()
+
+    resp = response.json()
+    log.debug('Upload response: %s', resp)
+
+    try:
+        file_id = resp['file_id']
+    except KeyError:
+        log.error('No file ID in upload response: %s', resp)
+        raise PillarError('No file ID in upload response: %s' % resp)
+
+    log.info('Uploaded %s to file ID %s', file_path, file_id)
+    return file_id
 
 
 def is_cancelled(future: asyncio.Future) -> bool:
     # assert future is not None  # for debugging purposes.
     cancelled = future is not None and future.cancelled()
     return cancelled
+
+
+class PillarOperatorMixin:
+    async def check_credentials(self, context, required_roles) -> bool:
+        """Checks credentials with Pillar, and if ok returns the user document from Pillar/MongoDB.
+
+        :raises UserNotLoggedInError: if the user is not logged in
+        :raises NotSubscribedToCloudError: if the user does not have any of the required roles
+        """
+
+        # self.report({'INFO'}, 'Checking Blender Cloud credentials')
+
+        try:
+            db_user = await check_pillar_credentials(required_roles)
+        except NotSubscribedToCloudError:
+            self._log_subscription_needed()
+            raise
+        except CredentialsNotSyncedError:
+            self.log.info('Credentials not synced, re-syncing automatically.')
+        else:
+            self.log.info('Credentials okay.')
+            return db_user
+
+        try:
+            db_user = await refresh_pillar_credentials(required_roles)
+        except NotSubscribedToCloudError:
+            self._log_subscription_needed()
+            raise
+        except CredentialsNotSyncedError:
+            self.log.info('Credentials not synced after refreshing, handling as not logged in.')
+            raise UserNotLoggedInError('Not logged in.')
+        except UserNotLoggedInError:
+            self.log.error('User not logged in on Blender ID.')
+            raise
+        else:
+            self.log.info('Credentials refreshed and ok.')
+            return db_user
+
+    def _log_subscription_needed(self):
+        self.log.warning(
+            'Please subscribe to the blender cloud at https://cloud.blender.org/join')
+        self.report({'INFO'},
+                    'Please subscribe to the blender cloud at https://cloud.blender.org/join')
+
+
+class AuthenticatedPillarOperatorMixin(PillarOperatorMixin):
+    """Checks credentials, to be used at the start of async_execute().
+
+    Sets self.user_id to the current user's ID, and self.db_user to the user info dict,
+    if authentication was succesful; sets both to None if not.
+    """
+
+    async def authenticate(self, context) -> bool:
+        from . import pillar
+
+        self.log.info('Checking credentials')
+        self.user_id = None
+        self.db_user = None
+        try:
+            self.db_user = await self.check_credentials(context, ())
+        except pillar.UserNotLoggedInError as ex:
+            self.log.info('Not logged in error raised: %s', ex)
+            self.report({'ERROR'}, 'Please log in on Blender ID first.')
+            self.quit()
+            return False
+
+        self.user_id = self.db_user['_id']
+        return True
+
+
+
+async def find_or_create_node(where: dict,
+                              additional_create_props: dict = None,
+                              projection: dict = None,
+                              may_create: bool = True) -> (pillarsdk.Node, bool):
+    """Finds a node by the `filter_props`, creates it using the additional props.
+
+    :returns: tuple (node, created), where 'created' is a bool indicating whether
+              a new node was created, or an exising one is returned.
+    """
+
+    params = {
+        'where': where,
+    }
+    if projection:
+        params['projection'] = projection
+
+    found_node = await pillar_call(pillarsdk.Node.find_first, params, caching=False)
+
+    if found_node is not None:
+        return found_node, False
+
+    if not may_create:
+        return None, False
+
+    # Augment the node properties to form a complete node.
+    node_props = where.copy()
+    if additional_create_props:
+        node_props.update(additional_create_props)
+
+    log.debug('Creating new node %s', node_props)
+    created_node = pillarsdk.Node.new(node_props)
+    created_ok = await pillar_call(created_node.create)
+    if not created_ok:
+        log.error('Blender Cloud addon: unable to create node on the Cloud.')
+        raise PillarError('Unable to create node on the Cloud')
+
+    return created_node, True
+
+
+async def attach_file_to_group(file_path: pathlib.Path,
+                               home_project_id: str,
+                               group_node_id: str,
+                               user_id: str = None) -> pillarsdk.Node:
+    """Creates an Asset node and attaches a file document to it."""
+
+    node = await pillar_call(pillarsdk.Node.create_asset_from_file,
+                             home_project_id,
+                             group_node_id,
+                             'file',
+                             str(file_path),
+                             extra_where=user_id and {'user': user_id},
+                             caching=False)
+
+    return node
+
+
+def node_to_id(node: pillarsdk.Node) -> dict:
+    """Converts a Node to a dict we can store in an ID property.
+
+    ID properties only support a handful of Python classes, so we have
+    to convert datetime.datetime to a string and remove None values.
+    """
+
+    def to_rna(value):
+        if isinstance(value, dict):
+            return {k: to_rna(v) for k, v in value.items()}
+        if isinstance(value, datetime.datetime):
+            return value.strftime(RFC1123_DATE_FORMAT)
+        return value
+
+    as_dict = to_rna(node.to_dict())
+    return pillarsdk.utils.remove_none_attributes(as_dict)

blender_cloud/settings_sync.py

@@ -0,0 +1,523 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
+"""Synchronises settings & startup file with the Cloud.
+Caching is disabled on many PillarSDK calls, as synchronisation can happen
+rapidly between multiple machines. This means that information can be outdated
+in seconds, rather than the minutes the cache system assumes.
+"""
+import functools
+import logging
+import pathlib
+import tempfile
+import shutil
+
+import bpy
+
+import asyncio
+
+import pillarsdk
+from pillarsdk import exceptions as sdk_exceptions
+from .pillar import pillar_call
+from . import async_loop, pillar, cache, blendfile, home_project
+
+SETTINGS_FILES_TO_UPLOAD = ['userpref.blend', 'startup.blend']
+
+# These are RNA keys inside the userpref.blend file, and their
+# Python properties names. These settings will not be synced.
+LOCAL_SETTINGS_RNA = [
+    (b'dpi', 'system.dpi'),
+    (b'virtual_pixel', 'system.virtual_pixel_mode'),
+    (b'compute_device_id', 'system.compute_device'),
+    (b'compute_device_type', 'system.compute_device_type'),
+    (b'fontdir', 'filepaths.font_directory'),
+    (b'textudir', 'filepaths.texture_directory'),
+    (b'renderdir', 'filepaths.render_output_directory'),
+    (b'pythondir', 'filepaths.script_directory'),
+    (b'sounddir', 'filepaths.sound_directory'),
+    (b'tempdir', 'filepaths.temporary_directory'),
+    (b'render_cachedir', 'filepaths.render_cache_directory'),
+    (b'i18ndir', 'filepaths.i18n_branches_directory'),
+    (b'image_editor', 'filepaths.image_editor'),
+    (b'anim_player', 'filepaths.animation_player'),
+]
+
+REQUIRES_ROLES_FOR_SYNC = set()  # no roles needed.
+SYNC_GROUP_NODE_NAME = 'Blender Sync'
+SYNC_GROUP_NODE_DESC = 'The [Blender Cloud Addon](https://cloud.blender.org/services' \
+                       '#blender-addon) will synchronize your Blender settings here.'
+log = logging.getLogger(__name__)
+
+
+def set_blender_sync_status(set_status: str):
+    def decorator(func):
+        @functools.wraps(func)
+        def wrapper(*args, **kwargs):
+            bss = bpy.context.window_manager.blender_sync_status
+            bss.status = set_status
+            try:
+                return func(*args, **kwargs)
+            finally:
+                bss.status = 'IDLE'
+
+        return wrapper
+
+    return decorator
+
+
+def async_set_blender_sync_status(set_status: str):
+    def decorator(func):
+        @functools.wraps(func)
+        async def wrapper(*args, **kwargs):
+            bss = bpy.context.window_manager.blender_sync_status
+            bss.status = set_status
+            try:
+                return await func(*args, **kwargs)
+            finally:
+                bss.status = 'IDLE'
+
+        return wrapper
+
+    return decorator
+
+
+async def find_sync_group_id(home_project_id: str,
+                             user_id: str,
+                             blender_version: str,
+                             *,
+                             may_create=True) -> str:
+    """Finds the group node in which to store sync assets.
+
+    If the group node doesn't exist and may_create=True, it creates it.
+    """
+
+    # Find the top-level sync group node. This should have been
+    # created by Pillar while creating the home project.
+    try:
+        sync_group, created = await pillar.find_or_create_node(
+            where={'project': home_project_id,
+                   'node_type': 'group',
+                   'parent': None,
+                   'name': SYNC_GROUP_NODE_NAME,
+                   'user': user_id},
+            projection={'_id': 1},
+            may_create=False)
+    except pillar.PillarError:
+        raise pillar.PillarError('Unable to find sync folder on the Cloud')
+
+    if not may_create and sync_group is None:
+        log.info("Sync folder doesn't exist, and not creating it either.")
+        return None, None
+
+    # Find/create the sub-group for the requested Blender version
+    try:
+        sub_sync_group, created = await pillar.find_or_create_node(
+            where={'project': home_project_id,
+                   'node_type': 'group',
+                   'parent': sync_group['_id'],
+                   'name': blender_version,
+                   'user': user_id},
+            additional_create_props={
+                'description': 'Sync folder for Blender %s' % blender_version,
+                'properties': {'status': 'published'},
+            },
+            projection={'_id': 1},
+            may_create=may_create)
+    except pillar.PillarError:
+        raise pillar.PillarError('Unable to create sync folder on the Cloud')
+
+    if not may_create and sub_sync_group is None:
+        log.info("Sync folder for Blender version %s doesn't exist, "
+                 "and not creating it either.", blender_version)
+        return sync_group['_id'], None
+
+    return sync_group['_id'], sub_sync_group['_id']
+
+
+@functools.lru_cache()
+async def available_blender_versions(home_project_id: str, user_id: str) -> list:
+    bss = bpy.context.window_manager.blender_sync_status
+
+    # Get the available Blender versions.
+    sync_group = await pillar_call(
+        pillarsdk.Node.find_first,
+        params={
+            'where': {'project': home_project_id,
+                      'node_type': 'group',
+                      'parent': None,
+                      'name': SYNC_GROUP_NODE_NAME,
+                      'user': user_id},
+            'projection': {'_id': 1},
+        },
+        caching=False)
+
+    if sync_group is None:
+        bss.report({'ERROR'}, 'No synced Blender settings in your Blender Cloud')
+        log.debug('-- unable to find sync group for home_project_id=%r and user_id=%r',
+                  home_project_id, user_id)
+        return []
+
+    sync_nodes = await pillar_call(
+        pillarsdk.Node.all,
+        params={
+            'where': {'project': home_project_id,
+                      'node_type': 'group',
+                      'parent': sync_group['_id'],
+                      'user': user_id},
+            'projection': {'_id': 1, 'name': 1},
+            'sort': '-name',
+        },
+        caching=False)
+
+    if not sync_nodes or not sync_nodes._items:
+        bss.report({'ERROR'}, 'No synced Blender settings in your Blender Cloud.')
+        return []
+
+    versions = [node.name for node in sync_nodes._items]
+    log.debug('Versions: %s', versions)
+
+    return versions
+
+
+# noinspection PyAttributeOutsideInit
+class PILLAR_OT_sync(pillar.PillarOperatorMixin,
+                     async_loop.AsyncModalOperatorMixin,
+                     bpy.types.Operator):
+    bl_idname = 'pillar.sync'
+    bl_label = 'Synchronise with Blender Cloud'
+    bl_description = 'Synchronises Blender settings with Blender Cloud'
+
+    log = logging.getLogger('bpy.ops.%s' % bl_idname)
+    home_project_id = None
+    sync_group_id = None  # top-level sync group node ID
+    sync_group_versioned_id = None  # sync group node ID for the given Blender version.
+
+    action = bpy.props.EnumProperty(
+        items=[
+            ('PUSH', 'Push', 'Push settings to the Blender Cloud'),
+            ('PULL', 'Pull', 'Pull settings from the Blender Cloud'),
+            ('REFRESH', 'Refresh', 'Refresh available versions'),
+            ('SELECT', 'Select', 'Select version to sync'),
+        ],
+        name='action')
+
+    CURRENT_BLENDER_VERSION = '%i.%i' % bpy.app.version[:2]
+    blender_version = bpy.props.StringProperty(name='blender_version',
+                                               description='Blender version to sync for',
+                                               default=CURRENT_BLENDER_VERSION)
+
+    def bss_report(self, level, message):
+        bss = bpy.context.window_manager.blender_sync_status
+        bss.report(level, message)
+
+    def invoke(self, context, event):
+        if self.action == 'SELECT':
+            # Synchronous action
+            return self.action_select(context)
+
+        if self.action in {'PUSH', 'PULL'} and not self.blender_version:
+            self.bss_report({'ERROR'}, 'No Blender version to sync for was given.')
+            return {'CANCELLED'}
+
+        return async_loop.AsyncModalOperatorMixin.invoke(self, context, event)
+
+    def action_select(self, context):
+        """Allows selection of the Blender version to use.
+
+        This is a synchronous action, as it requires a dialog box.
+        """
+
+        self.log.info('Performing action SELECT')
+
+        # Do a refresh before we can show the dropdown.
+        fut = asyncio.ensure_future(self.async_execute(context, action_override='REFRESH'))
+        loop = asyncio.get_event_loop()
+        loop.run_until_complete(fut)
+
+        self._state = 'SELECTING'
+        return context.window_manager.invoke_props_dialog(self)
+
+    def draw(self, context):
+        bss = bpy.context.window_manager.blender_sync_status
+        self.layout.prop(bss, 'version', text='Blender version')
+
+    def execute(self, context):
+        if self.action != 'SELECT':
+            log.debug('Ignoring execute() for action %r', self.action)
+            return {'FINISHED'}
+
+        log.debug('Performing execute() for action %r', self.action)
+        # Perform the sync when the user closes the dialog box.
+        bss = bpy.context.window_manager.blender_sync_status
+        bpy.ops.pillar.sync('INVOKE_DEFAULT',
+                            action='PULL',
+                            blender_version=bss.version)
+
+        return {'FINISHED'}
+
+    @async_set_blender_sync_status('SYNCING')
+    async def async_execute(self, context, *, action_override=None):
+        """Entry point of the asynchronous operator."""
+
+        action = action_override or self.action
+        self.bss_report({'INFO'}, 'Communicating with Blender Cloud')
+        self.log.info('Performing action %s', action)
+
+        try:
+            # Refresh credentials
+            try:
+                db_user = await self.check_credentials(context, REQUIRES_ROLES_FOR_SYNC)
+                self.user_id = db_user['_id']
+                log.debug('Found user ID: %s', self.user_id)
+            except pillar.NotSubscribedToCloudError:
+                self.log.exception('User not subscribed to cloud.')
+                self.bss_report({'SUBSCRIBE'}, 'Please subscribe to the Blender Cloud.')
+                self._state = 'QUIT'
+                return
+            except pillar.UserNotLoggedInError:
+                self.log.exception('Error checking/refreshing credentials.')
+                self.bss_report({'ERROR'}, 'Please log in on Blender ID first.')
+                self._state = 'QUIT'
+                return
+
+            # Find the home project.
+            try:
+                self.home_project_id = await home_project.get_home_project_id()
+            except sdk_exceptions.ForbiddenAccess:
+                self.log.exception('Forbidden access to home project.')
+                self.bss_report({'ERROR'}, 'Did not get access to home project.')
+                self._state = 'QUIT'
+                return
+            except sdk_exceptions.ResourceNotFound:
+                self.bss_report({'ERROR'}, 'Home project not found.')
+                self._state = 'QUIT'
+                return
+
+            # Only create the folder structure if we're pushing.
+            may_create = self.action == 'PUSH'
+            try:
+                gid, subgid = await find_sync_group_id(self.home_project_id,
+                                                       self.user_id,
+                                                       self.blender_version,
+                                                       may_create=may_create)
+                self.sync_group_id = gid
+                self.sync_group_versioned_id = subgid
+                self.log.debug('Found top-level group node ID: %s', self.sync_group_id)
+                self.log.debug('Found group node ID for %s: %s',
+                               self.blender_version, self.sync_group_versioned_id)
+            except sdk_exceptions.ForbiddenAccess:
+                self.log.exception('Unable to find Group ID')
+                self.bss_report({'ERROR'}, 'Unable to find sync folder.')
+                self._state = 'QUIT'
+                return
+
+            # Perform the requested action.
+            action_method = {
+                'PUSH': self.action_push,
+                'PULL': self.action_pull,
+                'REFRESH': self.action_refresh,
+            }[action]
+            await action_method(context)
+        except Exception as ex:
+            self.log.exception('Unexpected exception caught.')
+            self.bss_report({'ERROR'}, 'Unexpected error: %s' % ex)
+
+        self._state = 'QUIT'
+
+    async def action_push(self, context):
+        """Sends files to the Pillar server."""
+
+        self.log.info('Saved user preferences to disk before pushing to cloud.')
+        bpy.ops.wm.save_userpref()
+
+        config_dir = pathlib.Path(bpy.utils.user_resource('CONFIG'))
+
+        for fname in SETTINGS_FILES_TO_UPLOAD:
+            path = config_dir / fname
+            if not path.exists():
+                self.log.debug('Skipping non-existing %s', path)
+                continue
+
+            if self.signalling_future.cancelled():
+                self.bss_report({'WARNING'}, 'Upload aborted.')
+                return
+
+            self.bss_report({'INFO'}, 'Uploading %s' % fname)
+            try:
+                await pillar.attach_file_to_group(path,
+                                                  self.home_project_id,
+                                                  self.sync_group_versioned_id,
+                                                  self.user_id)
+            except sdk_exceptions.RequestEntityTooLarge as ex:
+                self.log.error('File too big to upload: %s' % ex)
+                self.log.error('To upload larger files, please subscribe to Blender Cloud.')
+                self.bss_report({'SUBSCRIBE'}, 'File %s too big to upload. '
+                                               'Subscribe for unlimited space.' % fname)
+                self._state = 'QUIT'
+                return
+
+        await self.action_refresh(context)
+
+        # After pushing, change the 'pull' version to the current version of Blender.
+        # Or to the latest version, if by some mistake somewhere the current push
+        # isn't available after all.
+        bss = bpy.context.window_manager.blender_sync_status
+        if self.CURRENT_BLENDER_VERSION in bss.available_blender_versions:
+            bss.version = self.CURRENT_BLENDER_VERSION
+        else:
+            bss.version = max(bss.available_blender_versions)
+
+        self.bss_report({'INFO'}, 'Settings pushed to Blender Cloud.')
+
+    async def action_pull(self, context):
+        """Loads files from the Pillar server."""
+
+        # If the sync group node doesn't exist, offer a list of groups that do.
+        if self.sync_group_id is None:
+            self.bss_report({'ERROR'},
+                            'There are no synced Blender settings in your Blender Cloud.')
+            return
+
+        if self.sync_group_versioned_id is None:
+            self.bss_report({'ERROR'}, 'Therre are no synced Blender settings for version %s' %
+                            self.blender_version)
+            return
+
+        self.bss_report({'INFO'}, 'Pulling settings from Blender Cloud')
+        with tempfile.TemporaryDirectory(prefix='bcloud-sync') as tempdir:
+            for fname in SETTINGS_FILES_TO_UPLOAD:
+                await self.download_settings_file(fname, tempdir)
+
+        self.bss_report({'WARNING'}, 'Settings pulled from Cloud, restart Blender to load them.')
+
+    async def action_refresh(self, context):
+        self.bss_report({'INFO'}, 'Refreshing available Blender versions.')
+
+        # Clear the LRU cache of available_blender_versions so that we can
+        # obtain new versions (if someone synced from somewhere else, for example)
+        available_blender_versions.cache_clear()
+
+        versions = await available_blender_versions(self.home_project_id, self.user_id)
+        bss = bpy.context.window_manager.blender_sync_status
+        bss.available_blender_versions = versions
+
+        if versions:
+            # There are versions to sync, so we can remove the status message.
+            # However, if there aren't any, the status message shows why, and
+            # shouldn't be erased.
+            self.bss_report({'INFO'}, '')
+
+    async def download_settings_file(self, fname: str, temp_dir: str):
+        config_dir = pathlib.Path(bpy.utils.user_resource('CONFIG'))
+        meta_path = cache.cache_directory('home-project', 'blender-sync')
+
+        self.bss_report({'INFO'}, 'Downloading %s from Cloud' % fname)
+
+        # Get the asset node
+        node_props = {'project': self.home_project_id,
+                      'node_type': 'asset',
+                      'parent': self.sync_group_versioned_id,
+                      'name': fname}
+        node = await pillar_call(pillarsdk.Node.find_first, {
+            'where': node_props,
+            'projection': {'_id': 1, 'properties.file': 1}
+        }, caching=False)
+        if node is None:
+            self.bss_report({'INFO'}, 'Unable to find %s on Blender Cloud' % fname)
+            self.log.info('Unable to find node on Blender Cloud for %s', fname)
+            return
+
+        async def file_downloaded(file_path: str, file_desc: pillarsdk.File, map_type: str):
+            # Allow the caller to adjust the file before we move it into place.
+
+            if fname.lower() == 'userpref.blend':
+                await self.update_userpref_blend(file_path)
+
+            # Move the file next to the final location; as it may be on a
+            # different filesystem than the temporary directory, this can
+            # fail, and we don't want to destroy the existing file.
+            local_temp = config_dir / (fname + '~')
+            local_final = config_dir / fname
+
+            # Make a backup copy of the file as it was before pulling.
+            if local_final.exists():
+                local_bak = config_dir / (fname + '-pre-bcloud-pull')
+                self.move_file(local_final, local_bak)
+
+            self.move_file(file_path, local_temp)
+            self.move_file(local_temp, local_final)
+
+        file_id = node.properties.file
+        await pillar.download_file_by_uuid(file_id,
+                                           temp_dir,
+                                           str(meta_path),
+                                           file_loaded_sync=file_downloaded,
+                                           future=self.signalling_future)
+
+    def move_file(self, src, dst):
+        self.log.info('Moving %s to %s', src, dst)
+        shutil.move(str(src), str(dst))
+
+    async def update_userpref_blend(self, file_path: str):
+        self.log.info('Overriding machine-local settings in %s', file_path)
+
+        # Remember some settings that should not be overwritten from the Cloud.
+        up = bpy.context.user_preferences
+        remembered = {}
+        for rna_key, python_key in LOCAL_SETTINGS_RNA:
+            assert '.' in python_key, 'Sorry, this code assumes there is a dot in the Python key'
+
+            try:
+                value = up.path_resolve(python_key)
+            except ValueError:
+                # Setting doesn't exist. This can happen, for example Cycles
+                # settings on a build that doesn't have Cycles enabled.
+                continue
+
+            # Map enums from strings (in Python) to ints (in DNA).
+            dot_index = python_key.rindex('.')
+            parent_key, prop_key = python_key[:dot_index], python_key[dot_index + 1:]
+            parent = up.path_resolve(parent_key)
+            prop = parent.bl_rna.properties[prop_key]
+            if prop.type == 'ENUM':
+                log.debug('Rewriting %s from %r to %r',
+                          python_key, value, prop.enum_items[value].value)
+                value = prop.enum_items[value].value
+            else:
+                log.debug('Keeping value of %s: %r', python_key, value)
+
+            remembered[rna_key] = value
+        log.debug('Overriding values: %s', remembered)
+
+        # Rewrite the userprefs.blend file to override the options.
+        with blendfile.open_blend(file_path, 'rb+') as blend:
+            prefs = next(block for block in blend.blocks
+                         if block.code == b'USER')
+
+            for key, value in remembered.items():
+                self.log.debug('prefs[%r] = %r' % (key, prefs[key]))
+                self.log.debug('  -> setting prefs[%r] = %r' % (key, value))
+                prefs[key] = value
+
+
+def register():
+    bpy.utils.register_class(PILLAR_OT_sync)
+
+
+def unregister():
+    bpy.utils.unregister_class(PILLAR_OT_sync)

blender_cloud/utils.py

@@ -0,0 +1,102 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
+import pathlib
+
+
+def sizeof_fmt(num: int, suffix='B') -> str:
+    """Returns a human-readable size.
+
+    Source: http://stackoverflow.com/a/1094933/875379
+    """
+
+    for unit in ['', 'Ki', 'Mi', 'Gi', 'Ti', 'Pi', 'Ei', 'Zi']:
+        if abs(num) < 1024:
+            return '%.1f %s%s' % (num, unit, suffix)
+        num /= 1024
+
+    return '%.1f Yi%s' % (num, suffix)
+
+
+def find_in_path(path: pathlib.Path, filename: str) -> pathlib.Path:
+    """Performs a breadth-first search for the filename.
+
+    Returns the path that contains the file, or None if not found.
+    """
+
+    import collections
+
+    # Be lenient on our input type.
+    if isinstance(path, str):
+        path = pathlib.Path(path)
+
+    if not path.exists():
+        return None
+    assert path.is_dir()
+
+    to_visit = collections.deque([path])
+    while to_visit:
+        this_path = to_visit.popleft()
+
+        for subpath in this_path.iterdir():
+            if subpath.is_dir():
+                to_visit.append(subpath)
+                continue
+
+            if subpath.name == filename:
+                return subpath
+
+    return None
+
+
+def pyside_cache(propname):
+    """Decorator, stores the result of the decorated callable in Python-managed memory.
+
+    This is to work around the warning at
+    https://www.blender.org/api/blender_python_api_master/bpy.props.html#bpy.props.EnumProperty
+    """
+
+    if callable(propname):
+        raise TypeError('Usage: pyside_cache("property_name")')
+
+    def decorator(wrapped):
+        """Stores the result of the callable in Python-managed memory.
+
+        This is to work around the warning at
+        https://www.blender.org/api/blender_python_api_master/bpy.props.html#bpy.props.EnumProperty
+        """
+
+        import functools
+
+        @functools.wraps(wrapped)
+        # We can't use (*args, **kwargs), because EnumProperty explicitly checks
+        # for the number of fixed positional arguments.
+        def wrapper(self, context):
+            result = None
+            try:
+                result = wrapped(self, context)
+                return result
+            finally:
+                rna_type, rna_info = getattr(self.bl_rna, propname)
+                rna_info['_cached_result'] = result
+        return wrapper
+    return decorator
+
+
+def redraw(self, context):
+    context.area.tag_redraw()

blender_cloud/wheels/__init__.py

@@ -1,3 +1,21 @@
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
 """External dependencies loader."""
 
 import glob
@@ -18,8 +36,9 @@ def load_wheel(module_name, fname_prefix):
 
     try:
         module = __import__(module_name)
-    except ImportError:
-        pass
+    except ImportError as ex:
+        log.debug('Unable to import %s directly, will try wheel: %s',
+                  module_name, ex)
     else:
         log.debug('Was able to load %s from %s, no need to load wheel %s',
                   module_name, module.__file__, fname_prefix)
@@ -30,7 +49,9 @@ def load_wheel(module_name, fname_prefix):
     if not wheels:
         raise RuntimeError('Unable to find wheel at %r' % path_pattern)
 
-    sys.path.append(wheels[0])
+    # If there are multiple wheels that match, load the latest one.
+    wheels.sort()
+    sys.path.append(wheels[-1])
     module = __import__(module_name)
     log.debug('Loaded %s from %s', module_name, module.__file__)
 
@@ -38,4 +59,4 @@ def load_wheel(module_name, fname_prefix):
 def load_wheels():
     load_wheel('lockfile', 'lockfile')
     load_wheel('cachecontrol', 'CacheControl')
-    load_wheel('pillarsdk', 'pillar_sdk')
+    load_wheel('pillarsdk', 'pillarsdk')

clear_wheels.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+
+git clean -n -d -X blender_cloud/wheels/
+
+echo "Press [ENTER] to actually delete those files."
+read dummy
+
+git clean -f -d -X blender_cloud/wheels/

requirements-dev.txt

@@ -0,0 +1,8 @@
+-r requirements.txt
+
+# Primary requirements
+pytest==3.0.3
+
+# Secondary requirements
+py==1.4.31
+

requirements.txt

@@ -1,2 +1,15 @@
-CacheControl==0.11.6
+# Primary requirements:
+-e git+https://github.com/sybrenstuvel/cachecontrol.git@sybren-filecache-delete-crash-fix#egg=CacheControl
 lockfile==0.12.2
+pillarsdk==1.6.1
+wheel==0.29.0
+
+# Secondary requirements:
+cffi==1.6.0
+cryptography==1.3.1
+idna==2.1
+pyasn1==0.1.9
+pycparser==2.14
+pyOpenSSL==16.0.0
+requests==2.10.0
+six==1.10.0

setup.py

@@ -1,10 +1,30 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
+# ##### BEGIN GPL LICENSE BLOCK #####
+#
+#  This program is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU General Public License
+#  as published by the Free Software Foundation; either version 2
+#  of the License, or (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software Foundation,
+#  Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# ##### END GPL LICENSE BLOCK #####
+
+import glob
+import os
 import sys
 import shutil
 import subprocess
 import re
 import pathlib
-from glob import glob
+import zipfile
 
 from distutils import log
 from distutils.core import Command
@@ -14,8 +34,18 @@ from distutils.command.install_egg_info import install_egg_info
 from setuptools import setup, find_packages
 
 requirement_re = re.compile('[><=]+')
+sys.dont_write_bytecode = True
 
 
+def set_default_path(var, default):
+    """convert CLI-arguments (string) to Paths"""
+
+    if var is None:
+        return default
+    return pathlib.Path(var)
+
+
+# noinspection PyAttributeOutsideInit
 class BuildWheels(Command):
     """Builds or downloads the dependencies as wheel files."""
 
@@ -23,31 +53,22 @@ class BuildWheels(Command):
     user_options = [
         ('wheels-path=', None, "wheel file installation path"),
         ('deps-path=', None, "path in which dependencies are built"),
-        ('pillar-sdk-path=', None, "subdir of deps-path containing the Pillar Python SDK"),
         ('cachecontrol-path=', None, "subdir of deps-path containing CacheControl"),
     ]
 
     def initialize_options(self):
         self.wheels_path = None  # path that will contain the installed wheels.
         self.deps_path = None  # path in which dependencies are built.
-        self.pillar_sdk_path = None  # subdir of deps_path containing the Pillar Python SDK
         self.cachecontrol_path = None  # subdir of deps_path containing CacheControl
 
     def finalize_options(self):
         self.my_path = pathlib.Path(__file__).resolve().parent
         package_path = self.my_path / self.distribution.get_name()
 
-        def set_default(var, default):
-            if var is None:
-                return default
-            return pathlib.Path(var)  # convert CLI-arguments (string) to Paths.
-
-        self.wheels_path = set_default(self.wheels_path, package_path / 'wheels')
-        self.deps_path = set_default(self.deps_path, self.my_path / 'build/deps')
-        self.pillar_sdk_path = set_default(self.pillar_sdk_path,
-                                           self.deps_path / 'pillar-python-sdk')
-        self.cachecontrol_path = set_default(self.cachecontrol_path,
-                                             self.deps_path / 'cachecontrol')
+        self.wheels_path = set_default_path(self.wheels_path, package_path / 'wheels')
+        self.deps_path = set_default_path(self.deps_path, self.my_path / 'build/deps')
+        self.cachecontrol_path = set_default_path(self.cachecontrol_path,
+                                                  self.deps_path / 'cachecontrol')
 
     def run(self):
         log.info('Storing wheels in %s', self.wheels_path)
@@ -73,23 +94,23 @@ class BuildWheels(Command):
         # Download lockfile, as there is a suitable wheel on pypi.
         if not list(self.wheels_path.glob('lockfile*.whl')):
             log.info('Downloading lockfile wheel')
-            subprocess.check_call([
-                'pip', 'download', '--dest', str(self.wheels_path), requirements['lockfile'][0]
-            ])
+            self.download_wheel(requirements['lockfile'])
 
-        # Build Pillar Python SDK.
-        if not list(self.wheels_path.glob('pillar-python-sdk*.whl')):
-            log.info('Building Pillar Python SDK in %s', self.pillar_sdk_path)
-            self.git_clone(self.pillar_sdk_path,
-                           'https://github.com/armadillica/pillar-python-sdk.git')
-            self.build_copy_wheel(self.pillar_sdk_path)
+        # Download Pillar Python SDK from pypi.
+        if not list(self.wheels_path.glob('pillarsdk*.whl')):
+            log.info('Downloading Pillar Python SDK wheel')
+            self.download_wheel(requirements['pillarsdk'])
 
         # Build CacheControl.
         if not list(self.wheels_path.glob('CacheControl*.whl')):
             log.info('Building CacheControl in %s', self.cachecontrol_path)
+            # self.git_clone(self.cachecontrol_path,
+            #                'https://github.com/ionrock/cachecontrol.git',
+            #                'v%s' % requirements['CacheControl'][1])
+            # FIXME: we need my clone until pull request #125 has been merged & released
             self.git_clone(self.cachecontrol_path,
-                           'https://github.com/ionrock/cachecontrol.git',
-                           'v%s' % requirements['CacheControl'][1])
+                           'https://github.com/sybrenstuvel/cachecontrol.git',
+                           'sybren-filecache-delete-crash-fix')
             self.build_copy_wheel(self.cachecontrol_path)
 
         # Ensure that the wheels are added to the data files.
@@ -97,6 +118,16 @@ class BuildWheels(Command):
             ('blender_cloud/wheels', (str(p) for p in self.wheels_path.glob('*.whl')))
         )
 
+    def download_wheel(self, requirement):
+        """Downloads a wheel from PyPI and saves it in self.wheels_path."""
+
+        subprocess.check_call([
+            'pip', 'download',
+            '--no-deps',
+            '--dest', str(self.wheels_path),
+            requirement[0]
+        ])
+
     def git_clone(self, workdir: pathlib.Path, git_url: str, checkout: str = None):
         if workdir.exists():
             # Directory exists, expect it to be set up correctly.
@@ -124,6 +155,8 @@ class BuildWheels(Command):
         log.info('copying %s to %s', wheel, self.wheels_path)
         shutil.copy(str(wheel), str(self.wheels_path))
 
+
+# noinspection PyAttributeOutsideInit
 class BlenderAddonBdist(bdist):
     """Ensures that 'python setup.py bdist' creates a zip file."""
 
@@ -137,6 +170,35 @@ class BlenderAddonBdist(bdist):
         super().run()
 
 
+# noinspection PyAttributeOutsideInit
+class BlenderAddonFdist(BlenderAddonBdist):
+    """Ensures that 'python setup.py fdist' creates a plain folder structure."""
+
+    user_options = [
+            ('dest-path=', None, 'addon installation path'),
+    ]
+
+    def initialize_options(self):
+        super().initialize_options()
+        self.dest_path = None # path that will contain the addon
+
+    def run(self):
+        super().run()
+
+        # dist_files is a list of tuples ('bdist', 'any', 'filepath')
+        filepath = self.distribution.dist_files[0][2]
+
+        # if dest_path is not specified use the filename as the dest_path (minus the .zip)
+        assert filepath.endswith('.zip')
+        target_folder = self.dest_path or filepath[:-4]
+
+        print('Unzipping the package on {}.'.format(target_folder))
+
+        with zipfile.ZipFile(filepath, 'r') as zip_ref:
+            zip_ref.extractall(target_folder)
+
+
+# noinspection PyAttributeOutsideInit
 class BlenderAddonInstall(install):
     """Ensures the module is placed at the root of the zip file."""
 
@@ -159,16 +221,18 @@ class AvoidEggInfo(install_egg_info):
 
 setup(
     cmdclass={'bdist': BlenderAddonBdist,
+              'fdist': BlenderAddonFdist,
               'install': BlenderAddonInstall,
               'install_egg_info': AvoidEggInfo,
               'wheels': BuildWheels},
     name='blender_cloud',
     description='The Blender Cloud addon allows browsing the Blender Cloud from Blender.',
-    version='1.0.0',
+    version='1.5.999',
     author='Sybren A. Stüvel',
     author_email='sybren@stuvel.eu',
     packages=find_packages('.'),
-    data_files=[('blender_cloud', ['README.md'])],
+    data_files=[('blender_cloud', ['README.md', 'README-flamenco.md']),
+                ('blender_cloud/icons', glob.glob('blender_cloud/icons/*'))],
     scripts=[],
     url='https://developer.blender.org/diffusion/BCA/',
     license='GNU General Public License v2 or later (GPLv2+)',

tests/test_utils.py

@@ -0,0 +1,25 @@
+"""Unittests for blender_cloud.utils."""
+
+import pathlib
+import unittest
+
+from blender_cloud import utils
+
+
+class FindInPathTest(unittest.TestCase):
+    def test_nonexistant_path(self):
+        path = pathlib.Path('/doesnotexistreally')
+        self.assertFalse(path.exists())
+        self.assertIsNone(utils.find_in_path(path, 'jemoeder.blend'))
+
+    def test_really_breadth_first(self):
+        """A depth-first test might find dir_a1/dir_a2/dir_a3/find_me.txt first."""
+
+        path = pathlib.Path(__file__).parent / 'test_really_breadth_first'
+        found = utils.find_in_path(path, 'find_me.txt')
+        self.assertEqual(path / 'dir_b1' / 'dir_b2' / 'find_me.txt', found)
+
+    def test_nonexistant_file(self):
+        path = pathlib.Path(__file__).parent / 'test_really_breadth_first'
+        found = utils.find_in_path(path, 'do_not_find_me.txt')
+        self.assertEqual(None, found)

update_version.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+if [ -z "$1" ]; then
+    echo "Usage: $0 new-version" >&2
+    exit 1
+fi
+
+BL_INFO_VER=$(echo "$1" | sed 's/\./, /g')
+
+sed "s/version='[^']*'/version='$1'/" -i setup.py
+sed "s/'version': ([^)]*)/'version': ($BL_INFO_VER)/" -i blender_cloud/__init__.py
+
+git diff
+echo
+echo "Don't forget to commit!"