Forgot to update poetry.lock

It looks like more recent versions of poetry do not handle well nested
local package deps.
https://github.com/python-poetry/poetry/issues/3098

.gitignore

@@ -3,6 +3,12 @@
 .coverage
 *.pyc
 __pycache__
+*.js.map
+*.css.map
+
+/cloud/templates/
+/cloud/static/assets/
+node_modules/
 
 /config_local.py
 
@@ -12,4 +18,10 @@ __pycache__
 /.eggs/
 /dump/
 /google_app*.json
-/docker/3_run/wheelhouse/
+/docker/2_buildpy/python/
+/docker/4_run/wheelhouse/
+/docker/4_run/deploy/
+/docker/4_run/staging/
+/celerybeat-schedule.bak
+/celerybeat-schedule.dat
+/celerybeat-schedule.dir

README.md

@@ -1,43 +1,102 @@
 # Blender Cloud
 
-Welcome to the [Blender Cloud](https://cloud.blender.org) code repo!
-Blender Cloud runs on the [Pillar](https://pillarframework.org) framework.
+Welcome to the [Blender Cloud](https://cloud.blender.org/) code repo!
+Blender Cloud runs on the [Pillar](https://pillarframework.org/) framework.
 
-## Quick setup
-Set up a node with these commands. Note that that script is already outdated...
+## Development setup
+Jumpstart Blender Cloud development with this simple guide.
+
+### System setup
+Blender Cloud relies on a number of services in order to run. Check out the [Pillar system setup](
+https://pillarframework.org/development/system_setup/#step-by-step-setup) to set this up.
+
+Add `cloud.local` to the `/etc/hosts` file on localhost. This is a development convention.
+
+### Check out the code
+Go to the local development directory and check out the following repositories, next to each other.
 
 ```
-#!/usr/bin/env bash
-
-mkdir -p /data/git
-mkdir -p /data/storage
-mkdir -p /data/config
-mkdir -p /data/certs
-
-sudo apt-get update
-sudo apt-get -y install python-pip
-pip install docker-compose
-
-cd /data/git
+cd /home/guest/Developer
 git clone git://git.blender.org/pillar-python-sdk.git
-git clone git://git.blender.org/pillar-server.git pillar
+git clone git://git.blender.org/pillar.git
 git clone git://git.blender.org/attract.git
 git clone git://git.blender.org/flamenco.git
+git clone git://git.blender.org/pillar-svnman.git
 git clone git://git.blender.org/blender-cloud.git
 ```
 
-## Deploying to production server
+### Initial setup and configuration
 
-First of all, add those aliases to the `[alias]` section of your `~/.gitconfig`
+Create a virtualenv for the project and install the requirements. Dependencies are managed via
+[Poetry](https://poetry.eustace.io/). Install it using `pip install -U --user poetry`.
+
+```
+cd blender-cloud
+pip install --user -U poetry
+poetry install
+```
+
+NOTE: After a dependency changed its own dependencies (say a new library was added as dependency of
+Pillar), you need to run `poetry update`. This will take the new dependencies into account and write
+them to the `poetry.lock` file.
+
+Build assets and templates for all Blender Cloud dependencies using Gulp.
+
+```
+./gulp all
+```
+
+Make a copy of the config_local example, which will be further edited as the application is
+configured.
+
+```
+cp config_local.example.py config_local.py
+```
+
+Setup the database with the initial collections and the admin user.
+
+```
+poetry run ./manage.py setup setup_db your_email
+```
+
+The command will return the following message:
+
+```
+Created project <project_id> for user <user_id>
+```
+
+Copy the value of `<project_id>` and assign it as value for `MAIN_PROJECT_ID`.
+
+Run the application:
+
+```
+poetry run ./manage.py runserver
+```
+
+
+## Development
+
+When ready to commit, change the remotes to work over SSH. For example:
+
+`git remote set-url origin git@git.blender.org:blender-cloud.git`
+
+For more information, check out [this guide](https://wiki.blender.org/wiki/Tools/Git#Commit_Access).
+
+
+## Preparing the production branch for deployment
+
+All revisions to deploy to production should be on the `production` branches of all the relevant
+repositories.
+
+Make sure you have these aliases in the `[alias]` section of your `~/.gitconfig`:
 
 ```
 prod = "!git checkout production && git fetch origin production && gitk --all"
 ff = "merge --ff-only"
-pp = "!git push && if [ -e deploy.sh ]; then ./deploy.sh; fi && git checkout master"
 ```
 
-The following commands should be executed for each subproject; specifically for
-Pillar and Attract:
+The following commands should be executed for each (sub)project; specifically for
+the current repository, Pillar, Attract, Flamenco, and Pillar-SVNMan:
 
 ```
 cd $projectdir
@@ -48,7 +107,7 @@ git stash  # if you still have local stuff.
 
 # pull from master, run unittests, push your changes to master.
 git pull
-py.test
+poetry run py.test
 git push
 
 # Switch to production branch, and investigate the situation.
@@ -58,37 +117,12 @@ git prod
 git ff master
 
 # Run tests again
-py.test
+poetry run py.test
 
-# Push the production branch and run dummy deploy script.
-git pp  # pp = "Push to Production"
-
-# The above alias waits for [ENTER] until all deploys are done.
-# Let it wait, perform the other commands in another terminal.
+# Push the production branch.
+git push
 ```
 
-Now follow the above receipe on the Blender Cloud project as well.
-Contrary to the subprojects, `git pp` will actually perform the deploy
-for real.
+## Deploying to production server
 
-Now you can press `[ENTER]` in the Pillar and Attract terminals that
-were still waiting for it.
-
-After everything is done, your (sub)projects should all be back on
-the master branch.
-
-
-## Updating dependencies via Docker images
-
-To update dependencies that need compiling, you need the `2_build` docker
-container. To rebuild the lot, run `docker/build.sh`.
-
-Follow these steps to deploy the new container on production:
-
-1. run `docker/build.sh`
-2. `docker push armadillica/blender_cloud`
-
-On the production machine:
-
-1. `docker pull armadillica/blender_cloud`
-2. `docker-compose up -d` (from the `/data/git/blender-cloud/docker` directory)
+See [deploy/README.md](deploy/README.md).

cloud/__init__.py

@@ -0,0 +1,155 @@
+import logging
+
+import flask
+from werkzeug.local import LocalProxy
+
+import pillarsdk
+import pillar.auth
+from pillar.api.utils import authorization
+from pillar.extension import PillarExtension
+
+EXTENSION_NAME = 'cloud'
+
+
+class CloudExtension(PillarExtension):
+    has_context_processor = True
+    user_roles = {'subscriber-pro', 'has_subscription'}
+    user_roles_indexable = {'subscriber-pro', 'has_subscription'}
+
+    user_caps = {
+        'has_subscription': {'can-renew-subscription'},
+    }
+
+    def __init__(self):
+        self._log = logging.getLogger('%s.CloudExtension' % __name__)
+
+    @property
+    def name(self):
+        return EXTENSION_NAME
+
+    def flask_config(self):
+        """Returns extension-specific defaults for the Flask configuration.
+
+        Use this to set sensible default values for configuration settings
+        introduced by the extension.
+
+        :rtype: dict
+        """
+
+        # Just so that it registers the management commands.
+        from . import cli
+
+        return {
+            'EXTERNAL_SUBSCRIPTIONS_MANAGEMENT_SERVER': 'https://store.blender.org/api/',
+            'EXTERNAL_SUBSCRIPTIONS_TIMEOUT_SECS': 10,
+            'BLENDER_ID_WEBHOOK_USER_CHANGED_SECRET': 'oos9wah1Zoa0Yau6ahThohleiChephoi',
+            'NODE_TAGS': ['animation', 'modeling', 'rigging', 'sculpting', 'shading', 'texturing', 'lighting',
+                          'character-pipeline',  'effects', 'video-editing', 'digital-painting', 'production-design',
+                          'walk-through'],
+        }
+
+    def eve_settings(self):
+        """Returns extensions to the Eve settings.
+
+        Currently only the DOMAIN key is used to insert new resources into
+        Eve's configuration.
+
+        :rtype: dict
+        """
+
+        return {}
+
+    def blueprints(self):
+        """Returns the list of top-level blueprints for the extension.
+
+        These blueprints will be mounted at the url prefix given to
+        app.load_extension().
+
+        :rtype: list of flask.Blueprint objects.
+        """
+        from . import routes
+        import cloud.stats.routes
+        return [
+            routes.blueprint,
+            cloud.stats.routes.blueprint,
+        ]
+
+    @property
+    def template_path(self):
+        import os.path
+        return os.path.join(os.path.dirname(__file__), 'templates')
+
+    @property
+    def static_path(self):
+        import os.path
+        return os.path.join(os.path.dirname(__file__), 'static')
+
+    def context_processor(self):
+        return {
+            'current_user_is_subscriber': authorization.user_has_cap('subscriber')
+        }
+
+    def is_cloud_project(self, project):
+        """Returns whether the project is set up for Blender Cloud.
+
+        Requires the presence of the 'cloud' key in extension_props
+        """
+
+        if project.extension_props is None:
+            # There are no extension_props on this project
+            return False
+
+        try:
+            pprops = project.extension_props[EXTENSION_NAME]
+        except AttributeError:
+            self._log.warning("is_cloud_project: Project url=%r doesn't have any "
+                              "extension properties.", project['url'])
+            if self._log.isEnabledFor(logging.DEBUG):
+                import pprint
+                self._log.debug('Project: %s', pprint.pformat(project.to_dict()))
+            return False
+        except KeyError:
+            # Not set up for Blender Cloud
+            return False
+
+        if pprops is None:
+            self._log.debug("is_cloud_project: Project url=%r doesn't have Blender Cloud"
+                            " extension properties.", project['url'])
+            return False
+        return True
+
+    @property
+    def has_project_settings(self) -> bool:
+        # Only available for admins
+        return pillar.auth.current_user.has_cap('admin')
+
+    def project_settings(self, project: pillarsdk.Project, **template_args: dict) -> flask.Response:
+        """Renders the project settings page for this extension.
+
+        Set YourExtension.has_project_settings = True and Pillar will call this function.
+
+        :param project: the project for which to render the settings.
+        :param template_args: additional template arguments.
+        :returns: a Flask HTTP response
+        """
+
+        from cloud.routes import project_settings
+
+        return project_settings(project, **template_args)
+
+    def setup_app(self, app):
+        from . import routes, webhooks, eve_hooks, email
+
+        routes.setup_app(app)
+        app.register_api_blueprint(webhooks.blueprint, '/webhooks')
+        eve_hooks.setup_app(app)
+        email.setup_app(app)
+
+
+def _get_current_cloud():
+    """Returns the Cloud extension of the current application."""
+    return flask.current_app.pillar_extensions[EXTENSION_NAME]
+
+
+current_cloud = LocalProxy(_get_current_cloud)
+"""Cloud extension of the current app."""

cloud/cli.py

@@ -0,0 +1,139 @@
+#!/usr/bin/env python
+
+import logging
+from urllib.parse import urljoin
+
+from flask import current_app
+from flask_script import Manager
+import requests
+
+from pillar.cli import manager
+from pillar.api import service
+from pillar.api.utils import authentication
+import cloud.setup
+
+log = logging.getLogger(__name__)
+
+manager_cloud = Manager(
+    current_app, usage="Blender Cloud scripts")
+
+
+@manager_cloud.command
+def create_groups():
+    """Creates the admin/demo/subscriber groups."""
+
+    import pprint
+
+    group_ids = {}
+    groups_coll = current_app.db('groups')
+
+    for group_name in ['admin', 'demo', 'subscriber']:
+        if groups_coll.find({'name': group_name}).count():
+            log.info('Group %s already exists, skipping', group_name)
+            continue
+        result = groups_coll.insert_one({'name': group_name})
+        group_ids[group_name] = result.inserted_id
+
+    service.fetch_role_to_group_id_map()
+    log.info('Created groups:\n%s', pprint.pformat(group_ids))
+
+
+@manager_cloud.command
+def reconcile_subscribers():
+    """For every user, check their subscription status with the store."""
+
+    import threading
+    import concurrent.futures
+
+    from pillar.auth import UserClass
+    from pillar.api import blender_id
+    from pillar.api.blender_cloud.subscription import do_update_subscription
+
+    sessions = threading.local()
+
+    service.fetch_role_to_group_id_map()
+
+    users_coll = current_app.data.driver.db['users']
+    found = users_coll.find({'auth.provider': 'blender-id'})
+    count_users = found.count()
+    count_skipped = count_processed = 0
+    log.info('Processing %i users', count_users)
+
+    lock = threading.Lock()
+
+    real_current_app = current_app._get_current_object()
+
+    api_token = real_current_app.config['BLENDER_ID_USER_INFO_TOKEN']
+    api_url = real_current_app.config['BLENDER_ID_USER_INFO_API']
+
+    def do_user(idx, user):
+        nonlocal count_skipped, count_processed
+
+        log.info('Processing %i/%i %s', idx + 1, count_users, user['email'])
+
+        # Get the Requests session for this thread.
+        try:
+            sess = sessions.session
+        except AttributeError:
+            sess = sessions.session = requests.Session()
+
+        # Get the info from Blender ID
+        bid_user_id = blender_id.get_user_blenderid(user)
+        if not bid_user_id:
+            with lock:
+                count_skipped += 1
+            return
+
+        url = urljoin(api_url, bid_user_id)
+        resp = sess.get(url, headers={'Authorization': f'Bearer {api_token}'})
+
+        if resp.status_code == 404:
+            log.info('User %s with Blender ID %s not found, skipping', user['email'], bid_user_id)
+            with lock:
+                count_skipped += 1
+            return
+
+        if resp.status_code != 200:
+            log.error('Unable to reach Blender ID (code %d), aborting', resp.status_code)
+            with lock:
+                count_skipped += 1
+            return
+
+        bid_user = resp.json()
+        if not bid_user:
+            log.error('Unable to parse response for user %s, aborting', user['email'])
+            with lock:
+                count_skipped += 1
+            return
+
+        # Actually update the user, and do it thread-safe just to be sure.
+        with real_current_app.app_context():
+            local_user = UserClass.construct('', user)
+            with lock:
+                do_update_subscription(local_user, bid_user)
+                count_processed += 1
+
+    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
+        future_to_user = {executor.submit(do_user, idx, user): user
+                          for idx, user in enumerate(found)}
+        for future in concurrent.futures.as_completed(future_to_user):
+            user = future_to_user[future]
+            try:
+                future.result()
+            except Exception as ex:
+                log.exception('Error updating user %s', user)
+
+    log.info('Done reconciling %d subscribers', count_users)
+    log.info('    processed: %d', count_processed)
+    log.info('    skipped  : %d', count_skipped)
+
+
+@manager_cloud.command
+def setup_for_film(project_url):
+    """Adds Blender Cloud film custom properties to a project."""
+
+    authentication.force_cli_user()
+    cloud.setup.setup_for_film(project_url)
+
+
+manager.add_command("cloud", manager_cloud)

cloud/email.py

@@ -0,0 +1,33 @@
+import functools
+import logging
+
+import flask
+
+from pillar.auth import UserClass
+
+log = logging.getLogger(__name__)
+
+
+def queue_welcome_mail(user: UserClass):
+    """Queue a welcome email for execution by Celery."""
+    assert user.email
+    log.info('queueing welcome email to %s', user.email)
+
+    subject = 'Welcome to Blender Cloud'
+    render = functools.partial(flask.render_template, subject=subject, user=user)
+    text = render('emails/welcome.txt')
+    html = render('emails/welcome.html')
+
+    from pillar.celery import email_tasks
+    email_tasks.send_email.delay(user.full_name, user.email, subject, text, html)
+
+
+def user_subscription_changed(user: UserClass, *, grant_roles: set, revoke_roles: set):
+    if user.has_cap('subscriber') and 'has_subscription' in grant_roles:
+        log.info('user %s just got a new subscription', user.email)
+        queue_welcome_mail(user)
+
+
+def setup_app(app):
+    from pillar.api.blender_cloud import subscription
+    subscription.user_subscription_updated.connect(user_subscription_changed)

cloud/eve_hooks.py

@@ -0,0 +1,39 @@
+import logging
+import typing
+
+from pillar.auth import UserClass
+
+from . import email
+
+log = logging.getLogger(__name__)
+
+
+def welcome_new_user(user_doc: dict):
+    """Sends a welcome email to a new user."""
+
+    user_email = user_doc.get('email')
+    if not user_email:
+        log.warning('user %s has no email address', user_doc.get('_id', '-no-id-'))
+        return
+
+    # Only send mail to new users when they actually are subscribers.
+    user = UserClass.construct('', user_doc)
+    if not (user.has_cap('subscriber') or user.has_cap('can-renew-subscription')):
+        log.debug('user %s is new, but not a subscriber, so no email for them.', user_email)
+        return
+
+    email.queue_welcome_mail(user)
+
+
+def welcome_new_users(user_docs: typing.List[dict]):
+    """Sends a welcome email to new users."""
+
+    for user_doc in user_docs:
+        try:
+            welcome_new_user(user_doc)
+        except Exception:
+            log.exception('error sending welcome mail to user %s', user_doc)
+
+
+def setup_app(app):
+    app.on_inserted_users += welcome_new_users

cloud/forms.py

@@ -0,0 +1,15 @@
+from flask_wtf import FlaskForm
+from wtforms import BooleanField, StringField
+from wtforms.fields.html5 import URLField
+from wtforms.validators import URL
+
+from pillar.web.utils.forms import FileSelectField
+
+
+class FilmProjectForm(FlaskForm):
+    video_url = URLField(validators=[URL()])
+    poster = FileSelectField('Poster Image', file_format='image')
+    logo = FileSelectField('Logo', file_format='image')
+    is_in_production = BooleanField('In Production')
+    is_featured = BooleanField('Featured')
+    theme_color = StringField('Theme Color')

cloud/routes.py

@@ -0,0 +1,658 @@
+import functools
+import json
+import logging
+import typing
+
+import bson
+from flask_login import login_required
+import flask
+import werkzeug.exceptions as wz_exceptions
+from flask import Blueprint, render_template, redirect, session, url_for, abort, flash, request
+from pillarsdk import Node, Project, User, exceptions as sdk_exceptions, Group
+from pillarsdk.exceptions import ResourceNotFound
+
+import pillar
+import pillarsdk
+from pillar import current_app
+from pillar.api.utils import authorization
+from pillar.auth import current_user
+from pillar.web.users import forms
+from pillar.web.utils import system_util, get_file, current_user_is_authenticated
+from pillar.web.utils import attach_project_pictures
+from pillar.web.settings import blueprint as blueprint_settings
+from pillar.web.nodes.routes import url_for_node
+from pillar.web.projects.routes import render_project
+from pillar.web.projects.routes import find_project_or_404
+from pillar.web.projects.routes import project_view
+from pillar.web.projects.routes import project_navigation_links
+
+from cloud import current_cloud
+from cloud.forms import FilmProjectForm
+from . import EXTENSION_NAME
+
+blueprint = Blueprint('cloud', __name__)
+log = logging.getLogger(__name__)
+
+
+@blueprint.route('/')
+def homepage():
+    if current_user.is_anonymous:
+        return redirect(url_for('cloud.welcome'))
+
+    return render_template(
+        'homepage.html',
+        api=system_util.pillar_api(),
+        **_homepage_context(),
+    )
+
+
+def _homepage_context() -> dict:
+    """Returns homepage template context variables."""
+
+    # Get latest blog posts
+    api = system_util.pillar_api()
+
+    # Get latest comments to any node
+    latest_comments = Node.latest('comments', api=api)
+
+    # Get a list of random featured assets
+    random_featured = get_random_featured_nodes()
+
+    # Parse results for replies
+    to_remove = []
+
+    @functools.lru_cache()
+    def _find_parent(parent_node_id) -> Node:
+        return Node.find(parent_node_id,
+                         {'projection': {
+                             '_id': 1,
+                             'name': 1,
+                             'node_type': 1,
+                             'project': 1,
+                             'parent': 1,
+                             'properties.url': 1,
+                         }},
+                         api=api)
+
+    for idx, comment in enumerate(latest_comments._items):
+        if comment.properties.is_reply:
+            try:
+                comment.attached_to = _find_parent(comment.parent.parent)
+            except ResourceNotFound:
+                # Remove this comment
+                to_remove.append(idx)
+        else:
+            comment.attached_to = comment.parent
+
+    for idx in reversed(to_remove):
+        del latest_comments._items[idx]
+
+    for comment in latest_comments._items:
+        if not comment.attached_to:
+            continue
+        comment.attached_to.url = url_for_node(node=comment.attached_to)
+        comment.url = url_for_node(node=comment)
+
+    main_project = Project.find(current_app.config['MAIN_PROJECT_ID'], api=api)
+    main_project.picture_header = get_file(main_project.picture_header, api=api)
+
+    return dict(
+        main_project=main_project,
+        latest_comments=latest_comments._items,
+        random_featured=random_featured)
+
+
+@blueprint.route('/design-system')
+def design_system():
+    """Display the design system page.
+
+    This endpoing is intended for development only, and returns a
+    rendered template only if the app is running in debug mode.
+    """
+    if not current_app.config['DEBUG']:
+        abort(404)
+    return render_template('design_system.html')
+
+
+@blueprint.route('/login')
+def login():
+    from flask import request
+
+    if request.args.get('force'):
+        log.debug('Forcing logout of user before rendering login page.')
+        pillar.auth.logout_user()
+
+    next_after_login = request.args.get('next')
+
+    if not next_after_login:
+        next_after_login = request.referrer
+
+    session['next_after_login'] = next_after_login
+    return redirect(url_for('users.oauth_authorize', provider='blender-id'))
+
+
+@blueprint.route('/welcome')
+def welcome():
+    # Workaround to cache rendering of a page if user not logged in
+    @current_app.cache.cached(timeout=3600, unless=current_user_is_authenticated)
+    def render_page():
+        return render_template('welcome.html')
+
+    return render_page()
+
+
+@blueprint.route('/about')
+def about():
+    return render_template('about.html')
+
+
+@blueprint.route('/services')
+def services():
+    return render_template('services.html')
+
+
+@blueprint.route('/learn')
+def learn():
+    return render_template('learn.html')
+
+
+@blueprint.route('/libraries')
+def libraries():
+    return render_template('libraries.html')
+
+
+@blueprint.route('/stats')
+def stats():
+    return render_template('stats.html')
+
+
+@blueprint.route('/join')
+def join():
+    """Join page"""
+    return redirect('https://store.blender.org/product/membership/')
+
+
+@blueprint.route('/renew')
+def renew_subscription():
+    return render_template('renew_subscription.html')
+
+
+def get_projects(category):
+    """Utility to get projects based on category. Should be moved on the API
+    and improved with more extensive filtering capabilities.
+    """
+    api = system_util.pillar_api()
+    projects = Project.all({
+        'where': {
+            'category': category,
+            'is_private': False},
+        'sort': '-_created',
+    }, api=api)
+    for project in projects._items:
+        attach_project_pictures(project, api)
+    return projects
+
+
+@blueprint.route('/courses')
+def courses():
+    @current_app.cache.cached(timeout=3600, unless=current_user_is_authenticated)
+    def render_page():
+        projects = get_projects('course')
+        return render_template(
+            'projects_index_collection.html',
+            title='courses',
+            projects=projects._items,
+            api=system_util.pillar_api())
+
+    return render_page()
+
+
+@blueprint.route('/open-projects')
+def open_projects():
+    @current_app.cache.cached(timeout=3600, unless=current_user_is_authenticated)
+    def render_page():
+        api = system_util.pillar_api()
+        projects = Project.all({
+            'where': {
+                'category': 'film',
+                'is_private': False
+            },
+            'sort': '-_created',
+        }, api=api)
+        for project in projects._items:
+            # Attach poster file (ensure the extension_props.cloud.poster
+            # attributes exists)
+            try:
+                # If the attribute exists, but is None, continue
+                if not project['extension_props'][EXTENSION_NAME]['poster']:
+                    continue
+                # Fetch the file and embed it in the document
+                project.extension_props.cloud.poster = get_file(
+                    project.extension_props.cloud.poster, api=api)
+                # Add convenience attribute that specifies the presence of the
+                # poster file
+                project.has_poster = True
+            # If there was a key error because one of the nested attributes is
+            # missing,
+            except KeyError:
+                continue
+
+        return render_template(
+            'films.html',
+            title='films',
+            projects=projects._items,
+            api=system_util.pillar_api())
+
+    return render_page()
+
+
+@blueprint.route('/workshops')
+def workshops():
+    @current_app.cache.cached(timeout=3600, unless=current_user_is_authenticated)
+    def render_page():
+        projects = get_projects('workshop')
+        return render_template(
+            'projects_index_collection.html',
+            title='workshops',
+            projects=projects._items,
+            api=system_util.pillar_api())
+
+    return render_page()
+
+
+def get_random_featured_nodes() -> typing.List[dict]:
+    """Returns a list of project/node combinations for featured nodes.
+
+    A random subset of 3 featured nodes from all public projects is returned.
+    Assumes that the user actually has access to the public projects' nodes.
+
+    The dict is a node, with a 'project' key that contains a projected project.
+    """
+
+    proj_coll = current_app.db('projects')
+    featured_nodes = proj_coll.aggregate([
+        {'$match': {'is_private': False}},
+        {'$project': {'nodes_featured': True,
+                      'url': True,
+                      'name': True,
+                      'summary': True,
+                      'picture_square': True}},
+        {'$unwind': {'path': '$nodes_featured'}},
+        {'$sample': {'size': 6}},
+        {'$lookup': {'from': 'nodes',
+                     'localField': 'nodes_featured',
+                     'foreignField': '_id',
+                     'as': 'node'}},
+        {'$unwind': {'path': '$node'}},
+        {'$match': {'node._deleted': {'$ne': True}}},
+        {'$project': {'url': True,
+                      'name': True,
+                      'summary': True,
+                      'picture_square': True,
+                      'node._id': True,
+                      'node.name': True,
+                      'node.permissions': True,
+                      'node.picture': True,
+                      'node.properties.content_type': True,
+                      'node.properties.duration_seconds': True,
+                      'node.properties.url': True,
+                      'node._created': True,
+                      }
+         },
+    ])
+
+    featured_node_documents = []
+    api = system_util.pillar_api()
+    for node_info in featured_nodes:
+        # Turn the project-with-node doc into a node-with-project doc.
+        node_document = node_info.pop('node')
+        node_document['project'] = node_info
+        node_document['_id'] = str(node_document['_id'])
+
+        node = Node(node_document)
+        node.picture = get_file(node.picture, api=api)
+        node.project.picture_square = get_file(node.project.picture_square, api=api)
+        featured_node_documents.append(node)
+
+    return featured_node_documents
+
+
+@blueprint_settings.route('/emails', methods=['GET', 'POST'])
+@login_required
+def emails():
+    """Main email settings.
+    """
+    if current_user.has_role('protected'):
+        return abort(404)  # TODO: make this 403, handle template properly
+    api = system_util.pillar_api()
+    user = User.find(current_user.objectid, api=api)
+
+    # Force creation of settings for the user (safely remove this code once
+    # implemented on account creation level, and after adding settings to all
+    # existing users)
+    if not user.settings:
+        user.settings = dict(email_communications=1)
+        user.update(api=api)
+
+    if user.settings.email_communications is None:
+        user.settings.email_communications = 1
+        user.update(api=api)
+
+    # Generate form
+    form = forms.UserSettingsEmailsForm(
+        email_communications=user.settings.email_communications)
+
+    if form.validate_on_submit():
+        try:
+            user.settings.email_communications = form.email_communications.data
+            user.update(api=api)
+            flash("Profile updated", 'success')
+        except sdk_exceptions.ResourceInvalid as e:
+            message = json.loads(e.content)
+            flash(message)
+
+    return render_template('users/settings/emails.html', form=form, title='emails')
+
+
+@blueprint_settings.route('/billing')
+@login_required
+def billing():
+    """View the subscription status of a user
+    """
+    from . import store
+
+    log.debug('START OF REQUEST')
+
+    if current_user.has_role('protected'):
+        return abort(404)  # TODO: make this 403, handle template properly
+
+    expiration_date = 'No subscription to expire'
+
+    # Classify the user based on their roles and capabilities
+    cap_subs = current_user.has_cap('subscriber')
+    if current_user.has_role('demo'):
+        user_cls = 'demo'
+    elif not cap_subs and current_user.has_cap('can-renew-subscription'):
+        # This user has an inactive but renewable subscription.
+        user_cls = 'subscriber-expired'
+    elif cap_subs:
+        if current_user.has_role('subscriber'):
+            # This user pays for their own subscription. Only in this case do we need to fetch
+            # the expiration date from the Store.
+            user_cls = 'subscriber'
+            store_user = store.fetch_subscription_info(current_user.email)
+            if store_user is None:
+                expiration_date = 'Unable to reach Blender Store to check'
+            else:
+                expiration_date = store_user['expiration_date'][:10]
+
+        elif current_user.has_role('org-subscriber'):
+            # An organisation pays for this subscription.
+            user_cls = 'subscriber-org'
+        else:
+            # This user gets the subscription cap from somewhere else (like an organisation).
+            user_cls = 'subscriber-other'
+    else:
+        user_cls = 'outsider'
+
+    return render_template(
+        'users/settings/billing.html',
+        user_cls=user_cls,
+        expiration_date=expiration_date,
+        title='billing')
+
+
+@blueprint.route('/terms-and-conditions')
+def terms_and_conditions():
+    return render_template('terms_and_conditions.html')
+
+
+@blueprint.route('/privacy')
+def privacy():
+    return render_template('privacy.html')
+
+
+@blueprint.route('/production')
+def production():
+    return render_template(
+        'production.html',
+        title='production')
+
+
+@blueprint.route('/emails/welcome.send')
+@login_required
+def emails_welcome_send():
+    from cloud import email
+    email.queue_welcome_mail(current_user)
+    return f'queued mail to {current_user.email}'
+
+
+@blueprint.route('/emails/welcome.html')
+@login_required
+def emails_welcome_html():
+    return render_template('emails/welcome.html',
+                           subject='Welcome to Blender Cloud',
+                           user=current_user)
+
+
+@blueprint.route('/emails/welcome.txt')
+@login_required
+def emails_welcome_txt():
+    txt = render_template('emails/welcome.txt',
+                          subject='Welcome to Blender Cloud',
+                          user=current_user)
+    return flask.Response(txt, content_type='text/plain; charset=utf-8')
+
+
+@blueprint.route('/p/<project_url>')
+def project_landing(project_url):
+    """Override Pillar project_view endpoint completely.
+
+    The first part of the function is identical to the one in Pillar, but the
+    second part (starting with 'Load custom project properties') extends the
+    behaviour to support film project landing pages.
+    """
+
+    template_name = None
+    if request.args.get('format') == 'jstree':
+        log.warning('projects.view(%r) endpoint called with format=jstree, '
+                    'redirecting to proper endpoint. URL is %s; referrer is %s',
+                    project_url, request.url, request.referrer)
+        return redirect(url_for('projects.jstree', project_url=project_url))
+
+    api = system_util.pillar_api()
+    project = find_project_or_404(project_url,
+                                  embedded={'header_node': 1},
+                                  api=api)
+
+    # Load the header video file, if there is any.
+    header_video_file = None
+    header_video_node = None
+    if project.header_node and project.header_node.node_type == 'asset' and \
+            project.header_node.properties.content_type == 'video':
+        header_video_node = project.header_node
+        header_video_file = get_file(project.header_node.properties.file)
+        header_video_node.picture = get_file(header_video_node.picture)
+
+    extra_context = {
+        'header_video_file': header_video_file,
+        'header_video_node': header_video_node}
+
+    # Load custom project properties. If the project has a 'cloud' extension prop,
+    # render it using the projects/landing.html template and try to attach a
+    # number of additional attributes (pages, images, etc.).
+    if 'extension_props' in project and EXTENSION_NAME in project['extension_props']:
+        extension_props = project['extension_props'][EXTENSION_NAME]
+        extension_props['logo'] = get_file(extension_props['logo'])
+
+        pages = Node.all({
+            'where': {
+                'project': project._id,
+                'node_type': 'page',
+                '_deleted': {'$ne': True}},
+            'projection': {'name': 1}}, api=api)
+
+        extra_context.update({'pages': pages._items})
+        template_name = 'projects/landing.html'
+
+    return render_project(project, api,
+                          extra_context=extra_context,
+                          template_name=template_name)
+
+
+@blueprint.route('/p/<project_url>/browse')
+@project_view()
+def project_browse(project: pillarsdk.Project):
+    """Project view displaying all top-level nodes.
+
+    We render a regular project view, but we introduce an additional template
+    variable: browse. By doing that we prevent the regular project view
+    from loading and fetch via AJAX a "group" node-like view instead (see
+    project_browse_view_nodes).
+    """
+    return render_template(
+        'projects/view.html',
+        api=system_util.pillar_api(),
+        project=project,
+        node=None,
+        show_project=True,
+        browse=True,
+        og_picture=None,
+        navigation_links=project_navigation_links(project, system_util.pillar_api()),
+        extension_sidebar_links=current_app.extension_sidebar_links(project))
+
+
+@blueprint.route('/p/<project_url>/browse/nodes')
+@project_view()
+def project_browse_view_nodes(project: pillarsdk.Project):
+    """Display top-level nodes for a Project.
+
+    This view is always meant to be served embedded, as part of project_browse.
+    """
+    api = system_util.pillar_api()
+    # Get top level nodes
+    projection = {
+        'project': 1,
+        'name': 1,
+        'picture': 1,
+        'node_type': 1,
+        'properties.order': 1,
+        'properties.status': 1,
+        'user': 1,
+        'properties.content_type': 1,
+        'permissions.world': 1}
+    where = {
+        'project': project['_id'],
+        'parent': {'$exists': False},
+        'properties.status': 'published',
+        '_deleted': {'$ne': True},
+        'node_type': {'$in': ['group', 'asset']},
+    }
+
+    try:
+        nodes = Node.all({
+            'projection': projection,
+            'where': where,
+            'sort': [('properties.order', 1), ('name', 1)]}, api=api)
+    except pillarsdk.exceptions.ForbiddenAccess:
+        return render_template('errors/403_embed.html')
+    nodes = nodes._items
+
+    for child in nodes:
+        child.picture = get_file(child.picture, api=api)
+    return render_template(
+        'projects/browse_embed.html',
+        nodes=nodes)
+
+
+def project_settings(project: pillarsdk.Project, **template_args: dict):
+    """Renders the project settings page for Blender Cloud projects.
+
+    If the project has been setup for Blender Cloud, check for the cloud.category
+    property, to render the proper form.
+    """
+
+    # Based on the project state, we can render a different template.
+    if not current_cloud.is_cloud_project(project):
+        return render_template('project_settings/offer_setup.html',
+                               project=project, **template_args)
+
+    cloud_props = project['extension_props'][EXTENSION_NAME]
+
+    category = cloud_props['category']
+    if category != 'film':
+        log.error('No interface available to edit %s projects, yet' % category)
+
+    form = FilmProjectForm()
+
+    # Iterate over the form fields and set the data if exists in the project document
+    for field_name in form.data:
+        if field_name not in cloud_props:
+            continue
+        # Skip csrf_token field
+        if field_name == 'csrf_token':
+            continue
+        form_field = getattr(form, field_name)
+        form_field.data = cloud_props[field_name]
+
+    return render_template('project_settings/settings.html',
+                           project=project,
+                           form=form,
+                           **template_args)
+
+
+@blueprint.route('/<project_url>/settings/film', methods=['POST'])
+@authorization.require_login(require_cap='admin')
+@project_view()
+def save_film_settings(project: pillarsdk.Project):
+    # Ensure that the project is setup for Cloud (see @attract_project_view for example)
+    form = FilmProjectForm()
+    if not form.validate_on_submit():
+        log.debug('Form submission failed')
+        # Return list of validation errors
+
+    updated_extension_props = {}
+    for field_name in form.data:
+        # Skip csrf_token field
+        if field_name == 'csrf_token':
+            continue
+        form_field = getattr(form, field_name)
+        # TODO(fsiddi) if form_field type is FileSelectField, convert it to ObjectId
+        # Currently this raises TypeError: Object of type 'ObjectId' is not JSON serializable
+
+        if form_field.data == '':
+            form_field.data = None
+        updated_extension_props[field_name] = form_field.data
+
+    # Update extension props and save project
+    extension_props = project['extension_props'][EXTENSION_NAME]
+    # Project is a Resource, so we update properties iteratively
+    for k, v in updated_extension_props.items():
+        extension_props[k] = v
+    project.update(api=system_util.pillar_api())
+    return '', 204
+
+
+@blueprint.route('/<project_url>/setup-for-film', methods=['POST'])
+@login_required
+@project_view()
+def setup_for_film(project: pillarsdk.Project):
+    import cloud.setup
+
+    project_id = project._id
+
+    if not project.has_method('PUT'):
+        log.warning('User %s tries to set up project %s for Blender Cloud, but has no PUT rights.',
+                    current_user, project_id)
+        raise wz_exceptions.Forbidden()
+
+    log.info('User %s sets up project %s for Blender Cloud', current_user, project_id)
+    cloud.setup.setup_for_film(project.url)
+
+    return '', 204
+
+
+def setup_app(app):
+    global _homepage_context
+    cached = app.cache.cached(timeout=300)
+    _homepage_context = cached(_homepage_context)

cloud/setup.py

@@ -0,0 +1,54 @@
+"""Setting up projects for Blender Cloud."""
+
+import logging
+
+from bson import ObjectId
+from eve.methods.put import put_internal
+from flask import current_app
+
+from pillar.api.utils import remove_private_keys
+from . import EXTENSION_NAME
+
+log = logging.getLogger(__name__)
+
+
+def setup_for_film(project_url):
+    """Add Blender Cloud extension_props specific for film projects.
+
+    Returns the updated project.
+    """
+
+    projects_collection = current_app.data.driver.db['projects']
+
+    # Find the project in the database.
+    project = projects_collection.find_one({'url': project_url})
+    if not project:
+        raise RuntimeError('Project %s does not exist.' % project_url)
+
+    # Set default extension properties. Be careful not to overwrite any properties that
+    # are already there.
+    all_extension_props = project.setdefault('extension_props', {})
+    cloud_extension_props = {
+        'category': 'film',
+        'theme_css': '',
+        # The accent color (can be 'blue' or '#FFBBAA' or 'rgba(1, 1, 1, 1)
+        'theme_color': '',
+        'is_in_production': False,
+        'video_url': '',  # Oembeddable url
+        'poster': None,  # File ObjectId
+        'logo': None,  # File ObjectId
+        # TODO(fsiddi) when we introduce other setup_for_* in Blender Cloud, make available
+        # at a higher scope
+        'is_featured': False,
+    }
+
+    all_extension_props.setdefault(EXTENSION_NAME, cloud_extension_props)
+
+    project_id = ObjectId(project['_id'])
+    project = remove_private_keys(project)
+    result, _, _, status_code = put_internal('projects', project, _id=project_id)
+
+    if status_code != 200:
+        raise RuntimeError("Can't update project %s, issues: %s", project_id, result)
+
+    log.info('Project %s was updated for Blender Cloud.', project_url)

cloud/stats/__init__.py

@@ -0,0 +1,89 @@
+"""Interesting usage metrics"""
+
+from flask import current_app
+
+
+def count_nodes(query=None) -> int:
+    pipeline = [
+        {'$match': {'_deleted': {'$ne': 'true'}}},
+        {
+            '$lookup':
+                {
+                    'from': "projects",
+                    'localField': "project",
+                    'foreignField': "_id",
+                    'as': "project",
+                }
+        },
+        {
+            '$unwind':
+                {
+                    'path': '$project',
+                }
+        },
+        {
+            '$project':
+                {
+                    'project.is_private': 1,
+                }
+        },
+        {'$match': {'project.is_private': False}},
+        {'$count': 'tot'}
+    ]
+    c = current_app.db()['nodes']
+    # If we provide a query, we extend the first $match step in the aggregation pipeline with
+    # with the extra parameters (for example node_type)
+    if query:
+        pipeline[0]['$match'].update(query)
+    # Return either a list with one item or an empty list
+    r = list(c.aggregate(pipeline=pipeline))
+    count = 0 if not r else r[0]['tot']
+    return count
+
+
+def count_users(query=None) -> int:
+    u = current_app.db()['users']
+    return u.count(query)
+
+
+def count_blender_sync(query=None) -> int:
+    pipeline = [
+        # 0 Find all startups.blend that are not deleted
+        {
+            '$match': {
+                '_deleted': {'$ne': 'true'},
+                'name': 'startup.blend',
+            }
+        },
+        # 1 Group them per project (drops any duplicates)
+        {'$group': {'_id': '$project'}},
+        # 2 Join the project info
+        {
+            '$lookup':
+                {
+                    'from': "projects",
+                    'localField': "_id",
+                    'foreignField': "_id",
+                    'as': "project",
+                }
+        },
+        # 3 Unwind the project list (there is always only one project)
+        {
+            '$unwind':
+                {
+                    'path': '$project',
+                }
+        },
+        # 4 Find all home projects
+        {'$match': {'project.category': 'home'}},
+        {'$count': 'tot'}
+    ]
+    c = current_app.db()['nodes']
+    # If we provide a query, we extend the first $match step in the aggregation pipeline with
+    # with the extra parameters (for example _created)
+    if query:
+        pipeline[0]['$match'].update(query)
+    # Return either a list with one item or an empty list
+    r = list(c.aggregate(pipeline=pipeline))
+    count = 0 if not r else r[0]['tot']
+    return count

cloud/stats/routes.py

@@ -0,0 +1,57 @@
+import logging
+import datetime
+import functools
+
+from flask import Blueprint, jsonify
+from cloud.stats import count_nodes, count_users, count_blender_sync
+
+blueprint = Blueprint('cloud.stats', __name__, url_prefix='/s')
+
+log = logging.getLogger(__name__)
+
+
+@functools.lru_cache()
+def get_stats(before: datetime.datetime):
+    query_comments = {'node_type': 'comment'}
+    query_assets = {'node_type': 'asset'}
+
+    date_query = {}
+
+    if before:
+        date_query = {'_created': {'$lt': before}}
+        query_comments.update(date_query)
+        query_assets.update(date_query)
+
+    stats = {
+        'comments': count_nodes(query_comments),
+        'assets': count_nodes(query_assets),
+        'users_total': count_users(date_query),
+        'users_blender_sync': count_blender_sync(date_query),
+    }
+    return stats
+
+
+@blueprint.route('/')
+@blueprint.route('/before/<int:before>')
+def index(before: int=0):
+    """
+    This endpoint is queried on a daily basis by grafista to retrieve cloud usage
+    stats. For assets and comments we take into considerations only those who belong
+    to public projects.
+
+    These is the data we retrieve
+
+    - Comments count
+    - Assets count (video, images and files)
+    - Users count (subscribers count goes via store)
+    - Blender Sync users
+    """
+
+    # TODO: Implement project-level metrics (and update ad every child update)
+    if before:
+        before = datetime.datetime.strptime(str(before), '%Y%m%d')
+    else:
+        today = datetime.date.today()
+        before = datetime.datetime(today.year, today.month, today.day)
+
+    return jsonify(get_stats(before))

cloud/store.py

@@ -0,0 +1,72 @@
+"""Blender Store interface."""
+
+import logging
+import typing
+
+from pillar import current_app
+
+log = logging.getLogger(__name__)
+
+
+def fetch_subscription_info(email: str) -> typing.Optional[dict]:
+    """Returns the user info dict from the external subscriptions management server.
+
+    :returns: the store user info, or None if the user can't be found or there
+        was an error communicating. A dict like this is returned:
+        {
+            "shop_id": 700,
+            "cloud_access": 1,
+            "paid_balance": 314.75,
+            "balance_currency": "EUR",
+            "start_date": "2014-08-25 17:05:46",
+            "expiration_date": "2016-08-24 13:38:45",
+            "subscription_status": "wc-active",
+            "expiration_date_approximate": true
+        }
+    """
+
+    from requests.adapters import HTTPAdapter
+    import requests.exceptions
+
+    external_subscriptions_server = current_app.config['EXTERNAL_SUBSCRIPTIONS_MANAGEMENT_SERVER']
+
+    if log.isEnabledFor(logging.DEBUG):
+        import urllib.parse
+
+        log_email = urllib.parse.quote(email)
+        log.debug('Connecting to store at %s?blenderid=%s',
+                  external_subscriptions_server, log_email)
+
+    # Retry a few times when contacting the store.
+    s = requests.Session()
+    s.mount(external_subscriptions_server, HTTPAdapter(max_retries=5))
+
+    try:
+        r = s.get(external_subscriptions_server,
+                  params={'blenderid': email},
+                  verify=current_app.config['TLS_CERT_FILE'],
+                  timeout=current_app.config.get('EXTERNAL_SUBSCRIPTIONS_TIMEOUT_SECS', 10))
+    except requests.exceptions.ConnectionError as ex:
+        log.error('Error connecting to %s: %s', external_subscriptions_server, ex)
+        return None
+    except requests.exceptions.Timeout as ex:
+        log.error('Timeout communicating with %s: %s', external_subscriptions_server, ex)
+        return None
+    except requests.exceptions.RequestException as ex:
+        log.error('Some error communicating with %s: %s', external_subscriptions_server, ex)
+        return None
+
+    if r.status_code != 200:
+        log.warning("Error communicating with %s, code=%i, unable to check "
+                    "subscription status of user %s",
+                    external_subscriptions_server, r.status_code, email)
+        return None
+
+    store_user = r.json()
+
+    if log.isEnabledFor(logging.DEBUG):
+        import json
+        log.debug('Received JSON from store API: %s',
+                  json.dumps(store_user, sort_keys=False, indent=4))
+
+    return store_user

cloud/tagged/__init__.py

@@ -0,0 +1,3 @@
+"""Routes for fetching tagged assets."""
+
+

cloud/tagged/routes.py

@@ -0,0 +1,16 @@
+import logging
+import datetime
+import functools
+
+from flask import Blueprint, jsonify
+
+blueprint = Blueprint('cloud.tagged', __name__, url_prefix='/tagged')
+
+log = logging.getLogger(__name__)
+
+
+@blueprint.route('/')
+def index():
+    """Return all tagged assets as JSON, grouped by tag."""
+
+

cloud/webhooks.py

@@ -0,0 +1,263 @@
+"""Blender ID webhooks."""
+
+import functools
+import hashlib
+import hmac
+import json
+import logging
+import typing
+
+from flask import Blueprint, request
+import werkzeug.exceptions as wz_exceptions
+
+from pillar import current_app
+from pillar.api.blender_cloud import subscription
+from pillar.api.utils.authentication import create_new_user_document, make_unique_username
+from pillar.auth import UserClass
+
+blueprint = Blueprint('cloud-webhooks', __name__)
+log = logging.getLogger(__name__)
+WEBHOOK_MAX_BODY_SIZE = 1024 * 10  # 10 kB is large enough for
+
+
+def webhook_payload(hmac_secret: str) -> dict:
+    """Obtains the webhook payload from the request, verifying its HMAC.
+
+    :returns the webhook payload as dictionary.
+    """
+    # Check the content type
+    if request.content_type != 'application/json':
+        log.info('request from %s to %s had bad content type %s',
+                 request.remote_addr, request, request.content_type)
+        raise wz_exceptions.BadRequest('Content type not supported')
+
+    # Check the length of the body
+    if request.content_length > WEBHOOK_MAX_BODY_SIZE:
+        raise wz_exceptions.BadRequest('Request too large')
+    body = request.get_data()
+    if len(body) > request.content_length:
+        raise wz_exceptions.BadRequest('Larger body than Content-Length header')
+
+    # Validate the request
+    mac = hmac.new(hmac_secret.encode(), body, hashlib.sha256)
+    req_hmac = request.headers.get('X-Webhook-HMAC', '')
+    our_hmac = mac.hexdigest()
+    if not hmac.compare_digest(req_hmac, our_hmac):
+        log.info('request from %s to %s had bad HMAC %r, expected %r',
+                 request.remote_addr, request, req_hmac, our_hmac)
+        raise wz_exceptions.BadRequest('Bad HMAC')
+
+    try:
+        return json.loads(body)
+    except json.JSONDecodeError as ex:
+        log.warning('request from %s to %s had bad JSON: %s',
+                    request.remote_addr, request, ex)
+        raise wz_exceptions.BadRequest('Bad JSON')
+
+
+def score(wh_payload: dict, user: dict) -> int:
+    """Determine how likely it is that this is the correct user to modify.
+
+    :param wh_payload: the info we received from Blender ID;
+        see user_modified()
+    :param user: the user in our database
+    :return: the score for this user
+    """
+
+    bid_str = str(wh_payload['id'])
+    try:
+        match_on_bid = any((auth['provider'] == 'blender-id' and auth['user_id'] == bid_str)
+                           for auth in user['auth'])
+    except KeyError:
+        match_on_bid = False
+
+    match_on_old_email = user.get('email', 'none') == wh_payload.get('old_email', 'nothere')
+    match_on_new_email = user.get('email', 'none') == wh_payload.get('email', 'nothere')
+    return match_on_bid * 10 + match_on_old_email + match_on_new_email * 2
+
+
+def insert_or_fetch_user(wh_payload: dict) -> typing.Optional[dict]:
+    """Fetch the user from the DB or create it.
+
+    Only creates it if the webhook payload indicates they could actually use
+    Blender Cloud (i.e. demo or subscriber). This prevents us from creating
+    Cloud accounts for Blender Network users.
+
+    :returns the user document, or None when not created.
+    """
+
+    users_coll = current_app.db('users')
+    my_log = log.getChild('insert_or_fetch_user')
+
+    bid_str = str(wh_payload['id'])
+    email = wh_payload['email']
+
+    # Find the user by their Blender ID, or any of their email addresses.
+    # We use one query to find all matching users. This is done as a
+    # consistency check; if more than one user is returned, we know the
+    # database is inconsistent with Blender ID and can emit a warning
+    # about this.
+    query = {'$or': [
+        {'auth.provider': 'blender-id', 'auth.user_id': bid_str},
+        {'email': {'$in': [wh_payload['old_email'], email]}},
+    ]}
+    db_users = list(users_coll.find(query))
+    user_count = len(db_users)
+    if user_count > 1:
+        # Now we have to pay the price for finding users in one query; we
+        # have to prioritise them and return the one we think is most reliable.
+        calc_score = functools.partial(score, wh_payload)
+        best_score = max(db_users, key=calc_score)
+
+        my_log.error('%d users found for query %s, picking user %s (%s)',
+                     user_count, query, best_score['_id'], best_score['email'])
+        return best_score
+    if user_count:
+        db_user = db_users[0]
+        my_log.debug('found user %s', db_user['email'])
+        return db_user
+
+    if wh_payload.get('date_deletion_requested'):
+        my_log.info('Received update for a deleted user %s, not creating', bid_str)
+        return None
+
+    # Pretend to create the user, so that we can inspect the resulting
+    # capabilities. This is more future-proof than looking at the list
+    # of roles in the webhook payload.
+    username = make_unique_username(email)
+    user_doc = create_new_user_document(email, bid_str, username,
+                                        provider='blender-id',
+                                        full_name=wh_payload['full_name'])
+
+    # Figure out the user's eventual roles. These aren't stored in the document yet,
+    # because that's handled by the badger service.
+    eventual_roles = [subscription.ROLES_BID_TO_PILLAR[r]
+                      for r in wh_payload.get('roles', [])
+                      if r in subscription.ROLES_BID_TO_PILLAR]
+    user_ob = UserClass.construct('', user_doc)
+    user_ob.roles = eventual_roles
+    user_ob.collect_capabilities()
+    create = (user_ob.has_cap('subscriber') or
+              user_ob.has_cap('can-renew-subscription') or
+              current_app.org_manager.user_is_unknown_member(email))
+    if not create:
+        my_log.info('Received update for unknown user %r without Cloud access (caps=%s)',
+                    wh_payload['old_email'], user_ob.capabilities)
+        return None
+
+    # Actually create the user in the database.
+    r, _, _, status = current_app.post_internal('users', user_doc)
+    if status != 201:
+        my_log.error('unable to create user %s: : %r %r', email, status, r)
+        raise wz_exceptions.InternalServerError('unable to create user')
+
+    user_doc.update(r)
+    my_log.info('created user %r = %s to allow immediate Cloud access', email, user_doc['_id'])
+    return user_doc
+
+
+@blueprint.route('/user-modified', methods=['POST'])
+def user_modified():
+    """Update the local user based on the info from Blender ID.
+
+    If the payload indicates the user has access to Blender Cloud (or at least
+    a renewable subscription), create the user if not already in our DB.
+
+    The payload we expect is a dictionary like:
+    {'id': 12345,  # the user's ID in Blender ID
+     'old_email': 'old@example.com',
+     'full_name': 'Harry',
+     'email': 'new@example'com,
+     'avatar_changed': True,
+     'roles': ['role1', 'role2', …]}
+    """
+    my_log = log.getChild('user_modified')
+    my_log.debug('Received request from %s', request.remote_addr)
+
+    hmac_secret = current_app.config['BLENDER_ID_WEBHOOK_USER_CHANGED_SECRET']
+    payload = webhook_payload(hmac_secret)
+
+    my_log.info('payload: %s', payload)
+
+    # Update the user
+    db_user = insert_or_fetch_user(payload)
+    if not db_user:
+        my_log.info('Received update for unknown user %r', payload['old_email'])
+        return '', 204
+
+    if payload.get('date_deletion_requested'):
+        delete_user(db_user, payload)
+        return '', 204
+
+    # Use direct database updates to change the email and full name.
+    # Also updates the db_user dict so that local_user below will have
+    # the updated information.
+    updates = {}
+    if db_user['email'] != payload['email']:
+        my_log.info('User changed email from %s to %s', payload['old_email'], payload['email'])
+        updates['email'] = payload['email']
+        db_user['email'] = payload['email']
+
+    if db_user['full_name'] != payload['full_name']:
+        my_log.info('User changed full name from %r to %r',
+                    db_user['full_name'], payload['full_name'])
+        if payload['full_name']:
+            updates['full_name'] = payload['full_name']
+        else:
+            # Fall back to the username when the full name was erased.
+            updates['full_name'] = db_user['username']
+        db_user['full_name'] = updates['full_name']
+
+    if payload.get('avatar_changed'):
+        import pillar.celery.avatar
+        my_log.info('User %s changed avatar, scheduling download', db_user['_id'])
+        pillar.celery.avatar.sync_avatar_for_user.delay(str(db_user['_id']))
+
+    if updates:
+        users_coll = current_app.db('users')
+        update_res = users_coll.update_one({'_id': db_user['_id']},
+                                           {'$set': updates})
+        if update_res.matched_count != 1:
+            my_log.error('Unable to find user %s to update, even though '
+                         'we found them by email address %s',
+                         db_user['_id'], payload['old_email'])
+
+    # Defer to Pillar to do the role updates.
+    local_user = UserClass.construct('', db_user)
+    subscription.do_update_subscription(local_user, payload)
+
+    return '', 204
+
+
+def delete_user(db_user, payload):
+    """Handle deletion request coming from BID."""
+    my_log = log.getChild('delete_user')
+    date_deletion_requested = payload['date_deletion_requested']
+    bid_str = str(payload['id'])
+    local_id = db_user['_id']
+    my_log.info(
+        'User %s with BID=%s requested deletion on %s, soft-deleting the user',
+        local_id, bid_str, date_deletion_requested,
+    )
+    # Delete all session tokens linked to this user
+    token_coll = current_app.db('tokens')
+    delete_res = token_coll.delete_many({'user': local_id})
+    my_log.info('Deleted %s session tokens of user %s', delete_res.deleted_count, local_id)
+
+    # Soft-delete the user and clear their PII
+    users_coll = current_app.db('users')
+    updates = {
+        '_deleted': True,
+        'email': None,
+        'full_name': None,
+        'username': None,
+        'auth': [],
+    }
+    update_res = users_coll.update_one({'_id': local_id}, {'$set': updates})
+    if update_res.matched_count != 1:
+        my_log.error(
+            'Soft-deleted %s users %s with BID=%s',
+            update_res.matched_count, local_id, bid_str,
+        )
+    else:
+        my_log.warning('Soft-deleted user %s with BID=%s', local_id, bid_str)

cloud_share_img.py

@@ -0,0 +1,296 @@
+#!/usr/bin/env python3
+
+from __future__ import print_function
+
+"""CLI command for sharing an image via Blender Cloud.
+
+Assumes that you are logged in on Blender ID with the Blender ID Add-on.
+
+The user_config_dir and user_data_dir functions come from
+ https://github.com/ActiveState/appdirs/blob/master/appdirs.py and
+ are licensed under the MIT license.
+"""
+
+import argparse
+import json
+import mimetypes
+import os.path
+import pprint
+import sys
+import webbrowser
+
+from urllib.parse import urljoin
+
+import requests
+
+cli = argparse.Namespace()  # CLI args from argparser
+sess = requests.Session()
+IMAGE_SHARING_GROUP_NODE_NAME = 'Image sharing'
+
+if sys.platform.startswith('java'):
+    import platform
+
+    os_name = platform.java_ver()[3][0]
+    if os_name.startswith('Windows'):  # "Windows XP", "Windows 7", etc.
+        system = 'win32'
+    elif os_name.startswith('Mac'):  # "Mac OS X", etc.
+        system = 'darwin'
+    else:  # "Linux", "SunOS", "FreeBSD", etc.
+        # Setting this to "linux2" is not ideal, but only Windows or Mac
+        # are actually checked for and the rest of the module expects
+        # *sys.platform* style strings.
+        system = 'linux2'
+else:
+    system = sys.platform
+
+
+def request(method: str, rel_url: str, **kwargs) -> requests.Response:
+    kwargs.setdefault('auth', (cli.token, ''))
+    url = urljoin(cli.server_url, rel_url)
+    return sess.request(method, url, **kwargs)
+
+
+def get(rel_url: str, **kwargs) -> requests.Response:
+    return request('GET', rel_url, **kwargs)
+
+
+def post(rel_url: str, **kwargs) -> requests.Response:
+    return request('POST', rel_url, **kwargs)
+
+
+def find_user_id() -> str:
+    """Returns the current user ID."""
+
+    print(15 * '=', 'User info', 15 * '=')
+    resp = get('/api/users/me')
+    resp.raise_for_status()
+
+    user_info = resp.json()
+    print('You are logged in as %(full_name)s (%(_id)s)' % user_info)
+
+    return user_info['_id']
+
+
+def find_home_project_id() -> dict:
+    resp = get('/api/bcloud/home-project')
+    resp.raise_for_status()
+
+    proj = resp.json()
+    proj_id = proj['_id']
+    print('Your home project ID is %s' % proj_id)
+    return proj_id
+
+
+def find_image_sharing_group_id(home_project_id, user_id) -> str:
+    """Find the top-level image sharing group node."""
+
+    node_doc = {
+        'project': home_project_id,
+        'node_type': 'group',
+        'name': IMAGE_SHARING_GROUP_NODE_NAME,
+        'user': user_id,
+    }
+
+    resp = get('/api/nodes', params={'where': json.dumps(node_doc)})
+    resp.raise_for_status()
+    items = resp.json()['_items']
+
+    if not items:
+        print('Share group not found, creating one.')
+        node_doc.update({
+            'properties': {},
+        })
+        resp = post('/api/nodes', json=node_doc)
+        resp.raise_for_status()
+        share_group = resp.json()
+    else:
+        share_group = items[0]
+
+    # print('Share group:', share_group)
+    return share_group['_id']
+
+
+def upload_image():
+    user_id = find_user_id()
+    home_project_id = find_home_project_id()
+    group_id = find_image_sharing_group_id(home_project_id, user_id)
+    basename = os.path.basename(cli.imgfile)
+    print('Sharing group ID is %s' % group_id)
+
+    # Upload the image to the project.
+    print('Uploading %r' % cli.imgfile)
+    mimetype, _ = mimetypes.guess_type(cli.imgfile, strict=False)
+    with open(cli.imgfile, mode='rb') as infile:
+        resp = post('api/storage/stream/%s' % home_project_id,
+                    files={'file': (basename, infile, mimetype)})
+    resp.raise_for_status()
+    file_upload_resp = resp.json()
+    file_upload_status = file_upload_resp.get('_status') or file_upload_resp.get('status')
+    if file_upload_status != 'ok':
+        raise ValueError('Received bad status %s from Pillar: %s' %
+                         (file_upload_status, json.dumps(file_upload_resp)))
+    file_id = file_upload_resp['file_id']
+    print('File ID is', file_id)
+
+    # Create the asset node
+    asset_node = {
+        'project': home_project_id,
+        'node_type': 'asset',
+        'name': basename,
+        'parent': group_id,
+        'properties': {
+            'content_type': mimetype,
+            'file': file_id,
+        },
+    }
+    resp = post('api/nodes', json=asset_node)
+    resp.raise_for_status()
+    node_info = resp.json()
+    node_id = node_info['_id']
+    print('Created asset node', node_id)
+
+    # Share the node to get a public URL.
+    resp = post('api/nodes/%s/share' % node_id)
+    resp.raise_for_status()
+    share_info = resp.json()
+    print(json.dumps(share_info, indent=4))
+
+    url = share_info.get('short_link')
+    print('Opening %s in a browser' % url)
+    webbrowser.open_new_tab(url)
+
+
+def find_credentials():
+    """Finds BlenderID credentials.
+
+    :rtype: str
+    :returns: the authentication token to use.
+    """
+    import glob
+
+    # Find BlenderID profile file.
+    configpath = user_config_dir('blender', 'Blender Foundation', roaming=True)
+    found = glob.glob(os.path.join(configpath, '*'))
+    for confpath in reversed(sorted(found)):
+        profiles_path = os.path.join(confpath, 'config', 'blender_id', 'profiles.json')
+        if not os.path.exists(profiles_path):
+            continue
+
+        print('Reading credentials from %s' % profiles_path)
+        with open(profiles_path) as infile:
+            profiles = json.load(infile, encoding='utf8')
+        if profiles:
+            break
+    else:
+        print('Unable to find Blender ID credentials. Log in with the Blender ID add-on in '
+              'Blender first.')
+        raise SystemExit()
+
+    active_profile = profiles[u'active_profile']
+    profile = profiles[u'profiles'][active_profile]
+    print('Logging in as %s' % profile[u'username'])
+
+    return profile[u'token']
+
+
+def main():
+    global cli
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument('imgfile', help='The image file to share.')
+    parser.add_argument('-u', '--server-url', default='https://cloud.blender.org/',
+                        help='URL of the Flamenco server.')
+    parser.add_argument('-t', '--token',
+                        help='Authentication token to use. If not given, your token from the '
+                             'Blender ID add-on is used.')
+
+    cli = parser.parse_args()
+    if not cli.token:
+        cli.token = find_credentials()
+
+    upload_image()
+
+
+def user_config_dir(appname=None, appauthor=None, version=None, roaming=False):
+    r"""Return full path to the user-specific config dir for this application.
+        "appname" is the name of application.
+            If None, just the system directory is returned.
+        "appauthor" (only used on Windows) is the name of the
+            appauthor or distributing body for this application. Typically
+            it is the owning company name. This falls back to appname. You may
+            pass False to disable it.
+        "version" is an optional version path element to append to the
+            path. You might want to use this if you want multiple versions
+            of your app to be able to run independently. If used, this
+            would typically be "<major>.<minor>".
+            Only applied when appname is present.
+        "roaming" (boolean, default False) can be set True to use the Windows
+            roaming appdata directory. That means that for users on a Windows
+            network setup for roaming profiles, this user data will be
+            sync'd on login. See
+            <http://technet.microsoft.com/en-us/library/cc766489(WS.10).aspx>
+            for a discussion of issues.
+    Typical user config directories are:
+        Mac OS X:               same as user_data_dir
+        Unix:                   ~/.config/<AppName>     # or in $XDG_CONFIG_HOME, if defined
+        Win *:                  same as user_data_dir
+    For Unix, we follow the XDG spec and support $XDG_CONFIG_HOME.
+    That means, by default "~/.config/<AppName>".
+    """
+    if system in {"win32", "darwin"}:
+        path = user_data_dir(appname, appauthor, None, roaming)
+    else:
+        path = os.getenv('XDG_CONFIG_HOME', os.path.expanduser("~/.config"))
+        if appname:
+            path = os.path.join(path, appname)
+    if appname and version:
+        path = os.path.join(path, version)
+    return path
+
+
+def user_data_dir(appname=None, appauthor=None, version=None, roaming=False):
+    r"""Return full path to the user-specific data dir for this application.
+        "appname" is the name of application.
+            If None, just the system directory is returned.
+        "appauthor" (only used on Windows) is the name of the
+            appauthor or distributing body for this application. Typically
+            it is the owning company name. This falls back to appname. You may
+            pass False to disable it.
+        "version" is an optional version path element to append to the
+            path. You might want to use this if you want multiple versions
+            of your app to be able to run independently. If used, this
+            would typically be "<major>.<minor>".
+            Only applied when appname is present.
+        "roaming" (boolean, default False) can be set True to use the Windows
+            roaming appdata directory. That means that for users on a Windows
+            network setup for roaming profiles, this user data will be
+            sync'd on login. See
+            <http://technet.microsoft.com/en-us/library/cc766489(WS.10).aspx>
+            for a discussion of issues.
+    Typical user data directories are:
+        Mac OS X:               ~/Library/Application Support/<AppName>
+        Unix:                   ~/.local/share/<AppName>    # or in $XDG_DATA_HOME, if defined
+        Win XP (not roaming):   C:\Documents and Settings\<username>\Application Data\<AppAuthor>\<AppName>
+        Win XP (roaming):       C:\Documents and Settings\<username>\Local Settings\Application Data\<AppAuthor>\<AppName>
+        Win 7  (not roaming):   C:\Users\<username>\AppData\Local\<AppAuthor>\<AppName>
+        Win 7  (roaming):       C:\Users\<username>\AppData\Roaming\<AppAuthor>\<AppName>
+    For Unix, we follow the XDG spec and support $XDG_DATA_HOME.
+    That means, by default "~/.local/share/<AppName>".
+    """
+    if system == "win32":
+        raise RuntimeError("Sorry, Windows is not supported for now.")
+    elif system == 'darwin':
+        path = os.path.expanduser('~/Library/Application Support/')
+        if appname:
+            path = os.path.join(path, appname)
+    else:
+        path = os.getenv('XDG_DATA_HOME', os.path.expanduser("~/.local/share"))
+        if appname:
+            path = os.path.join(path, appname)
+    if appname and version:
+        path = os.path.join(path, version)
+    return path
+
+
+if __name__ == '__main__':
+    main()

config_local.example.py

@@ -0,0 +1,43 @@
+import os
+
+DEBUG = True
+
+BLENDER_ID_ENDPOINT = 'http://id.local:8000/'
+
+SERVER_NAME = 'cloud.local:5001'
+SCHEME = 'http'
+PILLAR_SERVER_ENDPOINT = f'{SCHEME}://{SERVER_NAME}/api/'
+
+os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = 'true'
+os.environ['PILLAR_MONGO_DBNAME'] = 'cloud'
+os.environ['PILLAR_MONGO_PORT'] = '27017'
+os.environ['PILLAR_MONGO_HOST'] = 'mongo'
+
+os.environ['PILLAR_SERVER_ENDPOINT'] = PILLAR_SERVER_ENDPOINT
+
+SECRET_KEY = '##DEFINE##'
+
+OAUTH_CREDENTIALS = {
+    'blender-id': {
+        'id': 'CLOUD-OF-SNOWFLAKES-42',
+        'secret': '##DEFINE##',
+    }
+}
+
+MAIN_PROJECT_ID = '##DEFINE##'
+URLER_SERVICE_AUTH_TOKEN = '##DEFINE##'
+
+ZENCODER_API_KEY = '##DEFINE##'
+ZENCODER_NOTIFICATIONS_SECRET = '##DEFINE##'
+ZENCODER_NOTIFICATIONS_URL = 'http://zencoderfetcher/'
+
+# Special announcement on top of every page, for non-subscribers.
+# category: 'string', can be 'info', 'warning', 'danger', or 'success'.
+# message: 'string', any text, it gets markdowned.
+# icon: 'string', any icon in font-pillar. e.g. 'pi-heart-filled'
+UI_ANNOUNCEMENT_NON_SUBSCRIBERS = {
+    'category': 'danger',
+    'message': 'Spring will swing away the gray clouds, until then, '
+               '[take cover under Blender Cloud](https://cloud.blender.org)!',
+    'icon': 'pi-heart-filled',
+}

deploy.sh

@@ -1,124 +0,0 @@
-#!/bin/bash -e
-
-# Deploys the current production branch to the production machine.
-PROJECT_NAME="blender-cloud"
-DOCKER_NAME="blender_cloud"
-REMOTE_ROOT="/data/git/${PROJECT_NAME}"
-
-SSH="ssh -o ClearAllForwardings=yes cloud.blender.org"
-
-# macOS does not support readlink -f, so we use greadlink instead
-if [[ `uname` == 'Darwin' ]]; then
-    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
-    readlink='greadlink'
-else
-    readlink='readlink'
-fi
-
-ROOT="$(dirname "$($readlink -f "$0")")"
-cd ${ROOT}
-
-# Check that we're on production branch.
-if [ $(git rev-parse --abbrev-ref HEAD) != "production" ]; then
-    echo "You are NOT on the production branch, refusing to deploy." >&2
-    exit 1
-fi
-
-# Check that production branch has been pushed.
-if [ -n "$(git log origin/production..production --oneline)" ]; then
-    echo "WARNING: not all changes to the production branch have been pushed."
-    echo "Press [ENTER] to continue deploying current origin/production, CTRL+C to abort."
-    read dummy
-fi
-
-function find_module()
-{
-    MODULE_NAME=$1
-    MODULE_DIR=$(python <<EOT
-from __future__ import print_function
-import os.path
-try:
-    import ${MODULE_NAME}
-except ImportError:
-    raise SystemExit('${MODULE_NAME} not found on Python path. Are you in the correct venv?')
-
-print(os.path.dirname(os.path.dirname(${MODULE_NAME}.__file__)))
-EOT
-)
-
-    if [ $(git -C $MODULE_DIR rev-parse --abbrev-ref HEAD) != "production" ]; then
-        echo "${MODULE_NAME}: ($MODULE_DIR) NOT on the production branch, refusing to deploy." >&2
-        exit 1
-    fi
-
-    echo $MODULE_DIR
-}
-
-# Find our modules
-PILLAR_DIR=$(find_module pillar)
-ATTRACT_DIR=$(find_module attract)
-FLAMENCO_DIR=$(find_module flamenco)
-
-echo "Pillar  : $PILLAR_DIR"
-echo "Attract : $ATTRACT_DIR"
-echo "Flamenco: $FLAMENCO_DIR"
-
-if [ -z "$PILLAR_DIR" -o -z "$ATTRACT_DIR" -o -z "$FLAMENCO_DIR" ];
-then
-    exit 1
-fi
-
-# SSH to cloud to pull all files in
-function git_pull() {
-    PROJECT_NAME="$1"
-    BRANCH="$2"
-    REMOTE_ROOT="/data/git/${PROJECT_NAME}"
-
-    echo "==================================================================="
-    echo "UPDATING FILES ON ${PROJECT_NAME}"
-    ${SSH} git -C ${REMOTE_ROOT} fetch origin ${BRANCH}
-    ${SSH} git -C ${REMOTE_ROOT} log origin/${BRANCH}..${BRANCH} --oneline
-    ${SSH} git -C ${REMOTE_ROOT} merge --ff-only origin/${BRANCH}
-}
-
-git_pull pillar-python-sdk master
-git_pull pillar production
-git_pull attract production
-git_pull flamenco production
-git_pull blender-cloud production
-
-# Update the virtualenv
-#${SSH} -t docker exec ${DOCKER_NAME} /data/venv/bin/pip install -U -r ${REMOTE_ROOT}/requirements.txt --exists-action w
-
-# RSync the world
-$ATTRACT_DIR/rsync_ui.sh
-$FLAMENCO_DIR/rsync_ui.sh
-./rsync_ui.sh
-
-# Notify Bugsnag of this new deploy.
-echo
-echo "==================================================================="
-GIT_REVISION=$(${SSH} git -C ${REMOTE_ROOT} describe --always)
-echo "Notifying Bugsnag of this new deploy of revision ${GIT_REVISION}."
-BUGSNAG_API_KEY=$(${SSH} python -c "\"import sys; sys.path.append('${REMOTE_ROOT}'); import config_local; print(config_local.BUGSNAG_API_KEY)\"")
-curl --data "apiKey=${BUGSNAG_API_KEY}&revision=${GIT_REVISION}" https://notify.bugsnag.com/deploy
-echo
-
-# Wait for [ENTER] to restart the server
-echo
-echo "==================================================================="
-echo "NOTE: If you want to edit config_local.py on the server, do so now."
-echo "NOTE: Press [ENTER] to continue and restart the server process."
-read dummy
-${SSH} docker exec ${DOCKER_NAME} apache2ctl graceful
-echo "Server process restarted"
-
-echo
-echo "==================================================================="
-echo "Clearing front page from Redis cache."
-${SSH} docker exec redis redis-cli DEL pwview//
-
-echo
-echo "==================================================================="
-echo "Deploy of ${PROJECT_NAME} is done."
-echo "==================================================================="

deploy/2docker.sh

@@ -0,0 +1,130 @@
+#!/bin/bash -e
+
+STAGING_BRANCH=${STAGING_BRANCH:-production}
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [[ `uname` == 'Darwin' ]]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+
+ROOT="$(dirname "$(dirname "$($readlink -f "$0")")")"
+STAGINGDIR="$ROOT/docker/4_run/staging"
+PROJECT_NAME="$(basename $ROOT)"
+
+if [ -e $STAGINGDIR ]; then
+    echo "$STAGINGDIR already exists, press [ENTER] to destroy and re-install, Ctrl+C to abort."
+    read dummy
+    rm -rf $STAGINGDIR
+else
+    echo -n "Installing into $STAGINGDIR… "
+    echo "press [ENTER] to continue, Ctrl+C to abort."
+    read dummy
+fi
+
+cd ${ROOT}
+mkdir -p $STAGINGDIR
+REMOTE_ROOT="$STAGINGDIR/$PROJECT_NAME"
+
+if [ -z "$SKIP_BRANCH_CHECK" ]; then
+    # Check that we're on production branch.
+    if [ $(git rev-parse --abbrev-ref HEAD) != "$STAGING_BRANCH" ]; then
+        echo "You are NOT on the $STAGING_BRANCH branch, refusing to stage." >&2
+        exit 1
+    fi
+
+    # Check that production branch has been pushed.
+    if [ -n "$(git log origin/$STAGING_BRANCH..$STAGING_BRANCH --oneline)" ]; then
+        echo "WARNING: not all changes to the $STAGING_BRANCH branch have been pushed."
+        echo "Press [ENTER] to continue staging current origin/$STAGING_BRANCH, CTRL+C to abort."
+        read dummy
+    fi
+fi
+
+function find_module()
+{
+    MODULE_NAME=$1
+    MODULE_DIR=$(python <<EOT
+from __future__ import print_function
+import os.path
+try:
+    import ${MODULE_NAME}
+except ImportError:
+    raise SystemExit('${MODULE_NAME} not found on Python path. Are you in the correct venv?')
+
+print(os.path.dirname(os.path.dirname(${MODULE_NAME}.__file__)))
+EOT
+)
+    echo $MODULE_DIR
+}
+
+# Find our modules
+echo "==================================================================="
+echo "LOCAL MODULE LOCATIONS"
+PILLAR_DIR=$(find_module pillar)
+ATTRACT_DIR=$(find_module attract)
+FLAMENCO_DIR=$(find_module flamenco)
+SVNMAN_DIR=$(find_module svnman)
+SDK_DIR=$(find_module pillarsdk)
+
+echo "Pillar  : $PILLAR_DIR"
+echo "Attract : $ATTRACT_DIR"
+echo "Flamenco: $FLAMENCO_DIR"
+echo "SVNMan  : $SVNMAN_DIR"
+echo "SDK     : $SDK_DIR"
+
+if [ -z "$PILLAR_DIR" -o -z "$ATTRACT_DIR" -o -z "$FLAMENCO_DIR" -o -z "$SVNMAN_DIR" -o -z "$SDK_DIR" ];
+then
+    exit 1
+fi
+
+function git_clone() {
+    PROJECT_NAME="$1"
+    BRANCH="$2"
+    LOCAL_ROOT="$3"
+
+    echo "==================================================================="
+    echo "CLONING REPO ON $PROJECT_NAME @$BRANCH"
+    URL=$(git -C $LOCAL_ROOT remote get-url origin)
+    git -C $STAGINGDIR clone --depth 1 --branch $BRANCH $URL $PROJECT_NAME
+}
+
+if [ "$STAGING_BRANCH" == "production" ]; then
+    SDK_STAGING_BRANCH=master  # SDK doesn't have a production branch
+else
+    SDK_STAGING_BRANCH=$STAGING_BRANCH
+fi
+
+git_clone pillar-python-sdk $SDK_STAGING_BRANCH $SDK_DIR
+git_clone pillar $STAGING_BRANCH $PILLAR_DIR
+git_clone attract $STAGING_BRANCH $ATTRACT_DIR
+git_clone flamenco $STAGING_BRANCH $FLAMENCO_DIR
+git_clone pillar-svnman $STAGING_BRANCH $SVNMAN_DIR
+git_clone blender-cloud $STAGING_BRANCH $ROOT
+
+# Gulp everywhere
+GULP=$ROOT/node_modules/.bin/gulp
+if [ ! -e $GULP -o gulpfile.js -nt $GULP ]; then
+    npm install
+    touch $GULP  # installer doesn't always touch this after a build, so we do.
+fi
+
+# List of projects
+PROJECTS="pillar attract flamenco pillar-svnman blender-cloud"
+
+# Run ./gulp for every project
+for PROJECT in $PROJECTS; do
+    pushd $STAGINGDIR/$PROJECT; ./gulp --production; popd;
+done
+
+# Remove node_modules (only after all projects with interdependencies have been built)
+for PROJECT in $PROJECTS; do
+    pushd $STAGINGDIR/$PROJECT; rm -r node_modules; popd;
+done
+
+echo
+echo "==================================================================="
+echo "Staging of ${PROJECT_NAME} is ready for dockerisation."
+echo "==================================================================="

deploy/2server.sh

@@ -0,0 +1,80 @@
+#!/bin/bash -e
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [[ `uname` == 'Darwin' ]]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+ROOT="$(dirname "$(dirname "$($readlink -f "$0")")")"
+PROJECT_NAME="$(basename $ROOT)"
+DOCKER_IMAGE="armadillica/blender_cloud:latest"
+REMOTE_SECRET_CONFIG_DIR="/data/config"
+REMOTE_DOCKER_COMPOSE_DIR="/root/docker"
+
+#################################################################################
+case $1 in
+    cloud*)
+        DEPLOYHOST="$1"
+        ;;
+    *)
+        echo "Use $0 cloud{nr}|cloud.blender.org" >&2
+        exit 1
+esac
+SSH_OPTS="-o ClearAllForwardings=yes -o PermitLocalCommand=no"
+SSH="ssh $SSH_OPTS $DEPLOYHOST"
+SCP="scp $SSH_OPTS"
+
+echo -n "Deploying to $DEPLOYHOST… "
+
+if ! ping $DEPLOYHOST -q -c 1 -W 2 >/dev/null; then
+    echo "host $DEPLOYHOST cannot be pinged, refusing to deploy." >&2
+    exit 2
+fi
+
+cat <<EOT
+[ping OK]
+
+Make sure that you have pushed the $DOCKER_IMAGE
+docker image to Docker Hub.
+
+press [ENTER] to continue, Ctrl+C to abort.
+EOT
+read dummy
+
+#################################################################################
+echo "==================================================================="
+echo "Bringing remote Docker up to date…"
+$SSH mkdir -p $REMOTE_DOCKER_COMPOSE_DIR
+$SCP \
+    $ROOT/docker/{docker-compose.yml,renew-letsencrypt.sh,mongo-backup.{cron,sh}} \
+    $DEPLOYHOST:$REMOTE_DOCKER_COMPOSE_DIR
+$SSH -T <<EOT
+set -e
+cd $REMOTE_DOCKER_COMPOSE_DIR
+docker pull $DOCKER_IMAGE
+docker-compose up -d
+
+echo
+echo "==================================================================="
+echo "Clearing front page from Redis cache."
+docker exec redis redis-cli DEL pwview//
+EOT
+
+# Notify Sentry of this new deploy.
+# See https://sentry.io/blender-institute/blender-cloud/settings/release-tracking/
+# and https://docs.sentry.io/api/releases/post-organization-releases/
+# and https://sentry.io/api/
+echo
+echo "==================================================================="
+REVISION=$(date +'%Y%m%d.%H%M%S.%Z')
+echo "Notifying Sentry of this new deploy of revision $REVISION."
+SENTRY_RELEASE_URL="$($SSH env PYTHONPATH="$REMOTE_SECRET_CONFIG_DIR" python3 -c "\"import config_secrets; print(config_secrets.SENTRY_RELEASE_URL)\"")"
+curl -s "$SENTRY_RELEASE_URL" -XPOST -H 'Content-Type: application/json' -d "{\"version\": \"$REVISION\"}" | json_pp
+echo
+
+echo
+echo "==================================================================="
+echo "Deploy to $DEPLOYHOST done."
+echo "==================================================================="

deploy/README.md

@@ -0,0 +1,39 @@
+# Deploying to Production
+
+```
+workon blender-cloud  # activate your virtualenv
+cd $projectdir/deploy
+./full-pull.sh
+```
+
+## The Details
+
+Deployment consists of a few steps:
+
+1. Populate a staging directory with the files from the production branches of the various projects.
+2. Create Docker images.
+3. Push the docker images to Docker Hub.
+4. Pull the docker images on the production server and rebuild+restart the containers.
+
+The scripts involved are:
+
+- `2docker.sh`: performs step 1. above.
+- `build-{xxx}.sh`: performs steps 2. and 3. above.
+- `2server.sh`: performs step 4. above.
+
+The `full-{xxx}.sh` scripts perform all the steps, and call into `build-{xxx}.sh`.
+
+For `xxx` there are:
+
+- `all`: Rebuild all Docker images from scratch. This is good for getting the latest updates to the
+  base image.
+- `pull`: Pull the base and intermediate images from Docker Hub so that they are the same as the
+  last time someone pushed to production, then rebuilds the final Docker image.
+- `quick`: Just rebuild the final Docker image. Only use this if the last time a deployment to
+  the production server was done was by you, on the machine you're working on now.
+
+
+## Hacking Stuff
+
+To deploy another branch than `production`, do `export STAGING_BRANCH=otherbranch` before starting
+the above commands.

deploy/build-all.sh

@@ -0,0 +1,43 @@
+#!/bin/bash -e
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [[ `uname` == 'Darwin' ]]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+ROOT="$(dirname "$(dirname "$($readlink -f "$0")")")"
+
+case "$(basename "$0")" in
+    build-pull.sh)
+        docker pull armadillica/pillar_py:3.6
+        docker pull armadillica/pillar_wheelbuilder:latest
+        pushd "$ROOT/docker/3_buildwheels"
+        ./build.sh
+        popd
+        pushd "$ROOT/docker/4_run"
+        ./build.sh
+        ;;
+    build-quick.sh)
+        pushd "$ROOT/docker/4_run"
+        ./build.sh
+        ;;
+    build-all.sh)
+        pushd "$ROOT/docker"
+        ./full_rebuild.sh
+        ;;
+    *)
+        echo "Unknown script $0, aborting" >&2
+        exit 1
+esac
+
+popd
+echo
+echo "Press [ENTER] to push the new Docker images."
+read dummy
+docker push armadillica/pillar_py:3.6
+docker push armadillica/pillar_wheelbuilder:latest
+docker push armadillica/blender_cloud:latest
+echo
+echo "Build is done, ready to update the server."

deploy/build-pull.sh

@@ -0,0 +1 @@
+build-quick.sh

deploy/build-quick.sh

@@ -0,0 +1 @@
+build-all.sh

deploy/full-all.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+set -e
+
+NAME="$(basename "$0")"
+
+./2docker.sh
+./${NAME/full-/build-}
+./2server.sh cloud2

deploy/full-pull.sh

@@ -0,0 +1 @@
+full-all.sh

deploy/full-quick.sh

@@ -0,0 +1 @@
+full-all.sh

docker/1_base/Dockerfile

@@ -0,0 +1,10 @@
+FROM ubuntu:18.04
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+RUN set -ex; \
+    apt-get update; \
+    DEBIAN_FRONTEND=noninteractive apt-get install \
+      -qyy -o APT::Install-Recommends=false -o APT::Install-Suggests=false \
+      tzdata openssl ca-certificates locales; \
+    locale-gen en_US.UTF-8 en_GB.UTF-8 nl_NL.UTF-8
+ENV LANG en_US.UTF-8

docker/1_base/base.docker

@@ -1,16 +0,0 @@
-FROM ubuntu:16.04
-MAINTAINER Francesco Siddi <francesco@blender.org>
-
-RUN apt-get update && apt-get install -qyy \
-    -o APT::Install-Recommends=false -o APT::Install-Suggests=false \
-    python-pip libffi6 openssl ffmpeg rsyslog logrotate
-
-RUN mkdir -p /data/git/pillar \
-&& mkdir -p /data/storage \
-&& mkdir -p /data/config \
-&& mkdir -p /data/venv \
-&& mkdir -p /data/wheelhouse
-
-RUN pip install virtualenv
-RUN virtualenv /data/venv
-RUN . /data/venv/bin/activate && pip install -U pip && pip install wheel

docker/1_base/build.sh

@@ -1,3 +1,4 @@
 #!/usr/bin/env bash
 
-docker build -t pillar_base -f base.docker .;
+# Uses --no-cache to always get the latest upstream (security) upgrades.
+exec docker build --no-cache "$@" -t pillar_base .

docker/2_build/build-wheels.sh

@@ -1,3 +0,0 @@
-#!/usr/bin/env bash
-
-. /data/venv/bin/activate && pip wheel --wheel-dir=/data/wheelhouse -r /requirements.txt

docker/2_build/build.docker

@@ -1,26 +0,0 @@
-FROM pillar_base
-MAINTAINER Francesco Siddi <francesco@blender.org>
-
-RUN apt-get update && apt-get install -qy \
-git \
-gcc \
-libffi-dev \
-libssl-dev \
-pypy-dev \
-python-dev \
-python-imaging \
-zlib1g-dev \
-libjpeg-dev \
-libtiff-dev \
-python-crypto \
-python-openssl
-
-ENV WHEELHOUSE=/data/wheelhouse
-ENV PIP_WHEEL_DIR=/data/wheelhouse
-ENV PIP_FIND_LINKS=/data/wheelhouse
-
-VOLUME /data/wheelhouse
-
-ADD requirements.txt /requirements.txt
-ADD build-wheels.sh /build-wheels.sh
-ENTRYPOINT ["bash", "build-wheels.sh"]

docker/2_build/build.sh

@@ -1,11 +0,0 @@
-#!/usr/bin/env bash
-
-mkdir -p ../3_run/wheelhouse;
-cp ../../requirements.txt .;
-
-docker build -t pillar_build -f build.docker .;
-docker run --rm \
-       -v "$(pwd)"/../3_run/wheelhouse:/data/wheelhouse \
-       pillar_build;
-
-rm requirements.txt;

docker/2_buildpy/Python-3.6.6.tar.xz.md5

@@ -0,0 +1 @@
+c3f30a0aff425dda77d19e02f420d6ba  Python-3.6.6.tar.xz

docker/2_buildpy/build.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+
+set -e
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [ $(uname) == 'Darwin' ]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+
+PYTHONTARGET=$($readlink -f ./python)
+
+mkdir -p "$PYTHONTARGET"
+echo "Python will be built to $PYTHONTARGET"
+
+docker build -t pillar_build -f buildpy.docker .
+
+# Use the docker image to build Python 3.6 and mod-wsgi
+GID=$(id -g)
+docker run --rm -i \
+    -v "$PYTHONTARGET:/opt/python" \
+    pillar_build <<EOT
+set -e
+cd \$PYTHONSOURCE
+./configure \
+    --prefix=/opt/python \
+    --enable-ipv6 \
+    --enable-shared \
+    --with-ensurepip=upgrade
+make -j8 install
+
+# Make sure we can run Python
+ldconfig
+
+# Upgrade pip
+/opt/python/bin/python3 -m pip install -U pip
+
+# Build mod-wsgi-py3 for Python 3.6
+cd /dpkg/mod-wsgi-*
+./configure --with-python=/opt/python/bin/python3
+make -j8 install
+mkdir -p /opt/python/mod-wsgi
+cp /usr/lib/apache2/modules/mod_wsgi.so /opt/python/mod-wsgi
+
+chown -R $UID:$GID /opt/python/*
+EOT
+
+# Strip some stuff we don't need from the Python install.
+rm -rf $PYTHONTARGET/lib/python3.*/test
+rm -rf $PYTHONTARGET/lib/python3.*/config-3.*/libpython3.*.a
+find $PYTHONTARGET/lib -name '*.so.*' -o -name '*.so' | while read libname; do
+    chmod u+w "$libname"
+    strip "$libname"
+done
+
+# Create another docker image which contains the actual Python.
+# This one will serve as base for the Wheel builder and the
+# production image.
+docker build -t armadillica/pillar_py:3.6 -f includepy.docker .

docker/2_buildpy/buildpy.docker

@@ -0,0 +1,35 @@
+FROM pillar_base
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+RUN sed -i 's/^# deb-src/deb-src/' /etc/apt/sources.list && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -qy \
+        build-essential \
+        apache2-dev \
+        checkinstall \
+        curl
+
+RUN apt-get build-dep -y python3.6
+
+ADD Python-3.6.6.tar.xz.md5 /Python-3.6.6.tar.xz.md5
+
+# Install Python sources
+RUN curl -O https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz && \
+    md5sum -c Python-3.6.6.tar.xz.md5 && \
+    tar xf Python-3.6.6.tar.xz && \
+    rm -v Python-3.6.6.tar.xz
+
+# Install mod-wsgi sources
+RUN mkdir -p /dpkg && cd /dpkg && apt-get source libapache2-mod-wsgi-py3
+
+# To be able to install Python outside the docker.
+VOLUME /opt/python
+
+# To be able to run Python; after building, ldconfig has to be re-run to do this.
+# This makes it easier to use Python right after building (for example to build
+# mod-wsgi for Python 3.6).
+RUN echo /opt/python/lib > /etc/ld.so.conf.d/python.conf
+RUN ldconfig
+ENV PATH=/opt/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
+ENV PYTHONSOURCE=/Python-3.6.6

docker/2_buildpy/includepy.docker

@@ -0,0 +1,13 @@
+FROM pillar_base
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+ADD python /opt/python
+
+RUN echo /opt/python/lib > /etc/ld.so.conf.d/python.conf
+RUN ldconfig
+
+RUN echo Python is installed in /opt/python/ > README.python
+ENV PATH=/opt/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
+RUN cd /opt/python/bin && \
+    ln -s python3 python

docker/3_buildwheels/Dockerfile

@@ -0,0 +1,20 @@
+FROM armadillica/pillar_py:3.6
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+RUN set -ex; \
+    apt-get update; \
+    DEBIAN_FRONTEND=noninteractive apt-get install -qy \
+    git \
+    build-essential \
+    checkinstall \
+    libffi-dev \
+    libssl-dev \
+    libjpeg-dev \
+    zlib1g-dev
+
+ENV WHEELHOUSE=/data/wheelhouse
+ENV PIP_WHEEL_DIR=/data/wheelhouse
+ENV PIP_FIND_LINKS=/data/wheelhouse
+RUN mkdir -p $WHEELHOUSE
+
+VOLUME /data/wheelhouse

docker/3_buildwheels/build.sh

@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+
+DOCKER_IMAGE_NAME=armadillica/pillar_wheelbuilder
+
+set -e
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [ $(uname) == 'Darwin' ]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+
+TOPDEVDIR="$($readlink -f ../../..)"
+echo "Top-level development dir is $TOPDEVDIR"
+
+WHEELHOUSE="$($readlink -f ../4_run/wheelhouse)"
+if [ -z "$WHEELHOUSE" ]; then
+    echo "Error, ../4_run might not exist." >&2
+    exit 2
+fi
+
+echo "Wheelhouse is $WHEELHOUSE"
+mkdir -p "$WHEELHOUSE"
+rm -f "$WHEELHOUSE"/*
+
+docker build -t $DOCKER_IMAGE_NAME:latest .
+
+GID=$(id -g)
+docker run --rm -i \
+    -v "$WHEELHOUSE:/data/wheelhouse" \
+    -v "$TOPDEVDIR:/data/topdev" \
+    $DOCKER_IMAGE_NAME <<EOT
+set -e
+set -x
+
+# Globally upgrade Pip, so that we can get a compatible version of the cryptography package.
+# See https://github.com/pyca/cryptography/issues/5771
+pip3 install --upgrade pip setuptools wheel
+
+# Pin poetry to 1.0, as more recent version to not support nested filesystem package
+# dependencies.
+pip3 install wheel poetry==1.0 cryptography==2.7
+
+# Build wheels for all dependencies.
+cd /data/topdev/blender-cloud
+
+poetry install --no-dev
+
+# Apparently pip doesn't like projects without setup.py, so it think we have 'pillar-svnman' as
+# requirement (because that's the name of the directory). We have to grep that out.
+poetry run pip3 freeze | grep -v '\(pillar\)\|\(^-[ef] \)' > \$WHEELHOUSE/requirements.txt
+
+pip3 wheel --wheel-dir=\$WHEELHOUSE -r \$WHEELHOUSE/requirements.txt
+chown -R $UID:$GID \$WHEELHOUSE
+EOT
+
+# Remove our own projects, they shouldn't be installed as wheel (for now).
+rm -f $WHEELHOUSE/{attract,flamenco,pillar,pillarsdk}*.whl
+
+echo "Build of $DOCKER_IMAGE_NAME:latest is done."

docker/3_run/000-default.conf

@@ -1,39 +0,0 @@
-<VirtualHost *:80>
-	# EnableSendfile on
-	XSendFile on
-	XSendFilePath /data/storage/pillar
-	XSendFilePath /data/git/pillar
-	XSendFilePath /data/venv/lib/python2.7/site-packages/attract/static/
-	XSendFilePath /data/venv/lib/python2.7/site-packages/flamenco/static/
-	XsendFilePath /data/git/blender-cloud
-
-	ServerAdmin webmaster@localhost
-	DocumentRoot /var/www/html
-
-	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
-	# error, crit, alert, emerg.
-	# It is also possible to configure the loglevel for particular
-	# modules, e.g.
-	# LogLevel info ssl:warn
-
-	ErrorLog ${APACHE_LOG_DIR}/error.log
-	CustomLog ${APACHE_LOG_DIR}/access.log combined
-
-	WSGIDaemonProcess cloud processes=4 threads=1 maximum-requests=10000
-	WSGIPassAuthorization On
-
-	WSGIScriptAlias / /data/git/blender-cloud/runserver.wsgi \
-		process-group=cloud application-group=%{GLOBAL}
-
-	<Directory /data/git/blender-cloud>
-		<Files runserver.wsgi>
-			Require all granted
-		</Files>
-	</Directory>
-
-	# Temporary edit to remap the old cloudapi.blender.org to cloud.blender.org/api
-	RewriteEngine On
-	RewriteCond "%{HTTP_HOST}"   "^cloudapi\.blender\.org" [NC]
-	RewriteRule (.*) /api$1 [PT]
-
-</VirtualHost>

docker/3_run/build.sh

@@ -1,5 +0,0 @@
-#!/usr/bin/env bash
-
-cp ../../requirements.txt .;
-docker build -t armadillica/blender_cloud -f run.docker .;
-rm requirements.txt;

docker/3_run/docker-entrypoint.sh

@@ -1,25 +0,0 @@
-#!/usr/bin/env bash
-
-if [ ! -f /installed ]; then
-    echo "Installing pillar and pillar-sdk"
-    # TODO: curretly doing pip install -e takes a long time, so we symlink
-    # . /data/venv/bin/activate && pip install -e /data/git/pillar
-    ln -s /data/git/pillar/pillar /data/venv/lib/python2.7/site-packages/pillar
-    # . /data/venv/bin/activate && pip install -e /data/git/attract
-    ln -s /data/git/attract/attract /data/venv/lib/python2.7/site-packages/attract
-    # . /data/venv/bin/activate && pip install -e /data/git/flamenco/packages/flamenco
-    ln -s /data/git/flamenco/packages/flamenco/flamenco/ /data/venv/lib/python2.7/site-packages/flamenco
-    # . /data/venv/bin/activate && pip install -e /data/git/pillar-python-sdk
-    ln -s /data/git/pillar-python-sdk/pillarsdk /data/venv/lib/python2.7/site-packages/pillarsdk
-    touch installed
-fi
-
-if [ "$DEV" = "true" ]; then
-    echo "Running in development mode"
-    cd /data/git/blender-cloud
-    bash /manage.sh runserver --host='0.0.0.0'
-else
-    # Run Apache
-    a2enmod rewrite
-    /usr/sbin/apache2ctl -D FOREGROUND
-fi

docker/3_run/manage.sh

@@ -1,5 +0,0 @@
-#!/usr/bin/env bash -e
-
-. /data/venv/bin/activate
-cd /data/git/blender-cloud
-python manage.py "$@"

docker/3_run/run.docker

@@ -1,46 +0,0 @@
-FROM pillar_base
-
-RUN apt-get update && apt-get install -qyy \
--o APT::Install-Recommends=true -o APT::Install-Suggests=false \
-git \
-apache2 \
-libapache2-mod-wsgi \
-libapache2-mod-xsendfile \
-libjpeg8 \
-libtiff5 \
-nano vim curl \
-&& rm -rf /var/lib/apt/lists/*
-
-ENV APACHE_RUN_USER www-data
-ENV APACHE_RUN_GROUP www-data
-ENV APACHE_LOG_DIR /var/log/apache2
-ENV APACHE_PID_FILE /var/run/apache2.pid
-ENV APACHE_RUN_DIR /var/run/apache2
-ENV APACHE_LOCK_DIR /var/lock/apache2
-
-RUN mkdir -p $APACHE_RUN_DIR $APACHE_LOCK_DIR $APACHE_LOG_DIR
-
-ADD requirements.txt /requirements.txt
-ADD wheelhouse /data/wheelhouse
-
-RUN . /data/venv/bin/activate \
-&& pip install --no-index --find-links=/data/wheelhouse -r requirements.txt \
-&& rm /requirements.txt
-
-VOLUME /data/git/blender-cloud
-VOLUME /data/git/pillar
-VOLUME /data/git/pillar-python-sdk
-VOLUME /data/config
-VOLUME /data/storage
-
-ENV USE_X_SENDFILE True
-
-EXPOSE 80
-EXPOSE 5000
-
-ADD apache2.conf /etc/apache2/apache2.conf
-ADD 000-default.conf /etc/apache2/sites-available/000-default.conf
-ADD docker-entrypoint.sh /docker-entrypoint.sh
-ADD manage.sh /manage.sh
-
-ENTRYPOINT ["bash", "/docker-entrypoint.sh"]

docker/4_run/Dockerfile

@@ -0,0 +1,65 @@
+FROM armadillica/pillar_py:3.6
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+RUN set -ex; \
+    apt-get update; \
+    DEBIAN_FRONTEND=noninteractive apt-get install -qy \
+    -o APT::Install-Recommends=false -o APT::Install-Suggests=false \
+        git \
+        apache2 \
+        libapache2-mod-xsendfile \
+        libjpeg8 \
+        libtiff5 \
+        ffmpeg \
+        rsyslog logrotate \
+        nano vim-tiny curl; \
+    rm -rf /var/lib/apt/lists/*
+
+RUN ln -s /usr/bin/vim.tiny /usr/bin/vim
+
+ENV APACHE_RUN_USER www-data
+ENV APACHE_RUN_GROUP www-data
+ENV APACHE_LOG_DIR /var/log/apache2
+ENV APACHE_PID_FILE /var/run/apache2.pid
+ENV APACHE_RUN_DIR /var/run/apache2
+ENV APACHE_LOCK_DIR /var/lock/apache2
+
+RUN mkdir -p $APACHE_RUN_DIR $APACHE_LOCK_DIR $APACHE_LOG_DIR
+
+ADD wheelhouse /data/wheelhouse
+RUN pip3 install --no-index --find-links=/data/wheelhouse -r /data/wheelhouse/requirements.txt
+
+VOLUME /data/config
+VOLUME /data/storage
+VOLUME /var/log
+
+ENV USE_X_SENDFILE True
+
+EXPOSE 80
+EXPOSE 5000
+
+ADD apache/remoteip.conf /etc/apache2/mods-available/
+ADD apache/wsgi-py36.* /etc/apache2/mods-available/
+RUN a2enmod remoteip && a2enmod rewrite && a2enmod wsgi-py36
+
+ADD apache/apache2.conf /etc/apache2/apache2.conf
+ADD apache/000-default.conf /etc/apache2/sites-available/000-default.conf
+ADD apache/logrotate.conf /etc/logrotate.d/apache2
+ADD *.sh /
+
+# Remove some empty top-level directories we won't use anyway.
+RUN rmdir /media /home 2>/dev/null || true
+
+# This file includes some useful commands to have in the shell history
+# for easy access.
+ADD bash_history /root/.bash_history
+
+ENTRYPOINT /docker-entrypoint.sh
+
+# Add the most-changing files as last step for faster rebuilds.
+ADD config_local.py /data/git/blender-cloud/
+ADD staging /data/git
+RUN python3 -c "import re, secrets; \
+f = open('/data/git/blender-cloud/config_local.py', 'a'); \
+h = re.sub(r'[_.~-]', '', secrets.token_urlsafe())[:8]; \
+print(f'STATIC_FILE_HASH = {h!r}', file=f)"

docker/4_run/apache/000-default.conf

@@ -0,0 +1,56 @@
+<VirtualHost *:80>
+	XSendFile on
+	XSendFilePath /data/storage/pillar
+	XSendFilePath /data/git/pillar/pillar/web/static/
+	XSendFilePath /data/git/attract/attract/static/
+	XSendFilePath /data/git/flamenco/flamenco/static/
+	XsendFilePath /data/git/pillar-svnman/svnman/static/
+	XsendFilePath /data/git/blender-cloud/static/
+	XsendFilePath /data/git/blender-cloud/cloud/static/
+
+	ServerAdmin webmaster@localhost
+	DocumentRoot /var/www/html
+
+	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
+	# error, crit, alert, emerg.
+	# It is also possible to configure the loglevel for particular
+	# modules, e.g.
+	# LogLevel info ssl:warn
+
+	ErrorLog ${APACHE_LOG_DIR}/error.log
+	CustomLog ${APACHE_LOG_DIR}/access.log combined
+
+	WSGIDaemonProcess cloud processes=2 threads=64 maximum-requests=10000
+	WSGIPassAuthorization On
+
+	WSGIScriptAlias / /data/git/blender-cloud/runserver.wsgi \
+		process-group=cloud application-group=%{GLOBAL}
+
+	<Directory /data/git/blender-cloud>
+		<Files runserver.wsgi>
+			Require all granted
+		</Files>
+	</Directory>
+
+	# Temporary edit to remap the old cloudapi.blender.org to cloud.blender.org/api
+	RewriteEngine On
+	RewriteCond "%{HTTP_HOST}"   "^cloudapi\.blender\.org" [NC]
+	RewriteRule (.*) /api$1 [PT]
+
+	# Redirects for blender-cloud projects
+	RewriteRule "^/p/blender-cloud/?$"  "/blog" [R=301,L]
+	RewriteRule "^/agent327/?$"  "/p/agent-327" [R=301,L]
+	RewriteRule "^/caminandes/?$"  "/p/caminandes-3" [R=301,L]
+	RewriteRule "^/cf2/?$"  "/p/creature-factory-2" [R=301,L]
+	RewriteRule "^/characters/?$"  "/p/characters" [R=301,L]
+	RewriteRule "^/gallery/?$"  "/p/gallery" [R=301,L]
+	RewriteRule "^/hdri/?$"  "/p/hdri" [R=301,L]
+	RewriteRule "^/textures/?$"  "/p/textures" [R=301,L]
+	RewriteRule "^/training/?$"  "/courses" [R=301,L]
+	RewriteRule "^/spring/?$"  "/p/spring" [R=301,L]
+	RewriteRule "^/hero/?$"  "/p/hero" [R=301,L]
+	RewriteRule "^/coffee-run/?$"  "/p/coffee-run" [R=301,L]
+	RewriteRule "^/settlers/?$"  "/p/settlers" [R=301,L]
+	# Waking the forest was moved from the art gallery to its own workshop
+	RewriteRule "^/p/gallery/58cfec4f88ac8f1440aeb309/?$"  "/p/waking-the-forest" [R=301,L]
+</VirtualHost>

docker/4_run/apache/apache2.conf

@@ -133,9 +133,9 @@ AccessFileName .htaccess
 # Note that the use of %{X-Forwarded-For}i instead of %h is not recommended.
 # Use mod_remoteip instead.
 #
-LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
-LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
-LogFormat "%h %l %u %t \"%r\" %>s %O" common
+LogFormat "%v:%p %a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
+LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
+LogFormat "%a %l %u %t \"%r\" %>s %O" common
 LogFormat "%{Referer}i -> %U" referer
 LogFormat "%{User-agent}i" agent
 

docker/4_run/apache/logrotate.conf

@@ -0,0 +1,21 @@
+/var/log/apache2/*.log {
+    daily
+    missingok
+    rotate 14
+    size 100M
+    compress
+    delaycompress
+    notifempty
+    create 640 root adm
+    sharedscripts
+    postrotate
+        if /etc/init.d/apache2 status > /dev/null ; then \
+            /etc/init.d/apache2 reload > /dev/null; \
+        fi;
+    endscript
+    prerotate
+        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
+            run-parts /etc/logrotate.d/httpd-prerotate; \
+        fi; \
+    endscript
+}

docker/4_run/apache/remoteip.conf

@@ -0,0 +1,2 @@
+RemoteIPHeader X-Forwarded-For
+RemoteIPInternalProxy 172.16.0.0/12

docker/4_run/apache/wsgi-py36.conf

@@ -0,0 +1,122 @@
+<IfModule mod_wsgi.c>
+
+
+    #This config file is provided to give an overview of the directives,
+    #which are only allowed in the 'server config' context.
+    #For a detailed description of all avaiable directives please read
+    #http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives
+
+
+    #WSGISocketPrefix: Configure directory to use for daemon sockets.
+    #
+    #Apache's DEFAULT_REL_RUNTIMEDIR should be the proper place for WSGI's
+    #Socket. In case you want to mess with the permissions of the directory,
+    #you need to define WSGISocketPrefix to an alternative directory.
+    #See http://code.google.com/p/modwsgi/wiki/ConfigurationIssues for more
+    #information
+
+        #WSGISocketPrefix /var/run/apache2/wsgi
+
+
+    #WSGIPythonOptimize: Enables basic Python optimisation features.
+    #
+    #Sets the level of Python compiler optimisations. The default is '0'
+    #which means no optimisations are applied.
+    #Setting the optimisation level to '1' or above will have the effect
+    #of enabling basic Python optimisations and changes the filename
+    #extension for compiled (bytecode) files from .pyc to .pyo.
+    #When the optimisation level is set to '2', doc strings will not be
+    #generated and retained. This will result in a smaller memory footprint,
+    #but may cause some Python packages which interrogate doc strings in some
+    #way to fail.
+
+        #WSGIPythonOptimize 0
+
+
+    #WSGIPythonPath: Additional directories to search for Python modules,
+    #                overriding the PYTHONPATH environment variable.
+    #
+    #Used to specify additional directories to search for Python modules.
+    #If multiple directories are specified they should be separated by a ':'.
+
+        WSGIPythonPath /opt/python/lib/python3.6/site-packages
+
+    #WSGIPythonEggs: Directory to use for Python eggs cache.
+    #
+    #Used to specify the directory to be used as the Python eggs cache
+    #directory for all sub interpreters created within embedded mode.
+    #This directive achieves the same affect as having set the
+    #PYTHON_EGG_CACHE environment variable.
+    #Note that the directory specified must exist and be writable by the user
+    #that the Apache child processes run as. The directive only applies to
+    #mod_wsgi embedded mode. To set the Python eggs cache directory for
+    #mod_wsgi daemon processes, use the 'python-eggs' option to the
+    #WSGIDaemonProcess directive instead.
+
+        #WSGIPythonEggs directory
+
+
+
+    #WSGIRestrictEmbedded: Enable restrictions on use of embedded mode.
+    #
+    #The WSGIRestrictEmbedded directive determines whether mod_wsgi embedded
+    #mode is enabled or not. If set to 'On' and the restriction on embedded
+    #mode is therefore enabled, any attempt to make a request against a
+    #WSGI application which hasn't been properly configured so as to be
+    #delegated to a daemon mode process will fail with a HTTP internal server
+    #error response.
+
+        #WSGIRestrictEmbedded On|Off
+
+
+
+    #WSGIRestrictStdin: Enable restrictions on use of STDIN.
+    #WSGIRestrictStdout: Enable restrictions on use of STDOUT.
+    #WSGIRestrictSignal: Enable restrictions on use of signal().
+    #
+    #Well behaved WSGI applications neither should try to read/write from/to
+    #STDIN/STDOUT, nor should they try to register signal handlers. If your
+    #application needs an exception from this rule, you can disable the
+    #restrictions here.
+
+        #WSGIRestrictStdin On
+        #WSGIRestrictStdout On
+        #WSGIRestrictSignal On
+
+
+
+    #WSGIAcceptMutex: Specify type of accept mutex used by daemon processes.
+    #
+    #The WSGIAcceptMutex directive sets the method that mod_wsgi will use to
+    #serialize multiple daemon processes in a process group accepting requests
+    #on a socket connection from the Apache child processes. If this directive
+    #is not defined then the same type of mutex mechanism as used by Apache for
+    #the main Apache child processes when accepting connections from a client
+    #will be used. If set the method types are the same as for the Apache
+    #AcceptMutex directive.
+
+        #WSGIAcceptMutex default
+
+
+
+    #WSGIImportScript: Specify a script file to be loaded on process start.
+    #
+    #The WSGIImportScript directive can be used to specify a script file to be
+    #loaded when a process starts. Options must be provided to indicate the
+    #name of the process group and the application group into which the script
+    #will be loaded.
+
+        #WSGIImportScript process-group=name application-group=name
+
+
+    #WSGILazyInitialization: Enable/disable lazy initialisation of Python.
+    #
+    #The WSGILazyInitialization directives sets whether or not the Python
+    #interpreter is preinitialised within the Apache parent process or whether
+    #lazy initialisation is performed, and the Python interpreter only
+    #initialised in the Apache server processes or mod_wsgi daemon processes
+    #after they have forked from the Apache parent process.
+
+        #WSGILazyInitialization On|Off
+
+</IfModule>

docker/4_run/apache/wsgi-py36.load

@@ -0,0 +1 @@
+LoadModule wsgi_module /opt/python/mod-wsgi/mod_wsgi.so

docker/4_run/bash_history

@@ -0,0 +1,9 @@
+bash docker-entrypoint.sh
+env | sort
+apache2ctl start
+apache2ctl graceful
+/manage.sh operations worker -- -C
+celery status --broker amqp://guest:guest@rabbit:5672//
+celery events --broker amqp://guest:guest@rabbit:5672//
+tail -n 40 -f /var/log/apache2/access.log
+tail -n 40 -f /var/log/apache2/error.log

docker/4_run/build.sh

@@ -0,0 +1,5 @@
+#!/bin/bash -e
+
+docker build -t armadillica/blender_cloud:latest .
+
+echo "Done, built armadillica/blender_cloud:latest"

docker/4_run/celery-beat.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+source /install_scripts.sh
+source /manage.sh celery beat -- \
+    --schedule /data/storage/pillar/celerybeat-schedule.db \
+    --pid /data/storage/pillar/celerybeat.pid

docker/4_run/celery-worker.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+
+source /install_scripts.sh
+source /manage.sh celery worker -- -C

docker/4_run/config_local.py

@@ -0,0 +1,123 @@
+import os
+from collections import defaultdict
+
+DEBUG = False
+
+SCHEME = 'https'
+PREFERRED_URL_SCHEME = 'https'
+SERVER_NAME = 'cloud.blender.org'
+
+# os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = 'true'
+os.environ['PILLAR_MONGO_DBNAME'] = 'cloud'
+os.environ['PILLAR_MONGO_PORT'] = '27017'
+os.environ['PILLAR_MONGO_HOST'] = 'mongo'
+
+USE_X_SENDFILE = True
+
+STORAGE_BACKEND = 'gcs'
+
+CDN_SERVICE_DOMAIN = 'blendercloud-pro.r.worldssl.net'
+CDN_CONTENT_SUBFOLDER = ''
+CDN_STORAGE_ADDRESS = 'push-11.cdnsun.com'
+
+CACHE_TYPE = 'redis'  # null
+CACHE_KEY_PREFIX = 'pw_'
+CACHE_REDIS_HOST = 'redis'
+CACHE_REDIS_PORT = '6379'
+CACHE_REDIS_URL = 'redis://redis:6379'
+
+PILLAR_SERVER_ENDPOINT = 'https://cloud.blender.org/api/'
+
+BLENDER_ID_ENDPOINT = 'https://www.blender.org/id/'
+
+GCLOUD_APP_CREDENTIALS = '/data/config/google_app.json'
+GCLOUD_PROJECT = 'blender-cloud'
+
+MAIN_PROJECT_ID = '563a9c8cf0e722006ce97b03'
+# MAIN_PROJECT_ID = '57aa07c088bef606e89078bd'
+
+ALGOLIA_INDEX_USERS = 'pro_Users'
+ALGOLIA_INDEX_NODES = 'pro_Nodes'
+
+ZENCODER_NOTIFICATIONS_URL = 'https://cloud.blender.org/api/encoding/zencoder/notifications'
+
+FILE_LINK_VALIDITY = defaultdict(
+    lambda: 3600 * 24 * 30,  # default of 1 month.
+    gcs=3600 * 23,  # 23 hours for Google Cloud Storage.
+    cdnsun=3600 * 23
+)
+
+LOGGING = {
+    'version': 1,
+    'formatters': {
+        'default': {'format': '%(levelname)8s %(name)s %(message)s'}
+    },
+    'handlers': {
+        'console': {
+            'class': 'logging.StreamHandler',
+            'formatter': 'default',
+            'stream': 'ext://sys.stderr',
+        }
+    },
+    'loggers': {
+        'pillar': {'level': 'INFO'},
+        # 'pillar.auth': {'level': 'DEBUG'},
+        # 'pillar.api.blender_id': {'level': 'DEBUG'},
+        # 'pillar.api.blender_cloud.subscription': {'level': 'DEBUG'},
+        'bcloud': {'level': 'INFO'},
+        'cloud': {'level': 'INFO'},
+        'attract': {'level': 'INFO'},
+        'flamenco': {'level': 'INFO'},
+        # 'pillar.api.file_storage': {'level': 'DEBUG'},
+        # 'pillar.api.file_storage.ensure_valid_link': {'level': 'INFO'},
+        'pillar.api.file_storage.refresh_links_for_backend': {'level': 'DEBUG'},
+        'werkzeug': {'level': 'DEBUG'},
+        'eve': {'level': 'WARNING'},
+        # 'elasticsearch': {'level': 'DEBUG'},
+    },
+    'root': {
+        'level': 'WARNING',
+        'handlers': [
+            'console',
+        ],
+    }
+}
+
+# Latest version of the add-on.
+BLENDER_CLOUD_ADDON_VERSION = '1.9.0'
+
+REDIRECTS = {
+    # For old links, refer to the services page (hopefully it refreshes then)
+    'downloads/blender_cloud-latest-bundle.zip': 'https://cloud.blender.org/services#blender-addon',
+
+    # Latest Blender Cloud add-on.
+    'downloads/blender_cloud-latest-addon.zip':
+        f'https://storage.googleapis.com/institute-storage/addons/'
+        f'blender_cloud-{BLENDER_CLOUD_ADDON_VERSION}.addon.zip',
+
+    # Redirect old Grafista endpoint to /stats
+    '/stats/': '/stats',
+}
+
+UTM_LINKS = {
+    'cartoon_brew': {
+        'image': 'https://imgur.com/13nQTi3.png',
+        'link': 'https://store.blender.org/product/membership/'
+    }
+}
+
+SVNMAN_REPO_URL = 'https://svn.blender.cloud/repo/'
+SVNMAN_API_URL = 'https://svn.blender.cloud/api/'
+
+# Mail options, see pillar.celery.email_tasks.
+SMTP_HOST = 'proog.blender.org'
+SMTP_PORT = 25
+SMTP_USERNAME = 'server@blender.cloud'
+SMTP_TIMEOUT = 30  # timeout in seconds, https://docs.python.org/3/library/smtplib.html#smtplib.SMTP
+MAIL_RETRY = 180  # in seconds, delay until trying to send an email again.
+MAIL_DEFAULT_FROM_NAME = 'Blender Cloud'
+MAIL_DEFAULT_FROM_ADDR = 'cloudsupport@blender.org'
+
+# MUST be 8 characters long, see pillar.flask_extra.HashedPathConverter
+# STATIC_FILE_HASH = '12345678'
+# The value used in production is appended from Dockerfile.

docker/4_run/docker-entrypoint.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+source /install_scripts.sh
+
+# Make sure that log rotation works.
+mkdir -p ${APACHE_LOG_DIR}
+service cron start
+
+if [ "$DEV" = "true" ]; then
+    echo "Running in development mode"
+    cd /data/git/blender-cloud
+    exec bash /manage.sh runserver --host='0.0.0.0'
+else
+    exec /usr/sbin/apache2ctl -D FOREGROUND
+fi

docker/4_run/install_scripts.sh

@@ -0,0 +1,21 @@
+#!/bin/sh
+
+if [ -f /installed ]; then
+    return
+fi
+
+SITEPKG=$(echo /opt/python/lib/python3.*/site-packages)
+echo "Installing Blender Cloud packages into $SITEPKG"
+
+# TODO: 'pip3 install -e' runs 'setup.py develop', which runs 'setup.py egg_info',
+# which can't write the egg info to the read-only /data/git volume. This is why
+# we manually install the links.
+for SUBPROJ in /data/git/{pillar,pillar-python-sdk,attract,flamenco,pillar-svnman}; do
+    NAME=$(python3 $SUBPROJ/setup.py --name)
+    echo "... $NAME"
+    echo $SUBPROJ >> $SITEPKG/easy-install.pth
+    echo $SUBPROJ > $SITEPKG/$NAME.egg-link
+done
+echo "All packages installed."
+
+touch /installed

docker/4_run/manage.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+
+set -e
+cd /data/git/blender-cloud
+exec python manage.py "$@"

docker/README.md

@@ -0,0 +1,96 @@
+# Setting up a production machine
+
+To get the docker stack up and running, we use the following, on an Ubuntu 16.10 machine.
+
+## 0. Basic stuff
+
+Install the machine, use `locale-gen nl_NL.UTF-8` or similar commands to generate locale
+definitions. Set up automatic security updates and backups, the usual.
+
+## 1. Install Docker
+
+Install Docker itself, as described in the
+   [Docker CE for Ubuntu manual](https://store.docker.com/editions/community/docker-ce-server-ubuntu?tab=description):
+
+    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+    add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+        $(lsb_release -cs) stable"
+    apt-get update
+    apt-get install docker-ce
+
+## 2. Configure Docker to use "overlay"
+
+Configure Docker to use "overlay" instead of "aufs" for the images. This prevents
+[segfaults in auplink](https://bugs.launchpad.net/ubuntu/+source/aufs-tools/+bug/1442568).
+
+1. Set `DOCKER_OPTS="-s overlay"` in `/etc/defaults/docker`
+2. Copy `/lib/systemd/system/docker.service` to `/etc/systemd/system/docker.service`.
+   This allows later upgrading of docker without overwriting the changes we're about to do.
+2. Edit the `[Service]` section of `/etc/systemd/system/docker.service`:
+    1. Add `EnvironmentFile=/etc/default/docker`
+    2. Append ` $DOCKER_OPTS` to the `ExecStart` line
+3. Run `systemctl daemon-reload`
+4. Remove all your containers and images.
+5. Restart Docker: `systemctl restart docker`
+
+## 3. Pull the Blender Cloud docker image
+
+`docker pull armadillica/blender_cloud:latest`
+
+## 4. Get docker-compose + our repositories
+
+See the [Quick setup](../README.md) on how to get those. Then run:
+
+    cd /data/git/blender-cloud/docker
+    docker-compose up -d
+
+
+Set up permissions for Docker volumes; the following should be writable by
+
+- `/data/storage/pillar`: writable by `www-data` and `root` (do a `chown root:www-data`
+  and `chmod 2770`).
+- `/data/storage/db`: writable by uid 999.
+
+
+## 5. Set up TLS
+
+Place TLS certificates in `/data/certs/{cloud,cloudapi}.blender.org.pem`.
+They should contain (in order) the private key, the host certificate, and the
+CA certificate.
+
+
+## 6. Create a local config
+
+Blender Cloud expects the following files to exist:
+
+- `/data/git/blender_cloud/config_local.py` with machine-local configuration overrides
+- `/data/config/google_app.json` with Google Cloud Storage credentials.
+
+When run from Docker, the `docker/4_run/config_local.py` file will be used. Overrides for that file
+can be placed in `/data/config/config_secrets.py`.
+
+
+## 7. ElasticSearch & kibana
+
+ElasticSearch and Kibana run in our self-rolled images. This is needed because by default
+
+- ElasticSearch uses up to 2 GB of RAM, which is too much for our droplet, and
+- the Docker images contain the proprietary X-Pack plugin, which we don't want.
+
+This also gives us the opportunity to let Kibana do its optimization when we build the image, rather
+than every time the container is recreated.
+
+`/data/storage/elasticsearch` needs to be writable by UID 1000, GID 1000.
+
+Kibana connects to [ElasticProxy](https://github.com/armadillica/elasticproxy), which only allows
+GET, HEAD, and some specific POST requests. This ensures that the public-facing Kibana cannot be
+used to change the ElasticSearch database.
+
+Production Kibana can be placed in read-only mode, but this is not necessary now that we use
+ElasticProxy. However, I've left this in here as reference.
+
+`curl -XPUT 'localhost:9200/.kibana/_settings' -d '{ "index.blocks.read_only" : true }'`
+
+If editing is desired, temporarily turn off read-only mode:
+
+`curl -XPUT 'localhost:9200/.kibana/_settings' -d '{ "index.blocks.read_only" : false }'`

docker/build.sh

@@ -1,16 +0,0 @@
-#!/usr/bin/env bash
-
-set -x;
-set -e;
-
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-cd $DIR;
-
-cd 1_base/;
-bash build.sh;
-
-cd ../2_build/;
-bash build.sh;
-
-cd ../3_run/;
-bash build.sh;

docker/docker-compose.yml

@@ -1,68 +1,183 @@
-mongo:
-  image: mongo
-  container_name: mongo
-  restart: always
-  volumes:
-   - /data/storage/db:/data/db
-  ports:
-   - "127.0.0.1:27017:27017"
-redis:
-  image: redis
-  container_name: redis
-  restart: always
-blender_cloud:
-  image: armadillica/blender_cloud
-  container_name: blender_cloud
-  restart: always
-  environment:
-   VIRTUAL_HOST: http://cloudapi.blender.org/*,https://cloudapi.blender.org/*,http://cloud.blender.org/*,https://cloud.blender.org/*,http://pillar-web/*
-   VIRTUAL_HOST_WEIGHT: 10
-   FORCE_SSL: "true"
-  volumes:
-   - /data/git/blender-cloud:/data/git/blender-cloud:ro
-   - /data/git/attract:/data/git/attract:ro
-   - /data/git/flamenco:/data/git/flamenco:ro
-   - /data/git/pillar:/data/git/pillar:ro
-   - /data/git/pillar-python-sdk:/data/git/pillar-python-sdk:ro
-   - /data/config:/data/config:ro
-   - /data/storage/pillar:/data/storage/pillar
-  links:
-   - mongo
-   - redis
-# notifserv:
-#   container_name: notifserv
-#   image: armadillica/pillar-notifserv:cd8fa678436563ac3b800b2721e36830c32e4656
-#   restart: always
-#   links:
-#    - mongo
-#   environment:
-#    VIRTUAL_HOST: https://cloud.blender.org/notifications*,http://pillar-web/notifications*
-#    VIRTUAL_HOST_WEIGHT: 20
-#    FORCE_SSL: true
-grafista:
-  image: armadillica/grafista
-  container_name: grafista
-  restart: always
-  environment:
-    VIRTUAL_HOST: http://cloud.blender.org/stats/*,https://cloud.blender.org/stats/*,http://blender-cloud/stats/*
-    VIRTUAL_HOST_WEIGHT: 20
-    FORCE_SSL: "true"
-  volumes:
-    - /data/git/grafista:/data/git/grafista:ro
-    - /data/storage/grafista:/data/storage
-haproxy:
-  image: dockercloud/haproxy
-  container_name: haproxy
-  restart: always
-  ports:
-   - "443:443"
-   - "80:80"
-  environment:
-   - CERT_FOLDER=/certs/
-   - TIMEOUT=connect 5s, client 5m, server 10m
-  links:
-   - blender_cloud
-   - grafista
-   # - notifserv
-  volumes:
-   - '/data/certs:/certs'
+version:                     '3.4'
+services:
+  mongo:
+    image:                   mongo:3.4
+    container_name:          mongo
+    restart:                 always
+    volumes:
+      - /data/storage/db:/data/db
+      - /data/storage/db-bak:/data/db-bak # for backing up stuff etc.
+    ports:
+      - "127.0.0.1:27017:27017"
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  # Databases in use:
+  # 0: Flask Cache
+  # 1: Celery (backend)
+  # 2: Celery (broker)
+  redis:
+    image:                   redis:5.0
+    container_name:          redis
+    restart:                 always
+    ports:
+      - "127.0.0.1:6379:6379"
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  elastic:
+    # This image is defined in blender-cloud/docker/elastic
+    image:                   armadillica/elasticsearch:6.1.1
+    container_name:          elastic
+    restart:                 always
+    volumes:
+      #  NOTE:               this path must be writable by UID=1000 GID=1000.
+      - /data/storage/elastic:/usr/share/elasticsearch/data
+    ports:
+      - "127.0.0.1:9200:9200"
+    environment:
+      ES_JAVA_OPTS:          "-Xms256m -Xmx256m"
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  elasticproxy:
+    image:                   armadillica/elasticproxy:1.2
+    container_name:          elasticproxy
+    restart:                 always
+    command:                 /elasticproxy -elastic http://elastic:9200/
+    depends_on:
+      - elastic
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  kibana:
+    # This image is defined in blender-cloud/docker/elastic
+    image:                   armadillica/kibana:6.1.1
+    container_name:          kibana
+    restart:                 always
+    environment:
+      SERVER_NAME:           "stats.cloud.blender.org"
+      ELASTICSEARCH_URL:     http://elasticproxy:9200
+      CONSOLE_ENABLED:       'false'
+      VIRTUAL_HOST:          http://stats.cloud.blender.org/*,https://stats.cloud.blender.org/*,http://stats.cloud.local/*,https://stats.cloud.local/*
+      VIRTUAL_HOST_WEIGHT:   20
+      FORCE_SSL:             "true"
+
+      # See https://github.com/elastic/kibana/issues/5170#issuecomment-163042525
+      NODE_OPTIONS:          "--max-old-space-size=200"
+    depends_on:
+      - elasticproxy
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  blender_cloud:
+    image:                   armadillica/blender_cloud:latest
+    container_name:          blender_cloud
+    restart:                 always
+    environment:
+      VIRTUAL_HOST:          http://cloud.blender.org/*,https://cloud.blender.org/*,http://cloud.local/*,https://cloud.local/*
+      VIRTUAL_HOST_WEIGHT:   10
+      FORCE_SSL:             "true"
+      GZIP_COMPRESSION_TYPE: "text/html text/plain text/css application/javascript"
+      PILLAR_CONFIG:         /data/config/config_secrets.py
+    volumes:
+      # format:              HOST:CONTAINER
+      - /data/config:/data/config:ro
+      - /data/storage/pillar:/data/storage/pillar
+      - /data/log:/var/log
+    depends_on:
+      - mongo
+      - redis
+
+  celery_worker:
+    image:                   armadillica/blender_cloud:latest
+    entrypoint:              /celery-worker.sh
+    container_name:          celery_worker
+    restart:                 always
+    environment:
+      PILLAR_CONFIG:         /data/config/config_secrets.py
+    volumes:
+      # format:              HOST:CONTAINER
+      - /data/config:/data/config:ro
+      - /data/storage/pillar:/data/storage/pillar
+      - /data/log:/var/log
+    depends_on:
+      - mongo
+      - redis
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  celery_beat:
+    image:                   armadillica/blender_cloud:latest
+    entrypoint:              /celery-beat.sh
+    container_name:          celery_beat
+    restart:                 always
+    environment:
+      PILLAR_CONFIG:         /data/config/config_secrets.py
+    volumes:
+      # format:              HOST:CONTAINER
+      - /data/config:/data/config:ro
+      - /data/storage/pillar:/data/storage/pillar
+      - /data/log:/var/log
+    depends_on:
+      - mongo
+      - redis
+      - celery_worker
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  letsencrypt:
+    image:                   armadillica/picohttp:1.0
+    container_name:          letsencrypt
+    restart:                 always
+    environment:
+      WEBROOT:               /data/letsencrypt
+      LISTEN:                '[::]:80'
+      VIRTUAL_HOST:          http://cloud.blender.org/.well-known/*, http://stats.cloud.blender.org/.well-known/*
+      VIRTUAL_HOST_WEIGHT:   30
+    volumes:
+      - /data/letsencrypt:/data/letsencrypt
+
+  haproxy:
+    # This image is defined in blender-cloud/docker/haproxy
+    image:                   armadillica/haproxy:1.6.7
+    container_name:          haproxy
+    restart:                 always
+    ports:
+      - "443:443"
+      - "80:80"
+    environment:
+      - ADDITIONAL_SERVICES=docker:blender_cloud,docker:letsencrypt,docker:kibana
+      - CERT_FOLDER=/certs/
+      - TIMEOUT=connect 5s, client 5m, server 10m
+      - SSL_BIND_CIPHERS=ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
+      - SSL_BIND_OPTIONS=no-sslv3
+      - EXTRA_GLOBAL_SETTINGS=tune.ssl.default-dh-param 2048
+    depends_on:
+      - blender_cloud
+      - letsencrypt
+      - kibana
+    volumes:
+      - '/data/certs:/certs'
+      - /var/run/docker.sock:/var/run/docker.sock

docker/elastic/Dockerfile-elastic

@@ -0,0 +1,10 @@
+FROM docker.elastic.co/elasticsearch/elasticsearch:6.1.1
+LABEL maintainer Sybren A. Stüvel <sybren@blender.studio>
+
+RUN elasticsearch-plugin remove --purge  x-pack
+
+ADD elasticsearch.yml jvm.options /usr/share/elasticsearch/config/
+
+USER root
+RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/config/
+USER elasticsearch

docker/elastic/Dockerfile-kibana

@@ -0,0 +1,6 @@
+FROM docker.elastic.co/kibana/kibana:6.1.1
+LABEL maintainer Sybren A. Stüvel <sybren@blender.studio>
+
+RUN bin/kibana-plugin remove x-pack
+ADD kibana.yml /usr/share/kibana/config/kibana.yml
+RUN kibana 2>&1 | grep -m 1 "Optimization of .* complete"

docker/elastic/build.sh

@@ -0,0 +1,15 @@
+#!/bin/bash -e
+
+# When updating this, also update the versions in Dockerfile-*, and make sure that
+# it matches the versions of the elasticsearch and elasticsearch_dsl packages
+# used in Pillar. Those don't have to match exactly, but the major version should.
+VERSION=6.1.1
+
+docker build -t armadillica/elasticsearch:${VERSION} -f Dockerfile-elastic .
+docker build -t armadillica/kibana:${VERSION} -f Dockerfile-kibana .
+
+docker tag armadillica/elasticsearch:${VERSION} armadillica/elasticsearch:latest
+docker tag armadillica/kibana:${VERSION} armadillica/kibana:latest
+
+echo "Done, built armadillica/elasticsearch:${VERSION} and armadillica/kibana:${VERSION}"
+echo "Also tagged as armadillica/elasticsearch:latest and armadillica/kibana:latest"

docker/elastic/elasticsearch.yml

@@ -0,0 +1,7 @@
+cluster.name: "blender-cloud"
+network.host: 0.0.0.0
+
+# minimum_master_nodes need to be explicitly set when bound on a public IP
+# set to 1 to allow single node clusters
+# Details: https://github.com/elastic/elasticsearch/pull/17288
+discovery.zen.minimum_master_nodes: 1

docker/elastic/jvm.options

@@ -0,0 +1,112 @@
+## JVM configuration
+
+################################################################
+## IMPORTANT: JVM heap size
+################################################################
+##
+## You should always set the min and max JVM heap
+## size to the same value. For example, to set
+## the heap to 4 GB, set:
+##
+## -Xms4g
+## -Xmx4g
+##
+## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
+## for more information
+##
+################################################################
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+# Sybren: uncommented so that we can set those options using the ES_JAVA_OPTS environment variable.
+#-Xms512m
+#-Xmx512m
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+## GC configuration
+-XX:+UseConcMarkSweepGC
+-XX:CMSInitiatingOccupancyFraction=75
+-XX:+UseCMSInitiatingOccupancyOnly
+
+## optimizations
+
+# pre-touch memory pages used by the JVM during initialization
+-XX:+AlwaysPreTouch
+
+## basic
+
+# force the server VM (remove on 32-bit client JVMs)
+-server
+
+# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
+-Xss1m
+
+# set to headless, just in case
+-Djava.awt.headless=true
+
+# ensure UTF-8 encoding by default (e.g. filenames)
+-Dfile.encoding=UTF-8
+
+# use our provided JNA always versus the system one
+-Djna.nosys=true
+
+# use old-style file permissions on JDK9
+-Djdk.io.permissionsUseCanonicalPath=true
+
+# flags to configure Netty
+-Dio.netty.noUnsafe=true
+-Dio.netty.noKeySetOptimization=true
+-Dio.netty.recycler.maxCapacityPerThread=0
+
+# log4j 2
+-Dlog4j.shutdownHookEnabled=false
+-Dlog4j2.disable.jmx=true
+-Dlog4j.skipJansi=true
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+-XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps
+# ensure the directory exists and has sufficient space
+#-XX:HeapDumpPath=${heap.dump.path}
+
+## GC logging
+
+#-XX:+PrintGCDetails
+#-XX:+PrintGCTimeStamps
+#-XX:+PrintGCDateStamps
+#-XX:+PrintClassHistogram
+#-XX:+PrintTenuringDistribution
+#-XX:+PrintGCApplicationStoppedTime
+
+# log GC status to a file with time stamps
+# ensure the directory exists
+#-Xloggc:${loggc}
+
+# By default, the GC log file will not rotate.
+# By uncommenting the lines below, the GC log file
+# will be rotated every 128MB at most 32 times.
+#-XX:+UseGCLogFileRotation
+#-XX:NumberOfGCLogFiles=32
+#-XX:GCLogFileSize=128M
+
+# Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.
+# If documents were already indexed with unquoted fields in a previous version
+# of Elasticsearch, some operations may throw errors.
+#
+# WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided
+# only for migration purposes.
+#-Delasticsearch.json.allow_unquoted_field_names=true

docker/elastic/kibana.yml

@@ -0,0 +1,8 @@
+---
+
+server.name: kibana
+server.host: "0"
+elasticsearch.url: http://elasticsearch:9200
+
+# Hide dev tools
+console.enabled: false

docker/full_rebuild.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+
+set -xe
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+cd $DIR/1_base
+bash build.sh
+
+cd $DIR/2_buildpy
+bash build.sh
+
+cd $DIR/3_buildwheels
+bash build.sh
+
+cd $DIR/4_run
+bash build.sh

docker/haproxy/Dockerfile

@@ -0,0 +1,5 @@
+FROM dockercloud/haproxy:1.6.7
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+# Fix https://talosintelligence.com/vulnerability_reports/TALOS-2019-0782
+RUN sed 's/root::/root:!:/' -i /etc/shadow

docker/haproxy/build.sh

@@ -0,0 +1,10 @@
+#!/bin/bash -e
+
+# When updating this, also update the version in Dockerfile
+VERSION=1.6.7
+
+docker build -t armadillica/haproxy:${VERSION} .
+docker tag armadillica/haproxy:${VERSION} armadillica/haproxy:latest
+
+echo "Done, built armadillica/haproxy:${VERSION}"
+echo "Also tagged as armadillica/haproxy:latest"

docker/mongo-backup.cron

@@ -0,0 +1,5 @@
+# Change to suit your needs, then place in /etc/cron.d/mongo-backup
+# (so remove the .cron from the name)
+
+MAILTO=yourname@youraddress.org
+30 5 * * * root /root/docker/mongo-backup.sh

docker/mongo-backup.sh

@@ -0,0 +1,28 @@
+#!/bin/bash -e
+
+BACKUPDIR=/data/storage/db-bak
+DATE=$(date +'%Y-%m-%d-%H%M')
+ARCHIVE=$BACKUPDIR/mongo-live-$DATE.tar.xz
+
+# Just a sanity check before we give it to 'rm -rf'
+if [ -z "$DATE" ]; then
+    echo "Empty string found where the date should be, aborting."
+    exit 1
+fi
+
+
+# /data/db-bak in Docker is /data/storage/db-bak on the host.
+docker exec mongo mongodump -d cloud \
+    --out /data/db-bak/dump-$DATE \
+    --excludeCollection tokens \
+    --excludeCollection flamenco_task_logs \
+    --quiet
+
+cd $BACKUPDIR
+tar -Jcf $ARCHIVE dump-$DATE/
+rm -rf dump-$DATE
+
+TO_DELETE="$(ls $BACKUPDIR/mongo-live-*.tar.xz | head -n -7)"
+[ -z "$TO_DELETE" ] || rm "$TO_DELETE"
+
+rsync -a $BACKUPDIR/mongo-live-*.tar.xz cloud-backup@swami-direct.blender.cloud:

docker/renew-letsencrypt.sh

@@ -0,0 +1,27 @@
+#!/bin/bash -e
+
+# First time creating a certificate for a domain, use:
+# certbot certonly --webroot -w /data/letsencrypt -d $DOMAINNAME
+
+cd /data/letsencrypt
+
+certbot renew
+
+echo
+echo "Recreating HAProxy certificates"
+
+for certdir in /etc/letsencrypt/live/*; do
+    domain=$(basename $certdir)
+    echo "  - $domain"
+
+    cat $certdir/privkey.pem $certdir/fullchain.pem > $domain.pem
+    mv $domain.pem /data/certs/
+done
+
+
+echo
+echo -n "Restarting "
+docker restart haproxy
+
+echo "Certificate renewal completed."
+

gulp

@@ -0,0 +1,33 @@
+#!/bin/bash -ex
+
+GULP=./node_modules/.bin/gulp
+
+function install() {
+    npm install
+    touch $GULP  # installer doesn't always touch this after a build, so we do.
+}
+
+# Rebuild Gulp if missing or outdated.
+[ -e $GULP ] || install
+[ gulpfile.js -nt $GULP ] && install
+
+if [ "$1" == "watch" ]; then
+    # Treat "gulp watch" as "gulp && gulp watch"
+    $GULP
+elif [ "$1" == "all" ]; then
+    pushd .
+    # This is useful when building the Blender Cloud project for the first time.
+    # Run ./gulp in all depending projects (pillar, attract, flamenco, pillar-svnman)
+    declare -a repos=("pillar" "attract" "flamenco" "pillar-svnman")
+    for r in "${repos[@]}"
+    do
+        cd ../$r
+        ./gulp
+    done
+    popd
+    # Run "gulp" once inside the repo
+    $GULP
+    exit 1
+fi
+
+exec $GULP "$@"

gulpfile.js

@@ -0,0 +1,130 @@
+let argv         = require('minimist')(process.argv.slice(2));
+let autoprefixer = require('gulp-autoprefixer');
+let cache        = require('gulp-cached');
+let chmod        = require('gulp-chmod');
+let concat       = require('gulp-concat');
+let git          = require('gulp-git');
+let gulp         = require('gulp');
+let gulpif       = require('gulp-if');
+let pug          = require('gulp-pug');
+let plumber      = require('gulp-plumber');
+let rename       = require('gulp-rename');
+let sass         = require('gulp-sass');
+let sourcemaps   = require('gulp-sourcemaps');
+let uglify       = require('gulp-uglify-es').default;
+
+let enabled = {
+    uglify: argv.production,
+    maps: !argv.production,
+    failCheck: !argv.production,
+    prettyPug: !argv.production,
+    cachify: !argv.production,
+    cleanup: argv.production,
+    chmod: argv.production,
+};
+
+let destination = {
+    css: 'cloud/static/assets/css',
+    pug: 'cloud/templates',
+    js: 'cloud/static/assets/js',
+}
+
+let source = {
+    pillar: '../pillar/'
+}
+
+
+/* CSS */
+gulp.task('styles', function(done) {
+    gulp.src('src/styles/**/*.sass')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.maps, sourcemaps.init()))
+        .pipe(sass({
+            outputStyle: 'compressed'}
+            ))
+        .pipe(autoprefixer("last 3 versions"))
+        .pipe(gulpif(enabled.maps, sourcemaps.write(".")))
+        .pipe(gulp.dest(destination.css));
+    done();
+});
+
+
+/* Templates - Pug */
+gulp.task('templates', function(done) {
+    gulp.src('src/templates/**/*.pug')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.cachify, cache('templating')))
+        .pipe(pug({
+            pretty: enabled.prettyPug
+        }))
+        .pipe(gulp.dest(destination.pug));
+    // TODO(venomgfx): please check why 'gulp watch' doesn't pick up on .txt changes.
+    gulp.src('src/templates/**/*.txt')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.cachify, cache('templating')))
+        .pipe(gulp.dest(destination.pug));
+    done();
+});
+
+
+/* Tutti gets built by Pillar. See gulpfile.js in pillar.*/
+
+
+/* Individual Uglified Scripts */
+gulp.task('scripts', function(done) {
+    gulp.src('src/scripts/*.js')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.cachify, cache('scripting')))
+        .pipe(gulpif(enabled.maps, sourcemaps.init()))
+        .pipe(gulpif(enabled.uglify, uglify()))
+        .pipe(rename({suffix: '.min'}))
+        .pipe(gulpif(enabled.maps, sourcemaps.write(".")))
+        .pipe(gulpif(enabled.chmod, chmod(0o644)))
+        .pipe(gulp.dest(destination.js));
+    done();
+});
+
+
+// While developing, run 'gulp watch'
+gulp.task('watch',function(done) {
+    let watchStyles = [
+        'src/styles/**/*.sass',
+        source.pillar + 'src/styles/**/*.sass',
+    ];
+
+    let watchScripts = [
+        'src/scripts/**/*.js',
+        source.pillar + 'src/scripts/**/*.js',
+    ];
+
+    let watchTemplates = [
+        'src/templates/**/*.pug',
+        source.pillar + 'src/templates/**/*.pug',
+    ];
+
+    gulp.watch(watchStyles, gulp.series('styles'));
+    gulp.watch(watchScripts, gulp.series('scripts'));
+    gulp.watch(watchTemplates, gulp.series('templates'));
+    done();
+});
+
+
+// Erases all generated files in output directories.
+gulp.task('cleanup', function(done) {
+    let paths = [];
+    for (attr in destination) {
+        paths.push(destination[attr]);
+    }
+
+    git.clean({ args: '-f -X ' + paths.join(' ') }, function (err) {
+        if(err) throw err;
+    });
+    done();
+});
+
+
+// Run 'gulp' to build everything at once
+let tasks = [];
+if (enabled.cleanup) tasks.push('cleanup');
+
+gulp.task('default', gulp.parallel(tasks.concat(['styles', 'templates', 'scripts'])));

jwtkeys/README.md

@@ -0,0 +1,6 @@
+# Flamenco Server JWT keys
+
+To generate a keypair for `ES256`:
+
+    openssl ecparam -genkey -name prime256v1 -noout -out es256-private.pem
+    openssl ec -in es256-private.pem -pubout -out es256-public.pem

manage.py

@@ -1,40 +1,7 @@
 #!/usr/bin/env python
 
-from __future__ import print_function
-
-import logging
-from flask import current_app
 from pillar import cli
-from pillar.cli import manager_maintenance
-from cloud import app
-
-log = logging.getLogger(__name__)
-
-
-@manager_maintenance.command
-def reconcile_subscribers():
-    """For every user, check their subscription status with the store."""
-    from pillar.auth.subscriptions import fetch_user
-
-    users_coll = current_app.data.driver.db['users']
-    unsubscribed_users = []
-    for user in users_coll.find({'roles': 'subscriber'}):
-        print('Processing %s' % user['email'])
-        print('  Checking subscription')
-        user_store = fetch_user(user['email'])
-        if user_store['cloud_access'] == 0:
-            print('    Removing subscriber role')
-            users_coll.update(
-                {'_id': user['_id']},
-                {'$pull': {'roles': 'subscriber'}})
-            unsubscribed_users.append(user['email'])
-
-    if not unsubscribed_users:
-        return
-
-    print('The following users have been unsubscribed')
-    for user in unsubscribed_users:
-        print(user)
+from runserver import app
 
 cli.manager.app = app
 cli.manager.run()

package.json

@@ -0,0 +1,32 @@
+{
+	"name": "blender-cloud",
+	"license": "GPL-2.0+",
+	"author": "Blender Institute",
+	"repository": {
+		"type": "git",
+		"url": "git://git.blender.org/blender-cloud.git"
+	},
+	"devDependencies": {
+		"gulp": "~4.0",
+		"gulp-autoprefixer": "~6.0.0",
+		"gulp-cached": "~1.1.1",
+		"gulp-chmod": "~2.0.0",
+		"gulp-concat": "~2.6.1",
+		"gulp-if": "^2.0.2",
+		"gulp-git": "~2.8.0",
+		"gulp-plumber": "~1.2.0",
+		"gulp-pug": "~4.0.1",
+		"gulp-rename": "~1.4.0",
+		"gulp-sass": "~4.1.0",
+		"gulp-sourcemaps": "~2.6.4",
+		"gulp-uglify-es": "^1.0.4",
+		"minimist": "^1.2.0"
+	},
+	"dependencies": {
+		"bootstrap": "^4.1.3",
+		"jquery": "^3.3.1",
+		"natives": "^1.1.6",
+		"popper.js": "^1.14.4",
+		"video.js": "^7.2.2"
+	}
+}