Work around cryptography requiring Rust compiler

Pins all the build dependencies required by poetry, otherwise
poetry installs arbitrary versions of them, which leads to a wrong
version of cryptography being installed and the build failing.
Note that Cloud dependencies (pillar and the like) has to have their
build dependencies pinned in the same exact manner, for the same reason.

Pins all the runtime dependencies as well, because Cloud can only use
poetry==1.0 due to its source dependencies, so there's no
"poetry lock --no-update" and each "poetry lock" updates arbitrary packages.

.gitignore

@@ -3,6 +3,12 @@
 .coverage
 *.pyc
 __pycache__
+*.js.map
+*.css.map
+
+/cloud/templates/
+/cloud/static/assets/
+node_modules/
 
 /config_local.py
 
@@ -12,4 +18,10 @@ __pycache__
 /.eggs/
 /dump/
 /google_app*.json
-/docker/3_run/wheelhouse/
+/docker/2_buildpy/python/
+/docker/4_run/wheelhouse/
+/docker/4_run/deploy/
+/docker/4_run/staging/
+/celerybeat-schedule.bak
+/celerybeat-schedule.dat
+/celerybeat-schedule.dir

README.md

@@ -1,43 +1,102 @@
 # Blender Cloud
 
-Welcome to the [Blender Cloud](https://cloud.blender.org) code repo!
-Blender Cloud runs on the [Pillar](https://pillarframework.org) framework.
+Welcome to the [Blender Cloud](https://cloud.blender.org/) code repo!
+Blender Cloud runs on the [Pillar](https://pillarframework.org/) framework.
 
-## Quick setup
-Set up a node with these commands. Note that that script is already outdated...
+## Development setup
+Jumpstart Blender Cloud development with this simple guide.
+
+### System setup
+Blender Cloud relies on a number of services in order to run. Check out the [Pillar system setup](
+https://pillarframework.org/development/system_setup/#step-by-step-setup) to set this up.
+
+Add `cloud.local` to the `/etc/hosts` file on localhost. This is a development convention.
+
+### Check out the code
+Go to the local development directory and check out the following repositories, next to each other.
 
 ```
-#!/usr/bin/env bash
-
-mkdir -p /data/git
-mkdir -p /data/storage
-mkdir -p /data/config
-mkdir -p /data/certs
-
-sudo apt-get update
-sudo apt-get -y install python-pip
-pip install docker-compose
-
-cd /data/git
+cd /home/guest/Developer
 git clone git://git.blender.org/pillar-python-sdk.git
-git clone git://git.blender.org/pillar-server.git pillar
+git clone git://git.blender.org/pillar.git
 git clone git://git.blender.org/attract.git
 git clone git://git.blender.org/flamenco.git
+git clone git://git.blender.org/pillar-svnman.git
 git clone git://git.blender.org/blender-cloud.git
 ```
 
-## Deploying to production server
+### Initial setup and configuration
 
-First of all, add those aliases to the `[alias]` section of your `~/.gitconfig`
+Create a virtualenv for the project and install the requirements. Dependencies are managed via
+[Poetry](https://poetry.eustace.io/). Install it using `pip install -U --user poetry`.
+
+```
+cd blender-cloud
+pip install --user -U poetry
+poetry install
+```
+
+NOTE: After a dependency changed its own dependencies (say a new library was added as dependency of
+Pillar), you need to run `poetry update`. This will take the new dependencies into account and write
+them to the `poetry.lock` file.
+
+Build assets and templates for all Blender Cloud dependencies using Gulp.
+
+```
+./gulp all
+```
+
+Make a copy of the config_local example, which will be further edited as the application is
+configured.
+
+```
+cp config_local.example.py config_local.py
+```
+
+Setup the database with the initial collections and the admin user.
+
+```
+poetry run ./manage.py setup setup_db your_email
+```
+
+The command will return the following message:
+
+```
+Created project <project_id> for user <user_id>
+```
+
+Copy the value of `<project_id>` and assign it as value for `MAIN_PROJECT_ID`.
+
+Run the application:
+
+```
+poetry run ./manage.py runserver
+```
+
+
+## Development
+
+When ready to commit, change the remotes to work over SSH. For example:
+
+`git remote set-url origin git@git.blender.org:blender-cloud.git`
+
+For more information, check out [this guide](https://wiki.blender.org/wiki/Tools/Git#Commit_Access).
+
+
+## Preparing the production branch for deployment
+
+All revisions to deploy to production should be on the `production` branches of all the relevant
+repositories.
+
+Make sure you have these aliases in the `[alias]` section of your `~/.gitconfig`:
 
 ```
 prod = "!git checkout production && git fetch origin production && gitk --all"
 ff = "merge --ff-only"
-pp = "!git push && if [ -e deploy.sh ]; then ./deploy.sh; fi && git checkout master"
 ```
 
-The following commands should be executed for each subproject; specifically for
-Pillar and Attract:
+The following commands should be executed for each (sub)project; specifically for
+the current repository, Pillar, Attract, Flamenco, and Pillar-SVNMan:
 
 ```
 cd $projectdir
@@ -48,7 +107,7 @@ git stash  # if you still have local stuff.
 
 # pull from master, run unittests, push your changes to master.
 git pull
-py.test
+poetry run py.test
 git push
 
 # Switch to production branch, and investigate the situation.
@@ -58,37 +117,12 @@ git prod
 git ff master
 
 # Run tests again
-py.test
+poetry run py.test
 
-# Push the production branch and run dummy deploy script.
-git pp  # pp = "Push to Production"
-
-# The above alias waits for [ENTER] until all deploys are done.
-# Let it wait, perform the other commands in another terminal.
+# Push the production branch.
+git push
 ```
 
-Now follow the above receipe on the Blender Cloud project as well.
-Contrary to the subprojects, `git pp` will actually perform the deploy
-for real.
+## Deploying to production server
 
-Now you can press `[ENTER]` in the Pillar and Attract terminals that
-were still waiting for it.
-
-After everything is done, your (sub)projects should all be back on
-the master branch.
-
-
-## Updating dependencies via Docker images
-
-To update dependencies that need compiling, you need the `2_build` docker
-container. To rebuild the lot, run `docker/build.sh`.
-
-Follow these steps to deploy the new container on production:
-
-1. run `docker/build.sh`
-2. `docker push armadillica/blender_cloud`
-
-On the production machine:
-
-1. `docker pull armadillica/blender_cloud`
-2. `docker-compose up -d` (from the `/data/git/blender-cloud/docker` directory)
+See [deploy/README.md](deploy/README.md).

cloud/__init__.py

@@ -0,0 +1,155 @@
+import logging
+
+import flask
+from werkzeug.local import LocalProxy
+
+import pillarsdk
+import pillar.auth
+from pillar.api.utils import authorization
+from pillar.extension import PillarExtension
+
+EXTENSION_NAME = 'cloud'
+
+
+class CloudExtension(PillarExtension):
+    has_context_processor = True
+    user_roles = {'subscriber-pro', 'has_subscription'}
+    user_roles_indexable = {'subscriber-pro', 'has_subscription'}
+
+    user_caps = {
+        'has_subscription': {'can-renew-subscription'},
+    }
+
+    def __init__(self):
+        self._log = logging.getLogger('%s.CloudExtension' % __name__)
+
+    @property
+    def name(self):
+        return EXTENSION_NAME
+
+    def flask_config(self):
+        """Returns extension-specific defaults for the Flask configuration.
+
+        Use this to set sensible default values for configuration settings
+        introduced by the extension.
+
+        :rtype: dict
+        """
+
+        # Just so that it registers the management commands.
+        from . import cli
+
+        return {
+            'EXTERNAL_SUBSCRIPTIONS_MANAGEMENT_SERVER': 'https://store.blender.org/api/',
+            'EXTERNAL_SUBSCRIPTIONS_TIMEOUT_SECS': 10,
+            'BLENDER_ID_WEBHOOK_USER_CHANGED_SECRET': 'oos9wah1Zoa0Yau6ahThohleiChephoi',
+            'NODE_TAGS': ['animation', 'modeling', 'rigging', 'sculpting', 'shading', 'texturing', 'lighting',
+                          'character-pipeline',  'effects', 'video-editing', 'digital-painting', 'production-design',
+                          'walk-through'],
+        }
+
+    def eve_settings(self):
+        """Returns extensions to the Eve settings.
+
+        Currently only the DOMAIN key is used to insert new resources into
+        Eve's configuration.
+
+        :rtype: dict
+        """
+
+        return {}
+
+    def blueprints(self):
+        """Returns the list of top-level blueprints for the extension.
+
+        These blueprints will be mounted at the url prefix given to
+        app.load_extension().
+
+        :rtype: list of flask.Blueprint objects.
+        """
+        from . import routes
+        import cloud.stats.routes
+        return [
+            routes.blueprint,
+            cloud.stats.routes.blueprint,
+        ]
+
+    @property
+    def template_path(self):
+        import os.path
+        return os.path.join(os.path.dirname(__file__), 'templates')
+
+    @property
+    def static_path(self):
+        import os.path
+        return os.path.join(os.path.dirname(__file__), 'static')
+
+    def context_processor(self):
+        return {
+            'current_user_is_subscriber': authorization.user_has_cap('subscriber')
+        }
+
+    def is_cloud_project(self, project):
+        """Returns whether the project is set up for Blender Cloud.
+
+        Requires the presence of the 'cloud' key in extension_props
+        """
+
+        if project.extension_props is None:
+            # There are no extension_props on this project
+            return False
+
+        try:
+            pprops = project.extension_props[EXTENSION_NAME]
+        except AttributeError:
+            self._log.warning("is_cloud_project: Project url=%r doesn't have any "
+                              "extension properties.", project['url'])
+            if self._log.isEnabledFor(logging.DEBUG):
+                import pprint
+                self._log.debug('Project: %s', pprint.pformat(project.to_dict()))
+            return False
+        except KeyError:
+            # Not set up for Blender Cloud
+            return False
+
+        if pprops is None:
+            self._log.debug("is_cloud_project: Project url=%r doesn't have Blender Cloud"
+                            " extension properties.", project['url'])
+            return False
+        return True
+
+    @property
+    def has_project_settings(self) -> bool:
+        # Only available for admins
+        return pillar.auth.current_user.has_cap('admin')
+
+    def project_settings(self, project: pillarsdk.Project, **template_args: dict) -> flask.Response:
+        """Renders the project settings page for this extension.
+
+        Set YourExtension.has_project_settings = True and Pillar will call this function.
+
+        :param project: the project for which to render the settings.
+        :param template_args: additional template arguments.
+        :returns: a Flask HTTP response
+        """
+
+        from cloud.routes import project_settings
+
+        return project_settings(project, **template_args)
+
+    def setup_app(self, app):
+        from . import routes, webhooks, eve_hooks, email
+
+        routes.setup_app(app)
+        app.register_api_blueprint(webhooks.blueprint, '/webhooks')
+        eve_hooks.setup_app(app)
+        email.setup_app(app)
+
+
+def _get_current_cloud():
+    """Returns the Cloud extension of the current application."""
+    return flask.current_app.pillar_extensions[EXTENSION_NAME]
+
+
+current_cloud = LocalProxy(_get_current_cloud)
+"""Cloud extension of the current app."""

cloud/cli.py

@@ -0,0 +1,139 @@
+#!/usr/bin/env python
+
+import logging
+from urllib.parse import urljoin
+
+from flask import current_app
+from flask_script import Manager
+import requests
+
+from pillar.cli import manager
+from pillar.api import service
+from pillar.api.utils import authentication
+import cloud.setup
+
+log = logging.getLogger(__name__)
+
+manager_cloud = Manager(
+    current_app, usage="Blender Cloud scripts")
+
+
+@manager_cloud.command
+def create_groups():
+    """Creates the admin/demo/subscriber groups."""
+
+    import pprint
+
+    group_ids = {}
+    groups_coll = current_app.db('groups')
+
+    for group_name in ['admin', 'demo', 'subscriber']:
+        if groups_coll.find({'name': group_name}).count():
+            log.info('Group %s already exists, skipping', group_name)
+            continue
+        result = groups_coll.insert_one({'name': group_name})
+        group_ids[group_name] = result.inserted_id
+
+    service.fetch_role_to_group_id_map()
+    log.info('Created groups:\n%s', pprint.pformat(group_ids))
+
+
+@manager_cloud.command
+def reconcile_subscribers():
+    """For every user, check their subscription status with the store."""
+
+    import threading
+    import concurrent.futures
+
+    from pillar.auth import UserClass
+    from pillar.api import blender_id
+    from pillar.api.blender_cloud.subscription import do_update_subscription
+
+    sessions = threading.local()
+
+    service.fetch_role_to_group_id_map()
+
+    users_coll = current_app.data.driver.db['users']
+    found = users_coll.find({'auth.provider': 'blender-id'})
+    count_users = found.count()
+    count_skipped = count_processed = 0
+    log.info('Processing %i users', count_users)
+
+    lock = threading.Lock()
+
+    real_current_app = current_app._get_current_object()
+
+    api_token = real_current_app.config['BLENDER_ID_USER_INFO_TOKEN']
+    api_url = real_current_app.config['BLENDER_ID_USER_INFO_API']
+
+    def do_user(idx, user):
+        nonlocal count_skipped, count_processed
+
+        log.info('Processing %i/%i %s', idx + 1, count_users, user['email'])
+
+        # Get the Requests session for this thread.
+        try:
+            sess = sessions.session
+        except AttributeError:
+            sess = sessions.session = requests.Session()
+
+        # Get the info from Blender ID
+        bid_user_id = blender_id.get_user_blenderid(user)
+        if not bid_user_id:
+            with lock:
+                count_skipped += 1
+            return
+
+        url = urljoin(api_url, bid_user_id)
+        resp = sess.get(url, headers={'Authorization': f'Bearer {api_token}'})
+
+        if resp.status_code == 404:
+            log.info('User %s with Blender ID %s not found, skipping', user['email'], bid_user_id)
+            with lock:
+                count_skipped += 1
+            return
+
+        if resp.status_code != 200:
+            log.error('Unable to reach Blender ID (code %d), aborting', resp.status_code)
+            with lock:
+                count_skipped += 1
+            return
+
+        bid_user = resp.json()
+        if not bid_user:
+            log.error('Unable to parse response for user %s, aborting', user['email'])
+            with lock:
+                count_skipped += 1
+            return
+
+        # Actually update the user, and do it thread-safe just to be sure.
+        with real_current_app.app_context():
+            local_user = UserClass.construct('', user)
+            with lock:
+                do_update_subscription(local_user, bid_user)
+                count_processed += 1
+
+    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
+        future_to_user = {executor.submit(do_user, idx, user): user
+                          for idx, user in enumerate(found)}
+        for future in concurrent.futures.as_completed(future_to_user):
+            user = future_to_user[future]
+            try:
+                future.result()
+            except Exception as ex:
+                log.exception('Error updating user %s', user)
+
+    log.info('Done reconciling %d subscribers', count_users)
+    log.info('    processed: %d', count_processed)
+    log.info('    skipped  : %d', count_skipped)
+
+
+@manager_cloud.command
+def setup_for_film(project_url):
+    """Adds Blender Cloud film custom properties to a project."""
+
+    authentication.force_cli_user()
+    cloud.setup.setup_for_film(project_url)
+
+
+manager.add_command("cloud", manager_cloud)

cloud/email.py

@@ -0,0 +1,33 @@
+import functools
+import logging
+
+import flask
+
+from pillar.auth import UserClass
+
+log = logging.getLogger(__name__)
+
+
+def queue_welcome_mail(user: UserClass):
+    """Queue a welcome email for execution by Celery."""
+    assert user.email
+    log.info('queueing welcome email to %s', user.email)
+
+    subject = 'Welcome to Blender Cloud'
+    render = functools.partial(flask.render_template, subject=subject, user=user)
+    text = render('emails/welcome.txt')
+    html = render('emails/welcome.html')
+
+    from pillar.celery import email_tasks
+    email_tasks.send_email.delay(user.full_name, user.email, subject, text, html)
+
+
+def user_subscription_changed(user: UserClass, *, grant_roles: set, revoke_roles: set):
+    if user.has_cap('subscriber') and 'has_subscription' in grant_roles:
+        log.info('user %s just got a new subscription', user.email)
+        queue_welcome_mail(user)
+
+
+def setup_app(app):
+    from pillar.api.blender_cloud import subscription
+    subscription.user_subscription_updated.connect(user_subscription_changed)

cloud/eve_hooks.py

@@ -0,0 +1,39 @@
+import logging
+import typing
+
+from pillar.auth import UserClass
+
+from . import email
+
+log = logging.getLogger(__name__)
+
+
+def welcome_new_user(user_doc: dict):
+    """Sends a welcome email to a new user."""
+
+    user_email = user_doc.get('email')
+    if not user_email:
+        log.warning('user %s has no email address', user_doc.get('_id', '-no-id-'))
+        return
+
+    # Only send mail to new users when they actually are subscribers.
+    user = UserClass.construct('', user_doc)
+    if not (user.has_cap('subscriber') or user.has_cap('can-renew-subscription')):
+        log.debug('user %s is new, but not a subscriber, so no email for them.', user_email)
+        return
+
+    email.queue_welcome_mail(user)
+
+
+def welcome_new_users(user_docs: typing.List[dict]):
+    """Sends a welcome email to new users."""
+
+    for user_doc in user_docs:
+        try:
+            welcome_new_user(user_doc)
+        except Exception:
+            log.exception('error sending welcome mail to user %s', user_doc)
+
+
+def setup_app(app):
+    app.on_inserted_users += welcome_new_users

cloud/forms.py

@@ -0,0 +1,15 @@
+from flask_wtf import FlaskForm
+from wtforms import BooleanField, StringField
+from wtforms.fields.html5 import URLField
+from wtforms.validators import URL
+
+from pillar.web.utils.forms import FileSelectField
+
+
+class FilmProjectForm(FlaskForm):
+    video_url = URLField(validators=[URL()])
+    poster = FileSelectField('Poster Image', file_format='image')
+    logo = FileSelectField('Logo', file_format='image')
+    is_in_production = BooleanField('In Production')
+    is_featured = BooleanField('Featured')
+    theme_color = StringField('Theme Color')

cloud/routes.py

@@ -0,0 +1,658 @@
+import functools
+import json
+import logging
+import typing
+
+import bson
+from flask_login import login_required
+import flask
+import werkzeug.exceptions as wz_exceptions
+from flask import Blueprint, render_template, redirect, session, url_for, abort, flash, request
+from pillarsdk import Node, Project, User, exceptions as sdk_exceptions, Group
+from pillarsdk.exceptions import ResourceNotFound
+
+import pillar
+import pillarsdk
+from pillar import current_app
+from pillar.api.utils import authorization
+from pillar.auth import current_user
+from pillar.web.users import forms
+from pillar.web.utils import system_util, get_file, current_user_is_authenticated
+from pillar.web.utils import attach_project_pictures
+from pillar.web.settings import blueprint as blueprint_settings
+from pillar.web.nodes.routes import url_for_node
+from pillar.web.projects.routes import render_project
+from pillar.web.projects.routes import find_project_or_404
+from pillar.web.projects.routes import project_view
+from pillar.web.projects.routes import project_navigation_links
+
+from cloud import current_cloud
+from cloud.forms import FilmProjectForm
+from . import EXTENSION_NAME
+
+blueprint = Blueprint('cloud', __name__)
+log = logging.getLogger(__name__)
+
+
+@blueprint.route('/')
+def homepage():
+    if current_user.is_anonymous:
+        return redirect(url_for('cloud.welcome'))
+
+    return render_template(
+        'homepage.html',
+        api=system_util.pillar_api(),
+        **_homepage_context(),
+    )
+
+
+def _homepage_context() -> dict:
+    """Returns homepage template context variables."""
+
+    # Get latest blog posts
+    api = system_util.pillar_api()
+
+    # Get latest comments to any node
+    latest_comments = Node.latest('comments', api=api)
+
+    # Get a list of random featured assets
+    random_featured = get_random_featured_nodes()
+
+    # Parse results for replies
+    to_remove = []
+
+    @functools.lru_cache()
+    def _find_parent(parent_node_id) -> Node:
+        return Node.find(parent_node_id,
+                         {'projection': {
+                             '_id': 1,
+                             'name': 1,
+                             'node_type': 1,
+                             'project': 1,
+                             'parent': 1,
+                             'properties.url': 1,
+                         }},
+                         api=api)
+
+    for idx, comment in enumerate(latest_comments._items):
+        if comment.properties.is_reply:
+            try:
+                comment.attached_to = _find_parent(comment.parent.parent)
+            except ResourceNotFound:
+                # Remove this comment
+                to_remove.append(idx)
+        else:
+            comment.attached_to = comment.parent
+
+    for idx in reversed(to_remove):
+        del latest_comments._items[idx]
+
+    for comment in latest_comments._items:
+        if not comment.attached_to:
+            continue
+        comment.attached_to.url = url_for_node(node=comment.attached_to)
+        comment.url = url_for_node(node=comment)
+
+    main_project = Project.find(current_app.config['MAIN_PROJECT_ID'], api=api)
+    main_project.picture_header = get_file(main_project.picture_header, api=api)
+
+    return dict(
+        main_project=main_project,
+        latest_comments=latest_comments._items,
+        random_featured=random_featured)
+
+
+@blueprint.route('/design-system')
+def design_system():
+    """Display the design system page.
+
+    This endpoing is intended for development only, and returns a
+    rendered template only if the app is running in debug mode.
+    """
+    if not current_app.config['DEBUG']:
+        abort(404)
+    return render_template('design_system.html')
+
+
+@blueprint.route('/login')
+def login():
+    from flask import request
+
+    if request.args.get('force'):
+        log.debug('Forcing logout of user before rendering login page.')
+        pillar.auth.logout_user()
+
+    next_after_login = request.args.get('next')
+
+    if not next_after_login:
+        next_after_login = request.referrer
+
+    session['next_after_login'] = next_after_login
+    return redirect(url_for('users.oauth_authorize', provider='blender-id'))
+
+
+@blueprint.route('/welcome')
+def welcome():
+    # Workaround to cache rendering of a page if user not logged in
+    @current_app.cache.cached(timeout=3600, unless=current_user_is_authenticated)
+    def render_page():
+        return render_template('welcome.html')
+
+    return render_page()
+
+
+@blueprint.route('/about')
+def about():
+    return render_template('about.html')
+
+
+@blueprint.route('/services')
+def services():
+    return render_template('services.html')
+
+
+@blueprint.route('/learn')
+def learn():
+    return render_template('learn.html')
+
+
+@blueprint.route('/libraries')
+def libraries():
+    return render_template('libraries.html')
+
+
+@blueprint.route('/stats')
+def stats():
+    return render_template('stats.html')
+
+
+@blueprint.route('/join')
+def join():
+    """Join page"""
+    return redirect('https://store.blender.org/product/membership/')
+
+
+@blueprint.route('/renew')
+def renew_subscription():
+    return render_template('renew_subscription.html')
+
+
+def get_projects(category):
+    """Utility to get projects based on category. Should be moved on the API
+    and improved with more extensive filtering capabilities.
+    """
+    api = system_util.pillar_api()
+    projects = Project.all({
+        'where': {
+            'category': category,
+            'is_private': False},
+        'sort': '-_created',
+    }, api=api)
+    for project in projects._items:
+        attach_project_pictures(project, api)
+    return projects
+
+
+@blueprint.route('/courses')
+def courses():
+    @current_app.cache.cached(timeout=3600, unless=current_user_is_authenticated)
+    def render_page():
+        projects = get_projects('course')
+        return render_template(
+            'projects_index_collection.html',
+            title='courses',
+            projects=projects._items,
+            api=system_util.pillar_api())
+
+    return render_page()
+
+
+@blueprint.route('/open-projects')
+def open_projects():
+    @current_app.cache.cached(timeout=3600, unless=current_user_is_authenticated)
+    def render_page():
+        api = system_util.pillar_api()
+        projects = Project.all({
+            'where': {
+                'category': 'film',
+                'is_private': False
+            },
+            'sort': '-_created',
+        }, api=api)
+        for project in projects._items:
+            # Attach poster file (ensure the extension_props.cloud.poster
+            # attributes exists)
+            try:
+                # If the attribute exists, but is None, continue
+                if not project['extension_props'][EXTENSION_NAME]['poster']:
+                    continue
+                # Fetch the file and embed it in the document
+                project.extension_props.cloud.poster = get_file(
+                    project.extension_props.cloud.poster, api=api)
+                # Add convenience attribute that specifies the presence of the
+                # poster file
+                project.has_poster = True
+            # If there was a key error because one of the nested attributes is
+            # missing,
+            except KeyError:
+                continue
+
+        return render_template(
+            'films.html',
+            title='films',
+            projects=projects._items,
+            api=system_util.pillar_api())
+
+    return render_page()
+
+
+@blueprint.route('/workshops')
+def workshops():
+    @current_app.cache.cached(timeout=3600, unless=current_user_is_authenticated)
+    def render_page():
+        projects = get_projects('workshop')
+        return render_template(
+            'projects_index_collection.html',
+            title='workshops',
+            projects=projects._items,
+            api=system_util.pillar_api())
+
+    return render_page()
+
+
+def get_random_featured_nodes() -> typing.List[dict]:
+    """Returns a list of project/node combinations for featured nodes.
+
+    A random subset of 3 featured nodes from all public projects is returned.
+    Assumes that the user actually has access to the public projects' nodes.
+
+    The dict is a node, with a 'project' key that contains a projected project.
+    """
+
+    proj_coll = current_app.db('projects')
+    featured_nodes = proj_coll.aggregate([
+        {'$match': {'is_private': False}},
+        {'$project': {'nodes_featured': True,
+                      'url': True,
+                      'name': True,
+                      'summary': True,
+                      'picture_square': True}},
+        {'$unwind': {'path': '$nodes_featured'}},
+        {'$sample': {'size': 6}},
+        {'$lookup': {'from': 'nodes',
+                     'localField': 'nodes_featured',
+                     'foreignField': '_id',
+                     'as': 'node'}},
+        {'$unwind': {'path': '$node'}},
+        {'$match': {'node._deleted': {'$ne': True}}},
+        {'$project': {'url': True,
+                      'name': True,
+                      'summary': True,
+                      'picture_square': True,
+                      'node._id': True,
+                      'node.name': True,
+                      'node.permissions': True,
+                      'node.picture': True,
+                      'node.properties.content_type': True,
+                      'node.properties.duration_seconds': True,
+                      'node.properties.url': True,
+                      'node._created': True,
+                      }
+         },
+    ])
+
+    featured_node_documents = []
+    api = system_util.pillar_api()
+    for node_info in featured_nodes:
+        # Turn the project-with-node doc into a node-with-project doc.
+        node_document = node_info.pop('node')
+        node_document['project'] = node_info
+        node_document['_id'] = str(node_document['_id'])
+
+        node = Node(node_document)
+        node.picture = get_file(node.picture, api=api)
+        node.project.picture_square = get_file(node.project.picture_square, api=api)
+        featured_node_documents.append(node)
+
+    return featured_node_documents
+
+
+@blueprint_settings.route('/emails', methods=['GET', 'POST'])
+@login_required
+def emails():
+    """Main email settings.
+    """
+    if current_user.has_role('protected'):
+        return abort(404)  # TODO: make this 403, handle template properly
+    api = system_util.pillar_api()
+    user = User.find(current_user.objectid, api=api)
+
+    # Force creation of settings for the user (safely remove this code once
+    # implemented on account creation level, and after adding settings to all
+    # existing users)
+    if not user.settings:
+        user.settings = dict(email_communications=1)
+        user.update(api=api)
+
+    if user.settings.email_communications is None:
+        user.settings.email_communications = 1
+        user.update(api=api)
+
+    # Generate form
+    form = forms.UserSettingsEmailsForm(
+        email_communications=user.settings.email_communications)
+
+    if form.validate_on_submit():
+        try:
+            user.settings.email_communications = form.email_communications.data
+            user.update(api=api)
+            flash("Profile updated", 'success')
+        except sdk_exceptions.ResourceInvalid as e:
+            message = json.loads(e.content)
+            flash(message)
+
+    return render_template('users/settings/emails.html', form=form, title='emails')
+
+
+@blueprint_settings.route('/billing')
+@login_required
+def billing():
+    """View the subscription status of a user
+    """
+    from . import store
+
+    log.debug('START OF REQUEST')
+
+    if current_user.has_role('protected'):
+        return abort(404)  # TODO: make this 403, handle template properly
+
+    expiration_date = 'No subscription to expire'
+
+    # Classify the user based on their roles and capabilities
+    cap_subs = current_user.has_cap('subscriber')
+    if current_user.has_role('demo'):
+        user_cls = 'demo'
+    elif not cap_subs and current_user.has_cap('can-renew-subscription'):
+        # This user has an inactive but renewable subscription.
+        user_cls = 'subscriber-expired'
+    elif cap_subs:
+        if current_user.has_role('subscriber'):
+            # This user pays for their own subscription. Only in this case do we need to fetch
+            # the expiration date from the Store.
+            user_cls = 'subscriber'
+            store_user = store.fetch_subscription_info(current_user.email)
+            if store_user is None:
+                expiration_date = 'Unable to reach Blender Store to check'
+            else:
+                expiration_date = store_user['expiration_date'][:10]
+
+        elif current_user.has_role('org-subscriber'):
+            # An organisation pays for this subscription.
+            user_cls = 'subscriber-org'
+        else:
+            # This user gets the subscription cap from somewhere else (like an organisation).
+            user_cls = 'subscriber-other'
+    else:
+        user_cls = 'outsider'
+
+    return render_template(
+        'users/settings/billing.html',
+        user_cls=user_cls,
+        expiration_date=expiration_date,
+        title='billing')
+
+
+@blueprint.route('/terms-and-conditions')
+def terms_and_conditions():
+    return render_template('terms_and_conditions.html')
+
+
+@blueprint.route('/privacy')
+def privacy():
+    return render_template('privacy.html')
+
+
+@blueprint.route('/production')
+def production():
+    return render_template(
+        'production.html',
+        title='production')
+
+
+@blueprint.route('/emails/welcome.send')
+@login_required
+def emails_welcome_send():
+    from cloud import email
+    email.queue_welcome_mail(current_user)
+    return f'queued mail to {current_user.email}'
+
+
+@blueprint.route('/emails/welcome.html')
+@login_required
+def emails_welcome_html():
+    return render_template('emails/welcome.html',
+                           subject='Welcome to Blender Cloud',
+                           user=current_user)
+
+
+@blueprint.route('/emails/welcome.txt')
+@login_required
+def emails_welcome_txt():
+    txt = render_template('emails/welcome.txt',
+                          subject='Welcome to Blender Cloud',
+                          user=current_user)
+    return flask.Response(txt, content_type='text/plain; charset=utf-8')
+
+
+@blueprint.route('/p/<project_url>')
+def project_landing(project_url):
+    """Override Pillar project_view endpoint completely.
+
+    The first part of the function is identical to the one in Pillar, but the
+    second part (starting with 'Load custom project properties') extends the
+    behaviour to support film project landing pages.
+    """
+
+    template_name = None
+    if request.args.get('format') == 'jstree':
+        log.warning('projects.view(%r) endpoint called with format=jstree, '
+                    'redirecting to proper endpoint. URL is %s; referrer is %s',
+                    project_url, request.url, request.referrer)
+        return redirect(url_for('projects.jstree', project_url=project_url))
+
+    api = system_util.pillar_api()
+    project = find_project_or_404(project_url,
+                                  embedded={'header_node': 1},
+                                  api=api)
+
+    # Load the header video file, if there is any.
+    header_video_file = None
+    header_video_node = None
+    if project.header_node and project.header_node.node_type == 'asset' and \
+            project.header_node.properties.content_type == 'video':
+        header_video_node = project.header_node
+        header_video_file = get_file(project.header_node.properties.file)
+        header_video_node.picture = get_file(header_video_node.picture)
+
+    extra_context = {
+        'header_video_file': header_video_file,
+        'header_video_node': header_video_node}
+
+    # Load custom project properties. If the project has a 'cloud' extension prop,
+    # render it using the projects/landing.html template and try to attach a
+    # number of additional attributes (pages, images, etc.).
+    if 'extension_props' in project and EXTENSION_NAME in project['extension_props']:
+        extension_props = project['extension_props'][EXTENSION_NAME]
+        extension_props['logo'] = get_file(extension_props['logo'])
+
+        pages = Node.all({
+            'where': {
+                'project': project._id,
+                'node_type': 'page',
+                '_deleted': {'$ne': True}},
+            'projection': {'name': 1}}, api=api)
+
+        extra_context.update({'pages': pages._items})
+        template_name = 'projects/landing.html'
+
+    return render_project(project, api,
+                          extra_context=extra_context,
+                          template_name=template_name)
+
+
+@blueprint.route('/p/<project_url>/browse')
+@project_view()
+def project_browse(project: pillarsdk.Project):
+    """Project view displaying all top-level nodes.
+
+    We render a regular project view, but we introduce an additional template
+    variable: browse. By doing that we prevent the regular project view
+    from loading and fetch via AJAX a "group" node-like view instead (see
+    project_browse_view_nodes).
+    """
+    return render_template(
+        'projects/view.html',
+        api=system_util.pillar_api(),
+        project=project,
+        node=None,
+        show_project=True,
+        browse=True,
+        og_picture=None,
+        navigation_links=project_navigation_links(project, system_util.pillar_api()),
+        extension_sidebar_links=current_app.extension_sidebar_links(project))
+
+
+@blueprint.route('/p/<project_url>/browse/nodes')
+@project_view()
+def project_browse_view_nodes(project: pillarsdk.Project):
+    """Display top-level nodes for a Project.
+
+    This view is always meant to be served embedded, as part of project_browse.
+    """
+    api = system_util.pillar_api()
+    # Get top level nodes
+    projection = {
+        'project': 1,
+        'name': 1,
+        'picture': 1,
+        'node_type': 1,
+        'properties.order': 1,
+        'properties.status': 1,
+        'user': 1,
+        'properties.content_type': 1,
+        'permissions.world': 1}
+    where = {
+        'project': project['_id'],
+        'parent': {'$exists': False},
+        'properties.status': 'published',
+        '_deleted': {'$ne': True},
+        'node_type': {'$in': ['group', 'asset']},
+    }
+
+    try:
+        nodes = Node.all({
+            'projection': projection,
+            'where': where,
+            'sort': [('properties.order', 1), ('name', 1)]}, api=api)
+    except pillarsdk.exceptions.ForbiddenAccess:
+        return render_template('errors/403_embed.html')
+    nodes = nodes._items
+
+    for child in nodes:
+        child.picture = get_file(child.picture, api=api)
+    return render_template(
+        'projects/browse_embed.html',
+        nodes=nodes)
+
+
+def project_settings(project: pillarsdk.Project, **template_args: dict):
+    """Renders the project settings page for Blender Cloud projects.
+
+    If the project has been setup for Blender Cloud, check for the cloud.category
+    property, to render the proper form.
+    """
+
+    # Based on the project state, we can render a different template.
+    if not current_cloud.is_cloud_project(project):
+        return render_template('project_settings/offer_setup.html',
+                               project=project, **template_args)
+
+    cloud_props = project['extension_props'][EXTENSION_NAME]
+
+    category = cloud_props['category']
+    if category != 'film':
+        log.error('No interface available to edit %s projects, yet' % category)
+
+    form = FilmProjectForm()
+
+    # Iterate over the form fields and set the data if exists in the project document
+    for field_name in form.data:
+        if field_name not in cloud_props:
+            continue
+        # Skip csrf_token field
+        if field_name == 'csrf_token':
+            continue
+        form_field = getattr(form, field_name)
+        form_field.data = cloud_props[field_name]
+
+    return render_template('project_settings/settings.html',
+                           project=project,
+                           form=form,
+                           **template_args)
+
+
+@blueprint.route('/<project_url>/settings/film', methods=['POST'])
+@authorization.require_login(require_cap='admin')
+@project_view()
+def save_film_settings(project: pillarsdk.Project):
+    # Ensure that the project is setup for Cloud (see @attract_project_view for example)
+    form = FilmProjectForm()
+    if not form.validate_on_submit():
+        log.debug('Form submission failed')
+        # Return list of validation errors
+
+    updated_extension_props = {}
+    for field_name in form.data:
+        # Skip csrf_token field
+        if field_name == 'csrf_token':
+            continue
+        form_field = getattr(form, field_name)
+        # TODO(fsiddi) if form_field type is FileSelectField, convert it to ObjectId
+        # Currently this raises TypeError: Object of type 'ObjectId' is not JSON serializable
+
+        if form_field.data == '':
+            form_field.data = None
+        updated_extension_props[field_name] = form_field.data
+
+    # Update extension props and save project
+    extension_props = project['extension_props'][EXTENSION_NAME]
+    # Project is a Resource, so we update properties iteratively
+    for k, v in updated_extension_props.items():
+        extension_props[k] = v
+    project.update(api=system_util.pillar_api())
+    return '', 204
+
+
+@blueprint.route('/<project_url>/setup-for-film', methods=['POST'])
+@login_required
+@project_view()
+def setup_for_film(project: pillarsdk.Project):
+    import cloud.setup
+
+    project_id = project._id
+
+    if not project.has_method('PUT'):
+        log.warning('User %s tries to set up project %s for Blender Cloud, but has no PUT rights.',
+                    current_user, project_id)
+        raise wz_exceptions.Forbidden()
+
+    log.info('User %s sets up project %s for Blender Cloud', current_user, project_id)
+    cloud.setup.setup_for_film(project.url)
+
+    return '', 204
+
+
+def setup_app(app):
+    global _homepage_context
+    cached = app.cache.cached(timeout=300)
+    _homepage_context = cached(_homepage_context)

cloud/setup.py

@@ -0,0 +1,54 @@
+"""Setting up projects for Blender Cloud."""
+
+import logging
+
+from bson import ObjectId
+from eve.methods.put import put_internal
+from flask import current_app
+
+from pillar.api.utils import remove_private_keys
+from . import EXTENSION_NAME
+
+log = logging.getLogger(__name__)
+
+
+def setup_for_film(project_url):
+    """Add Blender Cloud extension_props specific for film projects.
+
+    Returns the updated project.
+    """
+
+    projects_collection = current_app.data.driver.db['projects']
+
+    # Find the project in the database.
+    project = projects_collection.find_one({'url': project_url})
+    if not project:
+        raise RuntimeError('Project %s does not exist.' % project_url)
+
+    # Set default extension properties. Be careful not to overwrite any properties that
+    # are already there.
+    all_extension_props = project.setdefault('extension_props', {})
+    cloud_extension_props = {
+        'category': 'film',
+        'theme_css': '',
+        # The accent color (can be 'blue' or '#FFBBAA' or 'rgba(1, 1, 1, 1)
+        'theme_color': '',
+        'is_in_production': False,
+        'video_url': '',  # Oembeddable url
+        'poster': None,  # File ObjectId
+        'logo': None,  # File ObjectId
+        # TODO(fsiddi) when we introduce other setup_for_* in Blender Cloud, make available
+        # at a higher scope
+        'is_featured': False,
+    }
+
+    all_extension_props.setdefault(EXTENSION_NAME, cloud_extension_props)
+
+    project_id = ObjectId(project['_id'])
+    project = remove_private_keys(project)
+    result, _, _, status_code = put_internal('projects', project, _id=project_id)
+
+    if status_code != 200:
+        raise RuntimeError("Can't update project %s, issues: %s", project_id, result)
+
+    log.info('Project %s was updated for Blender Cloud.', project_url)

cloud/stats/__init__.py

@@ -0,0 +1,89 @@
+"""Interesting usage metrics"""
+
+from flask import current_app
+
+
+def count_nodes(query=None) -> int:
+    pipeline = [
+        {'$match': {'_deleted': {'$ne': 'true'}}},
+        {
+            '$lookup':
+                {
+                    'from': "projects",
+                    'localField': "project",
+                    'foreignField': "_id",
+                    'as': "project",
+                }
+        },
+        {
+            '$unwind':
+                {
+                    'path': '$project',
+                }
+        },
+        {
+            '$project':
+                {
+                    'project.is_private': 1,
+                }
+        },
+        {'$match': {'project.is_private': False}},
+        {'$count': 'tot'}
+    ]
+    c = current_app.db()['nodes']
+    # If we provide a query, we extend the first $match step in the aggregation pipeline with
+    # with the extra parameters (for example node_type)
+    if query:
+        pipeline[0]['$match'].update(query)
+    # Return either a list with one item or an empty list
+    r = list(c.aggregate(pipeline=pipeline))
+    count = 0 if not r else r[0]['tot']
+    return count
+
+
+def count_users(query=None) -> int:
+    u = current_app.db()['users']
+    return u.count(query)
+
+
+def count_blender_sync(query=None) -> int:
+    pipeline = [
+        # 0 Find all startups.blend that are not deleted
+        {
+            '$match': {
+                '_deleted': {'$ne': 'true'},
+                'name': 'startup.blend',
+            }
+        },
+        # 1 Group them per project (drops any duplicates)
+        {'$group': {'_id': '$project'}},
+        # 2 Join the project info
+        {
+            '$lookup':
+                {
+                    'from': "projects",
+                    'localField': "_id",
+                    'foreignField': "_id",
+                    'as': "project",
+                }
+        },
+        # 3 Unwind the project list (there is always only one project)
+        {
+            '$unwind':
+                {
+                    'path': '$project',
+                }
+        },
+        # 4 Find all home projects
+        {'$match': {'project.category': 'home'}},
+        {'$count': 'tot'}
+    ]
+    c = current_app.db()['nodes']
+    # If we provide a query, we extend the first $match step in the aggregation pipeline with
+    # with the extra parameters (for example _created)
+    if query:
+        pipeline[0]['$match'].update(query)
+    # Return either a list with one item or an empty list
+    r = list(c.aggregate(pipeline=pipeline))
+    count = 0 if not r else r[0]['tot']
+    return count

cloud/stats/routes.py

@@ -0,0 +1,57 @@
+import logging
+import datetime
+import functools
+
+from flask import Blueprint, jsonify
+from cloud.stats import count_nodes, count_users, count_blender_sync
+
+blueprint = Blueprint('cloud.stats', __name__, url_prefix='/s')
+
+log = logging.getLogger(__name__)
+
+
+@functools.lru_cache()
+def get_stats(before: datetime.datetime):
+    query_comments = {'node_type': 'comment'}
+    query_assets = {'node_type': 'asset'}
+
+    date_query = {}
+
+    if before:
+        date_query = {'_created': {'$lt': before}}
+        query_comments.update(date_query)
+        query_assets.update(date_query)
+
+    stats = {
+        'comments': count_nodes(query_comments),
+        'assets': count_nodes(query_assets),
+        'users_total': count_users(date_query),
+        'users_blender_sync': count_blender_sync(date_query),
+    }
+    return stats
+
+
+@blueprint.route('/')
+@blueprint.route('/before/<int:before>')
+def index(before: int=0):
+    """
+    This endpoint is queried on a daily basis by grafista to retrieve cloud usage
+    stats. For assets and comments we take into considerations only those who belong
+    to public projects.
+
+    These is the data we retrieve
+
+    - Comments count
+    - Assets count (video, images and files)
+    - Users count (subscribers count goes via store)
+    - Blender Sync users
+    """
+
+    # TODO: Implement project-level metrics (and update ad every child update)
+    if before:
+        before = datetime.datetime.strptime(str(before), '%Y%m%d')
+    else:
+        today = datetime.date.today()
+        before = datetime.datetime(today.year, today.month, today.day)
+
+    return jsonify(get_stats(before))

cloud/store.py

@@ -0,0 +1,72 @@
+"""Blender Store interface."""
+
+import logging
+import typing
+
+from pillar import current_app
+
+log = logging.getLogger(__name__)
+
+
+def fetch_subscription_info(email: str) -> typing.Optional[dict]:
+    """Returns the user info dict from the external subscriptions management server.
+
+    :returns: the store user info, or None if the user can't be found or there
+        was an error communicating. A dict like this is returned:
+        {
+            "shop_id": 700,
+            "cloud_access": 1,
+            "paid_balance": 314.75,
+            "balance_currency": "EUR",
+            "start_date": "2014-08-25 17:05:46",
+            "expiration_date": "2016-08-24 13:38:45",
+            "subscription_status": "wc-active",
+            "expiration_date_approximate": true
+        }
+    """
+
+    from requests.adapters import HTTPAdapter
+    import requests.exceptions
+
+    external_subscriptions_server = current_app.config['EXTERNAL_SUBSCRIPTIONS_MANAGEMENT_SERVER']
+
+    if log.isEnabledFor(logging.DEBUG):
+        import urllib.parse
+
+        log_email = urllib.parse.quote(email)
+        log.debug('Connecting to store at %s?blenderid=%s',
+                  external_subscriptions_server, log_email)
+
+    # Retry a few times when contacting the store.
+    s = requests.Session()
+    s.mount(external_subscriptions_server, HTTPAdapter(max_retries=5))
+
+    try:
+        r = s.get(external_subscriptions_server,
+                  params={'blenderid': email},
+                  verify=current_app.config['TLS_CERT_FILE'],
+                  timeout=current_app.config.get('EXTERNAL_SUBSCRIPTIONS_TIMEOUT_SECS', 10))
+    except requests.exceptions.ConnectionError as ex:
+        log.error('Error connecting to %s: %s', external_subscriptions_server, ex)
+        return None
+    except requests.exceptions.Timeout as ex:
+        log.error('Timeout communicating with %s: %s', external_subscriptions_server, ex)
+        return None
+    except requests.exceptions.RequestException as ex:
+        log.error('Some error communicating with %s: %s', external_subscriptions_server, ex)
+        return None
+
+    if r.status_code != 200:
+        log.warning("Error communicating with %s, code=%i, unable to check "
+                    "subscription status of user %s",
+                    external_subscriptions_server, r.status_code, email)
+        return None
+
+    store_user = r.json()
+
+    if log.isEnabledFor(logging.DEBUG):
+        import json
+        log.debug('Received JSON from store API: %s',
+                  json.dumps(store_user, sort_keys=False, indent=4))
+
+    return store_user

cloud/tagged/__init__.py

@@ -0,0 +1,3 @@
+"""Routes for fetching tagged assets."""
+
+

cloud/tagged/routes.py

@@ -0,0 +1,16 @@
+import logging
+import datetime
+import functools
+
+from flask import Blueprint, jsonify
+
+blueprint = Blueprint('cloud.tagged', __name__, url_prefix='/tagged')
+
+log = logging.getLogger(__name__)
+
+
+@blueprint.route('/')
+def index():
+    """Return all tagged assets as JSON, grouped by tag."""
+
+

cloud/webhooks.py

@@ -0,0 +1,263 @@
+"""Blender ID webhooks."""
+
+import functools
+import hashlib
+import hmac
+import json
+import logging
+import typing
+
+from flask import Blueprint, request
+import werkzeug.exceptions as wz_exceptions
+
+from pillar import current_app
+from pillar.api.blender_cloud import subscription
+from pillar.api.utils.authentication import create_new_user_document, make_unique_username
+from pillar.auth import UserClass
+
+blueprint = Blueprint('cloud-webhooks', __name__)
+log = logging.getLogger(__name__)
+WEBHOOK_MAX_BODY_SIZE = 1024 * 10  # 10 kB is large enough for
+
+
+def webhook_payload(hmac_secret: str) -> dict:
+    """Obtains the webhook payload from the request, verifying its HMAC.
+
+    :returns the webhook payload as dictionary.
+    """
+    # Check the content type
+    if request.content_type != 'application/json':
+        log.info('request from %s to %s had bad content type %s',
+                 request.remote_addr, request, request.content_type)
+        raise wz_exceptions.BadRequest('Content type not supported')
+
+    # Check the length of the body
+    if request.content_length > WEBHOOK_MAX_BODY_SIZE:
+        raise wz_exceptions.BadRequest('Request too large')
+    body = request.get_data()
+    if len(body) > request.content_length:
+        raise wz_exceptions.BadRequest('Larger body than Content-Length header')
+
+    # Validate the request
+    mac = hmac.new(hmac_secret.encode(), body, hashlib.sha256)
+    req_hmac = request.headers.get('X-Webhook-HMAC', '')
+    our_hmac = mac.hexdigest()
+    if not hmac.compare_digest(req_hmac, our_hmac):
+        log.info('request from %s to %s had bad HMAC %r, expected %r',
+                 request.remote_addr, request, req_hmac, our_hmac)
+        raise wz_exceptions.BadRequest('Bad HMAC')
+
+    try:
+        return json.loads(body)
+    except json.JSONDecodeError as ex:
+        log.warning('request from %s to %s had bad JSON: %s',
+                    request.remote_addr, request, ex)
+        raise wz_exceptions.BadRequest('Bad JSON')
+
+
+def score(wh_payload: dict, user: dict) -> int:
+    """Determine how likely it is that this is the correct user to modify.
+
+    :param wh_payload: the info we received from Blender ID;
+        see user_modified()
+    :param user: the user in our database
+    :return: the score for this user
+    """
+
+    bid_str = str(wh_payload['id'])
+    try:
+        match_on_bid = any((auth['provider'] == 'blender-id' and auth['user_id'] == bid_str)
+                           for auth in user['auth'])
+    except KeyError:
+        match_on_bid = False
+
+    match_on_old_email = user.get('email', 'none') == wh_payload.get('old_email', 'nothere')
+    match_on_new_email = user.get('email', 'none') == wh_payload.get('email', 'nothere')
+    return match_on_bid * 10 + match_on_old_email + match_on_new_email * 2
+
+
+def insert_or_fetch_user(wh_payload: dict) -> typing.Optional[dict]:
+    """Fetch the user from the DB or create it.
+
+    Only creates it if the webhook payload indicates they could actually use
+    Blender Cloud (i.e. demo or subscriber). This prevents us from creating
+    Cloud accounts for Blender Network users.
+
+    :returns the user document, or None when not created.
+    """
+
+    users_coll = current_app.db('users')
+    my_log = log.getChild('insert_or_fetch_user')
+
+    bid_str = str(wh_payload['id'])
+    email = wh_payload['email']
+
+    # Find the user by their Blender ID, or any of their email addresses.
+    # We use one query to find all matching users. This is done as a
+    # consistency check; if more than one user is returned, we know the
+    # database is inconsistent with Blender ID and can emit a warning
+    # about this.
+    query = {'$or': [
+        {'auth.provider': 'blender-id', 'auth.user_id': bid_str},
+        {'email': {'$in': [wh_payload['old_email'], email]}},
+    ]}
+    db_users = list(users_coll.find(query))
+    user_count = len(db_users)
+    if user_count > 1:
+        # Now we have to pay the price for finding users in one query; we
+        # have to prioritise them and return the one we think is most reliable.
+        calc_score = functools.partial(score, wh_payload)
+        best_score = max(db_users, key=calc_score)
+
+        my_log.error('%d users found for query %s, picking user %s (%s)',
+                     user_count, query, best_score['_id'], best_score['email'])
+        return best_score
+    if user_count:
+        db_user = db_users[0]
+        my_log.debug('found user %s', db_user['email'])
+        return db_user
+
+    if wh_payload.get('date_deletion_requested'):
+        my_log.info('Received update for a deleted user %s, not creating', bid_str)
+        return None
+
+    # Pretend to create the user, so that we can inspect the resulting
+    # capabilities. This is more future-proof than looking at the list
+    # of roles in the webhook payload.
+    username = make_unique_username(email)
+    user_doc = create_new_user_document(email, bid_str, username,
+                                        provider='blender-id',
+                                        full_name=wh_payload['full_name'])
+
+    # Figure out the user's eventual roles. These aren't stored in the document yet,
+    # because that's handled by the badger service.
+    eventual_roles = [subscription.ROLES_BID_TO_PILLAR[r]
+                      for r in wh_payload.get('roles', [])
+                      if r in subscription.ROLES_BID_TO_PILLAR]
+    user_ob = UserClass.construct('', user_doc)
+    user_ob.roles = eventual_roles
+    user_ob.collect_capabilities()
+    create = (user_ob.has_cap('subscriber') or
+              user_ob.has_cap('can-renew-subscription') or
+              current_app.org_manager.user_is_unknown_member(email))
+    if not create:
+        my_log.info('Received update for unknown user %r without Cloud access (caps=%s)',
+                    wh_payload['old_email'], user_ob.capabilities)
+        return None
+
+    # Actually create the user in the database.
+    r, _, _, status = current_app.post_internal('users', user_doc)
+    if status != 201:
+        my_log.error('unable to create user %s: : %r %r', email, status, r)
+        raise wz_exceptions.InternalServerError('unable to create user')
+
+    user_doc.update(r)
+    my_log.info('created user %r = %s to allow immediate Cloud access', email, user_doc['_id'])
+    return user_doc
+
+
+@blueprint.route('/user-modified', methods=['POST'])
+def user_modified():
+    """Update the local user based on the info from Blender ID.
+
+    If the payload indicates the user has access to Blender Cloud (or at least
+    a renewable subscription), create the user if not already in our DB.
+
+    The payload we expect is a dictionary like:
+    {'id': 12345,  # the user's ID in Blender ID
+     'old_email': 'old@example.com',
+     'full_name': 'Harry',
+     'email': 'new@example'com,
+     'avatar_changed': True,
+     'roles': ['role1', 'role2', …]}
+    """
+    my_log = log.getChild('user_modified')
+    my_log.debug('Received request from %s', request.remote_addr)
+
+    hmac_secret = current_app.config['BLENDER_ID_WEBHOOK_USER_CHANGED_SECRET']
+    payload = webhook_payload(hmac_secret)
+
+    my_log.info('payload: %s', payload)
+
+    # Update the user
+    db_user = insert_or_fetch_user(payload)
+    if not db_user:
+        my_log.info('Received update for unknown user %r', payload['old_email'])
+        return '', 204
+
+    if payload.get('date_deletion_requested'):
+        delete_user(db_user, payload)
+        return '', 204
+
+    # Use direct database updates to change the email and full name.
+    # Also updates the db_user dict so that local_user below will have
+    # the updated information.
+    updates = {}
+    if db_user['email'] != payload['email']:
+        my_log.info('User changed email from %s to %s', payload['old_email'], payload['email'])
+        updates['email'] = payload['email']
+        db_user['email'] = payload['email']
+
+    if db_user['full_name'] != payload['full_name']:
+        my_log.info('User changed full name from %r to %r',
+                    db_user['full_name'], payload['full_name'])
+        if payload['full_name']:
+            updates['full_name'] = payload['full_name']
+        else:
+            # Fall back to the username when the full name was erased.
+            updates['full_name'] = db_user['username']
+        db_user['full_name'] = updates['full_name']
+
+    if payload.get('avatar_changed'):
+        import pillar.celery.avatar
+        my_log.info('User %s changed avatar, scheduling download', db_user['_id'])
+        pillar.celery.avatar.sync_avatar_for_user.delay(str(db_user['_id']))
+
+    if updates:
+        users_coll = current_app.db('users')
+        update_res = users_coll.update_one({'_id': db_user['_id']},
+                                           {'$set': updates})
+        if update_res.matched_count != 1:
+            my_log.error('Unable to find user %s to update, even though '
+                         'we found them by email address %s',
+                         db_user['_id'], payload['old_email'])
+
+    # Defer to Pillar to do the role updates.
+    local_user = UserClass.construct('', db_user)
+    subscription.do_update_subscription(local_user, payload)
+
+    return '', 204
+
+
+def delete_user(db_user, payload):
+    """Handle deletion request coming from BID."""
+    my_log = log.getChild('delete_user')
+    date_deletion_requested = payload['date_deletion_requested']
+    bid_str = str(payload['id'])
+    local_id = db_user['_id']
+    my_log.info(
+        'User %s with BID=%s requested deletion on %s, soft-deleting the user',
+        local_id, bid_str, date_deletion_requested,
+    )
+    # Delete all session tokens linked to this user
+    token_coll = current_app.db('tokens')
+    delete_res = token_coll.delete_many({'user': local_id})
+    my_log.info('Deleted %s session tokens of user %s', delete_res.deleted_count, local_id)
+
+    # Soft-delete the user and clear their PII
+    users_coll = current_app.db('users')
+    updates = {
+        '_deleted': True,
+        'email': None,
+        'full_name': None,
+        'username': None,
+        'auth': [],
+    }
+    update_res = users_coll.update_one({'_id': local_id}, {'$set': updates})
+    if update_res.matched_count != 1:
+        my_log.error(
+            'Soft-deleted %s users %s with BID=%s',
+            update_res.matched_count, local_id, bid_str,
+        )
+    else:
+        my_log.warning('Soft-deleted user %s with BID=%s', local_id, bid_str)

cloud_share_img.py

@@ -0,0 +1,296 @@
+#!/usr/bin/env python3
+
+from __future__ import print_function
+
+"""CLI command for sharing an image via Blender Cloud.
+
+Assumes that you are logged in on Blender ID with the Blender ID Add-on.
+
+The user_config_dir and user_data_dir functions come from
+ https://github.com/ActiveState/appdirs/blob/master/appdirs.py and
+ are licensed under the MIT license.
+"""
+
+import argparse
+import json
+import mimetypes
+import os.path
+import pprint
+import sys
+import webbrowser
+
+from urllib.parse import urljoin
+
+import requests
+
+cli = argparse.Namespace()  # CLI args from argparser
+sess = requests.Session()
+IMAGE_SHARING_GROUP_NODE_NAME = 'Image sharing'
+
+if sys.platform.startswith('java'):
+    import platform
+
+    os_name = platform.java_ver()[3][0]
+    if os_name.startswith('Windows'):  # "Windows XP", "Windows 7", etc.
+        system = 'win32'
+    elif os_name.startswith('Mac'):  # "Mac OS X", etc.
+        system = 'darwin'
+    else:  # "Linux", "SunOS", "FreeBSD", etc.
+        # Setting this to "linux2" is not ideal, but only Windows or Mac
+        # are actually checked for and the rest of the module expects
+        # *sys.platform* style strings.
+        system = 'linux2'
+else:
+    system = sys.platform
+
+
+def request(method: str, rel_url: str, **kwargs) -> requests.Response:
+    kwargs.setdefault('auth', (cli.token, ''))
+    url = urljoin(cli.server_url, rel_url)
+    return sess.request(method, url, **kwargs)
+
+
+def get(rel_url: str, **kwargs) -> requests.Response:
+    return request('GET', rel_url, **kwargs)
+
+
+def post(rel_url: str, **kwargs) -> requests.Response:
+    return request('POST', rel_url, **kwargs)
+
+
+def find_user_id() -> str:
+    """Returns the current user ID."""
+
+    print(15 * '=', 'User info', 15 * '=')
+    resp = get('/api/users/me')
+    resp.raise_for_status()
+
+    user_info = resp.json()
+    print('You are logged in as %(full_name)s (%(_id)s)' % user_info)
+
+    return user_info['_id']
+
+
+def find_home_project_id() -> dict:
+    resp = get('/api/bcloud/home-project')
+    resp.raise_for_status()
+
+    proj = resp.json()
+    proj_id = proj['_id']
+    print('Your home project ID is %s' % proj_id)
+    return proj_id
+
+
+def find_image_sharing_group_id(home_project_id, user_id) -> str:
+    """Find the top-level image sharing group node."""
+
+    node_doc = {
+        'project': home_project_id,
+        'node_type': 'group',
+        'name': IMAGE_SHARING_GROUP_NODE_NAME,
+        'user': user_id,
+    }
+
+    resp = get('/api/nodes', params={'where': json.dumps(node_doc)})
+    resp.raise_for_status()
+    items = resp.json()['_items']
+
+    if not items:
+        print('Share group not found, creating one.')
+        node_doc.update({
+            'properties': {},
+        })
+        resp = post('/api/nodes', json=node_doc)
+        resp.raise_for_status()
+        share_group = resp.json()
+    else:
+        share_group = items[0]
+
+    # print('Share group:', share_group)
+    return share_group['_id']
+
+
+def upload_image():
+    user_id = find_user_id()
+    home_project_id = find_home_project_id()
+    group_id = find_image_sharing_group_id(home_project_id, user_id)
+    basename = os.path.basename(cli.imgfile)
+    print('Sharing group ID is %s' % group_id)
+
+    # Upload the image to the project.
+    print('Uploading %r' % cli.imgfile)
+    mimetype, _ = mimetypes.guess_type(cli.imgfile, strict=False)
+    with open(cli.imgfile, mode='rb') as infile:
+        resp = post('api/storage/stream/%s' % home_project_id,
+                    files={'file': (basename, infile, mimetype)})
+    resp.raise_for_status()
+    file_upload_resp = resp.json()
+    file_upload_status = file_upload_resp.get('_status') or file_upload_resp.get('status')
+    if file_upload_status != 'ok':
+        raise ValueError('Received bad status %s from Pillar: %s' %
+                         (file_upload_status, json.dumps(file_upload_resp)))
+    file_id = file_upload_resp['file_id']
+    print('File ID is', file_id)
+
+    # Create the asset node
+    asset_node = {
+        'project': home_project_id,
+        'node_type': 'asset',
+        'name': basename,
+        'parent': group_id,
+        'properties': {
+            'content_type': mimetype,
+            'file': file_id,
+        },
+    }
+    resp = post('api/nodes', json=asset_node)
+    resp.raise_for_status()
+    node_info = resp.json()
+    node_id = node_info['_id']
+    print('Created asset node', node_id)
+
+    # Share the node to get a public URL.
+    resp = post('api/nodes/%s/share' % node_id)
+    resp.raise_for_status()
+    share_info = resp.json()
+    print(json.dumps(share_info, indent=4))
+
+    url = share_info.get('short_link')
+    print('Opening %s in a browser' % url)
+    webbrowser.open_new_tab(url)
+
+
+def find_credentials():
+    """Finds BlenderID credentials.
+
+    :rtype: str
+    :returns: the authentication token to use.
+    """
+    import glob
+
+    # Find BlenderID profile file.
+    configpath = user_config_dir('blender', 'Blender Foundation', roaming=True)
+    found = glob.glob(os.path.join(configpath, '*'))
+    for confpath in reversed(sorted(found)):
+        profiles_path = os.path.join(confpath, 'config', 'blender_id', 'profiles.json')
+        if not os.path.exists(profiles_path):
+            continue
+
+        print('Reading credentials from %s' % profiles_path)
+        with open(profiles_path) as infile:
+            profiles = json.load(infile, encoding='utf8')
+        if profiles:
+            break
+    else:
+        print('Unable to find Blender ID credentials. Log in with the Blender ID add-on in '
+              'Blender first.')
+        raise SystemExit()
+
+    active_profile = profiles[u'active_profile']
+    profile = profiles[u'profiles'][active_profile]
+    print('Logging in as %s' % profile[u'username'])
+
+    return profile[u'token']
+
+
+def main():
+    global cli
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument('imgfile', help='The image file to share.')
+    parser.add_argument('-u', '--server-url', default='https://cloud.blender.org/',
+                        help='URL of the Flamenco server.')
+    parser.add_argument('-t', '--token',
+                        help='Authentication token to use. If not given, your token from the '
+                             'Blender ID add-on is used.')
+
+    cli = parser.parse_args()
+    if not cli.token:
+        cli.token = find_credentials()
+
+    upload_image()
+
+
+def user_config_dir(appname=None, appauthor=None, version=None, roaming=False):
+    r"""Return full path to the user-specific config dir for this application.
+        "appname" is the name of application.
+            If None, just the system directory is returned.
+        "appauthor" (only used on Windows) is the name of the
+            appauthor or distributing body for this application. Typically
+            it is the owning company name. This falls back to appname. You may
+            pass False to disable it.
+        "version" is an optional version path element to append to the
+            path. You might want to use this if you want multiple versions
+            of your app to be able to run independently. If used, this
+            would typically be "<major>.<minor>".
+            Only applied when appname is present.
+        "roaming" (boolean, default False) can be set True to use the Windows
+            roaming appdata directory. That means that for users on a Windows
+            network setup for roaming profiles, this user data will be
+            sync'd on login. See
+            <http://technet.microsoft.com/en-us/library/cc766489(WS.10).aspx>
+            for a discussion of issues.
+    Typical user config directories are:
+        Mac OS X:               same as user_data_dir
+        Unix:                   ~/.config/<AppName>     # or in $XDG_CONFIG_HOME, if defined
+        Win *:                  same as user_data_dir
+    For Unix, we follow the XDG spec and support $XDG_CONFIG_HOME.
+    That means, by default "~/.config/<AppName>".
+    """
+    if system in {"win32", "darwin"}:
+        path = user_data_dir(appname, appauthor, None, roaming)
+    else:
+        path = os.getenv('XDG_CONFIG_HOME', os.path.expanduser("~/.config"))
+        if appname:
+            path = os.path.join(path, appname)
+    if appname and version:
+        path = os.path.join(path, version)
+    return path
+
+
+def user_data_dir(appname=None, appauthor=None, version=None, roaming=False):
+    r"""Return full path to the user-specific data dir for this application.
+        "appname" is the name of application.
+            If None, just the system directory is returned.
+        "appauthor" (only used on Windows) is the name of the
+            appauthor or distributing body for this application. Typically
+            it is the owning company name. This falls back to appname. You may
+            pass False to disable it.
+        "version" is an optional version path element to append to the
+            path. You might want to use this if you want multiple versions
+            of your app to be able to run independently. If used, this
+            would typically be "<major>.<minor>".
+            Only applied when appname is present.
+        "roaming" (boolean, default False) can be set True to use the Windows
+            roaming appdata directory. That means that for users on a Windows
+            network setup for roaming profiles, this user data will be
+            sync'd on login. See
+            <http://technet.microsoft.com/en-us/library/cc766489(WS.10).aspx>
+            for a discussion of issues.
+    Typical user data directories are:
+        Mac OS X:               ~/Library/Application Support/<AppName>
+        Unix:                   ~/.local/share/<AppName>    # or in $XDG_DATA_HOME, if defined
+        Win XP (not roaming):   C:\Documents and Settings\<username>\Application Data\<AppAuthor>\<AppName>
+        Win XP (roaming):       C:\Documents and Settings\<username>\Local Settings\Application Data\<AppAuthor>\<AppName>
+        Win 7  (not roaming):   C:\Users\<username>\AppData\Local\<AppAuthor>\<AppName>
+        Win 7  (roaming):       C:\Users\<username>\AppData\Roaming\<AppAuthor>\<AppName>
+    For Unix, we follow the XDG spec and support $XDG_DATA_HOME.
+    That means, by default "~/.local/share/<AppName>".
+    """
+    if system == "win32":
+        raise RuntimeError("Sorry, Windows is not supported for now.")
+    elif system == 'darwin':
+        path = os.path.expanduser('~/Library/Application Support/')
+        if appname:
+            path = os.path.join(path, appname)
+    else:
+        path = os.getenv('XDG_DATA_HOME', os.path.expanduser("~/.local/share"))
+        if appname:
+            path = os.path.join(path, appname)
+    if appname and version:
+        path = os.path.join(path, version)
+    return path
+
+
+if __name__ == '__main__':
+    main()

config_local.example.py

@@ -0,0 +1,43 @@
+import os
+
+DEBUG = True
+
+BLENDER_ID_ENDPOINT = 'http://id.local:8000/'
+
+SERVER_NAME = 'cloud.local:5001'
+SCHEME = 'http'
+PILLAR_SERVER_ENDPOINT = f'{SCHEME}://{SERVER_NAME}/api/'
+
+os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = 'true'
+os.environ['PILLAR_MONGO_DBNAME'] = 'cloud'
+os.environ['PILLAR_MONGO_PORT'] = '27017'
+os.environ['PILLAR_MONGO_HOST'] = 'mongo'
+
+os.environ['PILLAR_SERVER_ENDPOINT'] = PILLAR_SERVER_ENDPOINT
+
+SECRET_KEY = '##DEFINE##'
+
+OAUTH_CREDENTIALS = {
+    'blender-id': {
+        'id': 'CLOUD-OF-SNOWFLAKES-42',
+        'secret': '##DEFINE##',
+    }
+}
+
+MAIN_PROJECT_ID = '##DEFINE##'
+URLER_SERVICE_AUTH_TOKEN = '##DEFINE##'
+
+ZENCODER_API_KEY = '##DEFINE##'
+ZENCODER_NOTIFICATIONS_SECRET = '##DEFINE##'
+ZENCODER_NOTIFICATIONS_URL = 'http://zencoderfetcher/'
+
+# Special announcement on top of every page, for non-subscribers.
+# category: 'string', can be 'info', 'warning', 'danger', or 'success'.
+# message: 'string', any text, it gets markdowned.
+# icon: 'string', any icon in font-pillar. e.g. 'pi-heart-filled'
+UI_ANNOUNCEMENT_NON_SUBSCRIBERS = {
+    'category': 'danger',
+    'message': 'Spring will swing away the gray clouds, until then, '
+               '[take cover under Blender Cloud](https://cloud.blender.org)!',
+    'icon': 'pi-heart-filled',
+}

deploy.sh

@@ -1,124 +0,0 @@
-#!/bin/bash -e
-
-# Deploys the current production branch to the production machine.
-PROJECT_NAME="blender-cloud"
-DOCKER_NAME="blender_cloud"
-REMOTE_ROOT="/data/git/${PROJECT_NAME}"
-
-SSH="ssh -o ClearAllForwardings=yes cloud.blender.org"
-
-# macOS does not support readlink -f, so we use greadlink instead
-if [[ `uname` == 'Darwin' ]]; then
-    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
-    readlink='greadlink'
-else
-    readlink='readlink'
-fi
-
-ROOT="$(dirname "$($readlink -f "$0")")"
-cd ${ROOT}
-
-# Check that we're on production branch.
-if [ $(git rev-parse --abbrev-ref HEAD) != "production" ]; then
-    echo "You are NOT on the production branch, refusing to deploy." >&2
-    exit 1
-fi
-
-# Check that production branch has been pushed.
-if [ -n "$(git log origin/production..production --oneline)" ]; then
-    echo "WARNING: not all changes to the production branch have been pushed."
-    echo "Press [ENTER] to continue deploying current origin/production, CTRL+C to abort."
-    read dummy
-fi
-
-function find_module()
-{
-    MODULE_NAME=$1
-    MODULE_DIR=$(python <<EOT
-from __future__ import print_function
-import os.path
-try:
-    import ${MODULE_NAME}
-except ImportError:
-    raise SystemExit('${MODULE_NAME} not found on Python path. Are you in the correct venv?')
-
-print(os.path.dirname(os.path.dirname(${MODULE_NAME}.__file__)))
-EOT
-)
-
-    if [ $(git -C $MODULE_DIR rev-parse --abbrev-ref HEAD) != "production" ]; then
-        echo "${MODULE_NAME}: ($MODULE_DIR) NOT on the production branch, refusing to deploy." >&2
-        exit 1
-    fi
-
-    echo $MODULE_DIR
-}
-
-# Find our modules
-PILLAR_DIR=$(find_module pillar)
-ATTRACT_DIR=$(find_module attract)
-FLAMENCO_DIR=$(find_module flamenco)
-
-echo "Pillar  : $PILLAR_DIR"
-echo "Attract : $ATTRACT_DIR"
-echo "Flamenco: $FLAMENCO_DIR"
-
-if [ -z "$PILLAR_DIR" -o -z "$ATTRACT_DIR" -o -z "$FLAMENCO_DIR" ];
-then
-    exit 1
-fi
-
-# SSH to cloud to pull all files in
-function git_pull() {
-    PROJECT_NAME="$1"
-    BRANCH="$2"
-    REMOTE_ROOT="/data/git/${PROJECT_NAME}"
-
-    echo "==================================================================="
-    echo "UPDATING FILES ON ${PROJECT_NAME}"
-    ${SSH} git -C ${REMOTE_ROOT} fetch origin ${BRANCH}
-    ${SSH} git -C ${REMOTE_ROOT} log origin/${BRANCH}..${BRANCH} --oneline
-    ${SSH} git -C ${REMOTE_ROOT} merge --ff-only origin/${BRANCH}
-}
-
-git_pull pillar-python-sdk master
-git_pull pillar production
-git_pull attract production
-git_pull flamenco production
-git_pull blender-cloud production
-
-# Update the virtualenv
-#${SSH} -t docker exec ${DOCKER_NAME} /data/venv/bin/pip install -U -r ${REMOTE_ROOT}/requirements.txt --exists-action w
-
-# RSync the world
-$ATTRACT_DIR/rsync_ui.sh
-$FLAMENCO_DIR/rsync_ui.sh
-./rsync_ui.sh
-
-# Notify Bugsnag of this new deploy.
-echo
-echo "==================================================================="
-GIT_REVISION=$(${SSH} git -C ${REMOTE_ROOT} describe --always)
-echo "Notifying Bugsnag of this new deploy of revision ${GIT_REVISION}."
-BUGSNAG_API_KEY=$(${SSH} python -c "\"import sys; sys.path.append('${REMOTE_ROOT}'); import config_local; print(config_local.BUGSNAG_API_KEY)\"")
-curl --data "apiKey=${BUGSNAG_API_KEY}&revision=${GIT_REVISION}" https://notify.bugsnag.com/deploy
-echo
-
-# Wait for [ENTER] to restart the server
-echo
-echo "==================================================================="
-echo "NOTE: If you want to edit config_local.py on the server, do so now."
-echo "NOTE: Press [ENTER] to continue and restart the server process."
-read dummy
-${SSH} docker exec ${DOCKER_NAME} apache2ctl graceful
-echo "Server process restarted"
-
-echo
-echo "==================================================================="
-echo "Clearing front page from Redis cache."
-${SSH} docker exec redis redis-cli DEL pwview//
-
-echo
-echo "==================================================================="
-echo "Deploy of ${PROJECT_NAME} is done."
-echo "==================================================================="

deploy/2docker.sh

@@ -0,0 +1,130 @@
+#!/bin/bash -e
+
+STAGING_BRANCH=${STAGING_BRANCH:-production}
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [[ `uname` == 'Darwin' ]]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+
+ROOT="$(dirname "$(dirname "$($readlink -f "$0")")")"
+STAGINGDIR="$ROOT/docker/4_run/staging"
+PROJECT_NAME="$(basename $ROOT)"
+
+if [ -e $STAGINGDIR ]; then
+    echo "$STAGINGDIR already exists, press [ENTER] to destroy and re-install, Ctrl+C to abort."
+    read dummy
+    rm -rf $STAGINGDIR
+else
+    echo -n "Installing into $STAGINGDIR… "
+    echo "press [ENTER] to continue, Ctrl+C to abort."
+    read dummy
+fi
+
+cd ${ROOT}
+mkdir -p $STAGINGDIR
+REMOTE_ROOT="$STAGINGDIR/$PROJECT_NAME"
+
+if [ -z "$SKIP_BRANCH_CHECK" ]; then
+    # Check that we're on production branch.
+    if [ $(git rev-parse --abbrev-ref HEAD) != "$STAGING_BRANCH" ]; then
+        echo "You are NOT on the $STAGING_BRANCH branch, refusing to stage." >&2
+        exit 1
+    fi
+
+    # Check that production branch has been pushed.
+    if [ -n "$(git log origin/$STAGING_BRANCH..$STAGING_BRANCH --oneline)" ]; then
+        echo "WARNING: not all changes to the $STAGING_BRANCH branch have been pushed."
+        echo "Press [ENTER] to continue staging current origin/$STAGING_BRANCH, CTRL+C to abort."
+        read dummy
+    fi
+fi
+
+function find_module()
+{
+    MODULE_NAME=$1
+    MODULE_DIR=$(python <<EOT
+from __future__ import print_function
+import os.path
+try:
+    import ${MODULE_NAME}
+except ImportError:
+    raise SystemExit('${MODULE_NAME} not found on Python path. Are you in the correct venv?')
+
+print(os.path.dirname(os.path.dirname(${MODULE_NAME}.__file__)))
+EOT
+)
+    echo $MODULE_DIR
+}
+
+# Find our modules
+echo "==================================================================="
+echo "LOCAL MODULE LOCATIONS"
+PILLAR_DIR=$(find_module pillar)
+ATTRACT_DIR=$(find_module attract)
+FLAMENCO_DIR=$(find_module flamenco)
+SVNMAN_DIR=$(find_module svnman)
+SDK_DIR=$(find_module pillarsdk)
+
+echo "Pillar  : $PILLAR_DIR"
+echo "Attract : $ATTRACT_DIR"
+echo "Flamenco: $FLAMENCO_DIR"
+echo "SVNMan  : $SVNMAN_DIR"
+echo "SDK     : $SDK_DIR"
+
+if [ -z "$PILLAR_DIR" -o -z "$ATTRACT_DIR" -o -z "$FLAMENCO_DIR" -o -z "$SVNMAN_DIR" -o -z "$SDK_DIR" ];
+then
+    exit 1
+fi
+
+function git_clone() {
+    PROJECT_NAME="$1"
+    BRANCH="$2"
+    LOCAL_ROOT="$3"
+
+    echo "==================================================================="
+    echo "CLONING REPO ON $PROJECT_NAME @$BRANCH"
+    URL=$(git -C $LOCAL_ROOT remote get-url origin)
+    git -C $STAGINGDIR clone --depth 1 --branch $BRANCH $URL $PROJECT_NAME
+}
+
+if [ "$STAGING_BRANCH" == "production" ]; then
+    SDK_STAGING_BRANCH=master  # SDK doesn't have a production branch
+else
+    SDK_STAGING_BRANCH=$STAGING_BRANCH
+fi
+
+git_clone pillar-python-sdk $SDK_STAGING_BRANCH $SDK_DIR
+git_clone pillar $STAGING_BRANCH $PILLAR_DIR
+git_clone attract $STAGING_BRANCH $ATTRACT_DIR
+git_clone flamenco $STAGING_BRANCH $FLAMENCO_DIR
+git_clone pillar-svnman $STAGING_BRANCH $SVNMAN_DIR
+git_clone blender-cloud $STAGING_BRANCH $ROOT
+
+# Gulp everywhere
+GULP=$ROOT/node_modules/.bin/gulp
+if [ ! -e $GULP -o gulpfile.js -nt $GULP ]; then
+    npm install
+    touch $GULP  # installer doesn't always touch this after a build, so we do.
+fi
+
+# List of projects
+PROJECTS="pillar attract flamenco pillar-svnman blender-cloud"
+
+# Run ./gulp for every project
+for PROJECT in $PROJECTS; do
+    pushd $STAGINGDIR/$PROJECT; ./gulp --production; popd;
+done
+
+# Remove node_modules (only after all projects with interdependencies have been built)
+for PROJECT in $PROJECTS; do
+    pushd $STAGINGDIR/$PROJECT; rm -r node_modules; popd;
+done
+
+echo
+echo "==================================================================="
+echo "Staging of ${PROJECT_NAME} is ready for dockerisation."
+echo "==================================================================="

deploy/2server.sh

@@ -0,0 +1,80 @@
+#!/bin/bash -e
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [[ `uname` == 'Darwin' ]]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+ROOT="$(dirname "$(dirname "$($readlink -f "$0")")")"
+PROJECT_NAME="$(basename $ROOT)"
+DOCKER_IMAGE="armadillica/blender_cloud:latest"
+REMOTE_SECRET_CONFIG_DIR="/data/config"
+REMOTE_DOCKER_COMPOSE_DIR="/root/docker"
+
+#################################################################################
+case $1 in
+    cloud*)
+        DEPLOYHOST="$1"
+        ;;
+    *)
+        echo "Use $0 cloud{nr}|cloud.blender.org" >&2
+        exit 1
+esac
+SSH_OPTS="-o ClearAllForwardings=yes -o PermitLocalCommand=no"
+SSH="ssh $SSH_OPTS $DEPLOYHOST"
+SCP="scp $SSH_OPTS"
+
+echo -n "Deploying to $DEPLOYHOST… "
+
+if ! ping $DEPLOYHOST -q -c 1 -W 2 >/dev/null; then
+    echo "host $DEPLOYHOST cannot be pinged, refusing to deploy." >&2
+    exit 2
+fi
+
+cat <<EOT
+[ping OK]
+
+Make sure that you have pushed the $DOCKER_IMAGE
+docker image to Docker Hub.
+
+press [ENTER] to continue, Ctrl+C to abort.
+EOT
+read dummy
+
+#################################################################################
+echo "==================================================================="
+echo "Bringing remote Docker up to date…"
+$SSH mkdir -p $REMOTE_DOCKER_COMPOSE_DIR
+$SCP \
+    $ROOT/docker/{docker-compose.yml,renew-letsencrypt.sh,mongo-backup.{cron,sh}} \
+    $DEPLOYHOST:$REMOTE_DOCKER_COMPOSE_DIR
+$SSH -T <<EOT
+set -e
+cd $REMOTE_DOCKER_COMPOSE_DIR
+docker pull $DOCKER_IMAGE
+docker-compose up -d
+
+echo
+echo "==================================================================="
+echo "Clearing front page from Redis cache."
+docker exec redis redis-cli DEL pwview//
+EOT
+
+# Notify Sentry of this new deploy.
+# See https://sentry.io/blender-institute/blender-cloud/settings/release-tracking/
+# and https://docs.sentry.io/api/releases/post-organization-releases/
+# and https://sentry.io/api/
+echo
+echo "==================================================================="
+REVISION=$(date +'%Y%m%d.%H%M%S.%Z')
+echo "Notifying Sentry of this new deploy of revision $REVISION."
+SENTRY_RELEASE_URL="$($SSH env PYTHONPATH="$REMOTE_SECRET_CONFIG_DIR" python3 -c "\"import config_secrets; print(config_secrets.SENTRY_RELEASE_URL)\"")"
+curl -s "$SENTRY_RELEASE_URL" -XPOST -H 'Content-Type: application/json' -d "{\"version\": \"$REVISION\"}" | json_pp
+echo
+
+echo
+echo "==================================================================="
+echo "Deploy to $DEPLOYHOST done."
+echo "==================================================================="

deploy/README.md

@@ -0,0 +1,39 @@
+# Deploying to Production
+
+```
+workon blender-cloud  # activate your virtualenv
+cd $projectdir/deploy
+./full-pull.sh
+```
+
+## The Details
+
+Deployment consists of a few steps:
+
+1. Populate a staging directory with the files from the production branches of the various projects.
+2. Create Docker images.
+3. Push the docker images to Docker Hub.
+4. Pull the docker images on the production server and rebuild+restart the containers.
+
+The scripts involved are:
+
+- `2docker.sh`: performs step 1. above.
+- `build-{xxx}.sh`: performs steps 2. and 3. above.
+- `2server.sh`: performs step 4. above.
+
+The `full-{xxx}.sh` scripts perform all the steps, and call into `build-{xxx}.sh`.
+
+For `xxx` there are:
+
+- `all`: Rebuild all Docker images from scratch. This is good for getting the latest updates to the
+  base image.
+- `pull`: Pull the base and intermediate images from Docker Hub so that they are the same as the
+  last time someone pushed to production, then rebuilds the final Docker image.
+- `quick`: Just rebuild the final Docker image. Only use this if the last time a deployment to
+  the production server was done was by you, on the machine you're working on now.
+
+
+## Hacking Stuff
+
+To deploy another branch than `production`, do `export STAGING_BRANCH=otherbranch` before starting
+the above commands.

deploy/build-all.sh

@@ -0,0 +1,43 @@
+#!/bin/bash -e
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [[ `uname` == 'Darwin' ]]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+ROOT="$(dirname "$(dirname "$($readlink -f "$0")")")"
+
+case "$(basename "$0")" in
+    build-pull.sh)
+        docker pull armadillica/pillar_py:3.6
+        docker pull armadillica/pillar_wheelbuilder:latest
+        pushd "$ROOT/docker/3_buildwheels"
+        ./build.sh
+        popd
+        pushd "$ROOT/docker/4_run"
+        ./build.sh
+        ;;
+    build-quick.sh)
+        pushd "$ROOT/docker/4_run"
+        ./build.sh
+        ;;
+    build-all.sh)
+        pushd "$ROOT/docker"
+        ./full_rebuild.sh
+        ;;
+    *)
+        echo "Unknown script $0, aborting" >&2
+        exit 1
+esac
+
+popd
+echo
+echo "Press [ENTER] to push the new Docker images."
+read dummy
+docker push armadillica/pillar_py:3.6
+docker push armadillica/pillar_wheelbuilder:latest
+docker push armadillica/blender_cloud:latest
+echo
+echo "Build is done, ready to update the server."

deploy/build-pull.sh

@@ -0,0 +1 @@
+build-quick.sh

deploy/build-quick.sh

@@ -0,0 +1 @@
+build-all.sh

deploy/full-all.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+set -e
+
+NAME="$(basename "$0")"
+
+./2docker.sh
+./${NAME/full-/build-}
+./2server.sh cloud2

deploy/full-pull.sh

@@ -0,0 +1 @@
+full-all.sh

deploy/full-quick.sh

@@ -0,0 +1 @@
+full-all.sh

docker/1_base/Dockerfile

@@ -0,0 +1,10 @@
+FROM ubuntu:18.04
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+RUN set -ex; \
+    apt-get update; \
+    DEBIAN_FRONTEND=noninteractive apt-get install \
+      -qyy -o APT::Install-Recommends=false -o APT::Install-Suggests=false \
+      tzdata openssl ca-certificates locales; \
+    locale-gen en_US.UTF-8 en_GB.UTF-8 nl_NL.UTF-8
+ENV LANG en_US.UTF-8

docker/1_base/base.docker

@@ -1,16 +0,0 @@
-FROM ubuntu:16.04
-MAINTAINER Francesco Siddi <francesco@blender.org>
-
-RUN apt-get update && apt-get install -qyy \
-    -o APT::Install-Recommends=false -o APT::Install-Suggests=false \
-    python-pip libffi6 openssl ffmpeg rsyslog logrotate
-
-RUN mkdir -p /data/git/pillar \
-&& mkdir -p /data/storage \
-&& mkdir -p /data/config \
-&& mkdir -p /data/venv \
-&& mkdir -p /data/wheelhouse
-
-RUN pip install virtualenv
-RUN virtualenv /data/venv
-RUN . /data/venv/bin/activate && pip install -U pip && pip install wheel

docker/1_base/build.sh

@@ -1,3 +1,4 @@
 #!/usr/bin/env bash
 
-docker build -t pillar_base -f base.docker .;
+# Uses --no-cache to always get the latest upstream (security) upgrades.
+exec docker build --no-cache "$@" -t pillar_base .

docker/2_build/build-wheels.sh

@@ -1,3 +0,0 @@
-#!/usr/bin/env bash
-
-. /data/venv/bin/activate && pip wheel --wheel-dir=/data/wheelhouse -r /requirements.txt

docker/2_build/build.docker

@@ -1,26 +0,0 @@
-FROM pillar_base
-MAINTAINER Francesco Siddi <francesco@blender.org>
-
-RUN apt-get update && apt-get install -qy \
-git \
-gcc \
-libffi-dev \
-libssl-dev \
-pypy-dev \
-python-dev \
-python-imaging \
-zlib1g-dev \
-libjpeg-dev \
-libtiff-dev \
-python-crypto \
-python-openssl
-
-ENV WHEELHOUSE=/data/wheelhouse
-ENV PIP_WHEEL_DIR=/data/wheelhouse
-ENV PIP_FIND_LINKS=/data/wheelhouse
-
-VOLUME /data/wheelhouse
-
-ADD requirements.txt /requirements.txt
-ADD build-wheels.sh /build-wheels.sh
-ENTRYPOINT ["bash", "build-wheels.sh"]

docker/2_build/build.sh

@@ -1,11 +0,0 @@
-#!/usr/bin/env bash
-
-mkdir -p ../3_run/wheelhouse;
-cp ../../requirements.txt .;
-
-docker build -t pillar_build -f build.docker .;
-docker run --rm \
-       -v "$(pwd)"/../3_run/wheelhouse:/data/wheelhouse \
-       pillar_build;
-
-rm requirements.txt;

docker/2_buildpy/Python-3.6.6.tar.xz.md5

@@ -0,0 +1 @@
+c3f30a0aff425dda77d19e02f420d6ba  Python-3.6.6.tar.xz

docker/2_buildpy/build.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+
+set -e
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [ $(uname) == 'Darwin' ]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+
+PYTHONTARGET=$($readlink -f ./python)
+
+mkdir -p "$PYTHONTARGET"
+echo "Python will be built to $PYTHONTARGET"
+
+docker build -t pillar_build -f buildpy.docker .
+
+# Use the docker image to build Python 3.6 and mod-wsgi
+GID=$(id -g)
+docker run --rm -i \
+    -v "$PYTHONTARGET:/opt/python" \
+    pillar_build <<EOT
+set -e
+cd \$PYTHONSOURCE
+./configure \
+    --prefix=/opt/python \
+    --enable-ipv6 \
+    --enable-shared \
+    --with-ensurepip=upgrade
+make -j8 install
+
+# Make sure we can run Python
+ldconfig
+
+# Upgrade pip
+/opt/python/bin/python3 -m pip install -U pip
+
+# Build mod-wsgi-py3 for Python 3.6
+cd /dpkg/mod-wsgi-*
+./configure --with-python=/opt/python/bin/python3
+make -j8 install
+mkdir -p /opt/python/mod-wsgi
+cp /usr/lib/apache2/modules/mod_wsgi.so /opt/python/mod-wsgi
+
+chown -R $UID:$GID /opt/python/*
+EOT
+
+# Strip some stuff we don't need from the Python install.
+rm -rf $PYTHONTARGET/lib/python3.*/test
+rm -rf $PYTHONTARGET/lib/python3.*/config-3.*/libpython3.*.a
+find $PYTHONTARGET/lib -name '*.so.*' -o -name '*.so' | while read libname; do
+    chmod u+w "$libname"
+    strip "$libname"
+done
+
+# Create another docker image which contains the actual Python.
+# This one will serve as base for the Wheel builder and the
+# production image.
+docker build -t armadillica/pillar_py:3.6 -f includepy.docker .

docker/2_buildpy/buildpy.docker

@@ -0,0 +1,35 @@
+FROM pillar_base
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+RUN sed -i 's/^# deb-src/deb-src/' /etc/apt/sources.list && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -qy \
+        build-essential \
+        apache2-dev \
+        checkinstall \
+        curl
+
+RUN apt-get build-dep -y python3.6
+
+ADD Python-3.6.6.tar.xz.md5 /Python-3.6.6.tar.xz.md5
+
+# Install Python sources
+RUN curl -O https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz && \
+    md5sum -c Python-3.6.6.tar.xz.md5 && \
+    tar xf Python-3.6.6.tar.xz && \
+    rm -v Python-3.6.6.tar.xz
+
+# Install mod-wsgi sources
+RUN mkdir -p /dpkg && cd /dpkg && apt-get source libapache2-mod-wsgi-py3
+
+# To be able to install Python outside the docker.
+VOLUME /opt/python
+
+# To be able to run Python; after building, ldconfig has to be re-run to do this.
+# This makes it easier to use Python right after building (for example to build
+# mod-wsgi for Python 3.6).
+RUN echo /opt/python/lib > /etc/ld.so.conf.d/python.conf
+RUN ldconfig
+ENV PATH=/opt/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
+ENV PYTHONSOURCE=/Python-3.6.6

docker/2_buildpy/includepy.docker

@@ -0,0 +1,13 @@
+FROM pillar_base
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+ADD python /opt/python
+
+RUN echo /opt/python/lib > /etc/ld.so.conf.d/python.conf
+RUN ldconfig
+
+RUN echo Python is installed in /opt/python/ > README.python
+ENV PATH=/opt/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
+RUN cd /opt/python/bin && \
+    ln -s python3 python

docker/3_buildwheels/Dockerfile

@@ -0,0 +1,20 @@
+FROM armadillica/pillar_py:3.6
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+RUN set -ex; \
+    apt-get update; \
+    DEBIAN_FRONTEND=noninteractive apt-get install -qy \
+    git \
+    build-essential \
+    checkinstall \
+    libffi-dev \
+    libssl-dev \
+    libjpeg-dev \
+    zlib1g-dev
+
+ENV WHEELHOUSE=/data/wheelhouse
+ENV PIP_WHEEL_DIR=/data/wheelhouse
+ENV PIP_FIND_LINKS=/data/wheelhouse
+RUN mkdir -p $WHEELHOUSE
+
+VOLUME /data/wheelhouse

docker/3_buildwheels/build.sh

@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+
+DOCKER_IMAGE_NAME=armadillica/pillar_wheelbuilder
+
+set -e
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [ $(uname) == 'Darwin' ]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+
+TOPDEVDIR="$($readlink -f ../../..)"
+echo "Top-level development dir is $TOPDEVDIR"
+
+WHEELHOUSE="$($readlink -f ../4_run/wheelhouse)"
+if [ -z "$WHEELHOUSE" ]; then
+    echo "Error, ../4_run might not exist." >&2
+    exit 2
+fi
+
+echo "Wheelhouse is $WHEELHOUSE"
+mkdir -p "$WHEELHOUSE"
+rm -f "$WHEELHOUSE"/*
+
+docker build -t $DOCKER_IMAGE_NAME:latest .
+
+GID=$(id -g)
+docker run --rm -i \
+    -v "$WHEELHOUSE:/data/wheelhouse" \
+    -v "$TOPDEVDIR:/data/topdev" \
+    $DOCKER_IMAGE_NAME <<EOT
+set -e
+set -x
+
+# Globally upgrade Pip, so that we can get a compatible version of the cryptography package.
+# See https://github.com/pyca/cryptography/issues/5771
+pip3 install --upgrade pip setuptools wheel
+
+# Pin poetry to 1.0, as more recent version to not support nested filesystem package
+# dependencies.
+pip3 install wheel poetry==1.0 cryptography==2.7
+
+# Build wheels for all dependencies.
+cd /data/topdev/blender-cloud
+
+poetry install --no-dev
+
+# Apparently pip doesn't like projects without setup.py, so it think we have 'pillar-svnman' as
+# requirement (because that's the name of the directory). We have to grep that out.
+poetry run pip3 freeze | grep -v '\(pillar\)\|\(^-[ef] \)' > \$WHEELHOUSE/requirements.txt
+
+pip3 wheel --wheel-dir=\$WHEELHOUSE -r \$WHEELHOUSE/requirements.txt
+chown -R $UID:$GID \$WHEELHOUSE
+EOT
+
+# Remove our own projects, they shouldn't be installed as wheel (for now).
+rm -f $WHEELHOUSE/{attract,flamenco,pillar,pillarsdk}*.whl
+
+echo "Build of $DOCKER_IMAGE_NAME:latest is done."

docker/3_run/000-default.conf

@@ -1,39 +0,0 @@
-<VirtualHost *:80>
-	# EnableSendfile on
-	XSendFile on
-	XSendFilePath /data/storage/pillar
-	XSendFilePath /data/git/pillar
-	XSendFilePath /data/venv/lib/python2.7/site-packages/attract/static/
-	XSendFilePath /data/venv/lib/python2.7/site-packages/flamenco/static/
-	XsendFilePath /data/git/blender-cloud
-
-	ServerAdmin webmaster@localhost
-	DocumentRoot /var/www/html
-
-	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
-	# error, crit, alert, emerg.
-	# It is also possible to configure the loglevel for particular
-	# modules, e.g.
-	# LogLevel info ssl:warn
-
-	ErrorLog ${APACHE_LOG_DIR}/error.log
-	CustomLog ${APACHE_LOG_DIR}/access.log combined
-
-	WSGIDaemonProcess cloud processes=4 threads=1 maximum-requests=10000
-	WSGIPassAuthorization On
-
-	WSGIScriptAlias / /data/git/blender-cloud/runserver.wsgi \
-		process-group=cloud application-group=%{GLOBAL}
-
-	<Directory /data/git/blender-cloud>
-		<Files runserver.wsgi>
-			Require all granted
-		</Files>
-	</Directory>
-
-	# Temporary edit to remap the old cloudapi.blender.org to cloud.blender.org/api
-	RewriteEngine On
-	RewriteCond "%{HTTP_HOST}"   "^cloudapi\.blender\.org" [NC]
-	RewriteRule (.*) /api$1 [PT]
-
-</VirtualHost>

docker/3_run/build.sh

@@ -1,5 +0,0 @@
-#!/usr/bin/env bash
-
-cp ../../requirements.txt .;
-docker build -t armadillica/blender_cloud -f run.docker .;
-rm requirements.txt;

docker/3_run/docker-entrypoint.sh

@@ -1,25 +0,0 @@
-#!/usr/bin/env bash
-
-if [ ! -f /installed ]; then
-    echo "Installing pillar and pillar-sdk"
-    # TODO: curretly doing pip install -e takes a long time, so we symlink
-    # . /data/venv/bin/activate && pip install -e /data/git/pillar
-    ln -s /data/git/pillar/pillar /data/venv/lib/python2.7/site-packages/pillar
-    # . /data/venv/bin/activate && pip install -e /data/git/attract
-    ln -s /data/git/attract/attract /data/venv/lib/python2.7/site-packages/attract
-    # . /data/venv/bin/activate && pip install -e /data/git/flamenco/packages/flamenco
-    ln -s /data/git/flamenco/packages/flamenco/flamenco/ /data/venv/lib/python2.7/site-packages/flamenco
-    # . /data/venv/bin/activate && pip install -e /data/git/pillar-python-sdk
-    ln -s /data/git/pillar-python-sdk/pillarsdk /data/venv/lib/python2.7/site-packages/pillarsdk
-    touch installed
-fi
-
-if [ "$DEV" = "true" ]; then
-    echo "Running in development mode"
-    cd /data/git/blender-cloud
-    bash /manage.sh runserver --host='0.0.0.0'
-else
-    # Run Apache
-    a2enmod rewrite
-    /usr/sbin/apache2ctl -D FOREGROUND
-fi

docker/3_run/manage.sh

@@ -1,5 +0,0 @@
-#!/usr/bin/env bash -e
-
-. /data/venv/bin/activate
-cd /data/git/blender-cloud
-python manage.py "$@"

docker/3_run/run.docker

@@ -1,46 +0,0 @@
-FROM pillar_base
-
-RUN apt-get update && apt-get install -qyy \
--o APT::Install-Recommends=true -o APT::Install-Suggests=false \
-git \
-apache2 \
-libapache2-mod-wsgi \
-libapache2-mod-xsendfile \
-libjpeg8 \
-libtiff5 \
-nano vim curl \
-&& rm -rf /var/lib/apt/lists/*
-
-ENV APACHE_RUN_USER www-data
-ENV APACHE_RUN_GROUP www-data
-ENV APACHE_LOG_DIR /var/log/apache2
-ENV APACHE_PID_FILE /var/run/apache2.pid
-ENV APACHE_RUN_DIR /var/run/apache2
-ENV APACHE_LOCK_DIR /var/lock/apache2
-
-RUN mkdir -p $APACHE_RUN_DIR $APACHE_LOCK_DIR $APACHE_LOG_DIR
-
-ADD requirements.txt /requirements.txt
-ADD wheelhouse /data/wheelhouse
-
-RUN . /data/venv/bin/activate \
-&& pip install --no-index --find-links=/data/wheelhouse -r requirements.txt \
-&& rm /requirements.txt
-
-VOLUME /data/git/blender-cloud
-VOLUME /data/git/pillar
-VOLUME /data/git/pillar-python-sdk
-VOLUME /data/config
-VOLUME /data/storage
-
-ENV USE_X_SENDFILE True
-
-EXPOSE 80
-EXPOSE 5000
-
-ADD apache2.conf /etc/apache2/apache2.conf
-ADD 000-default.conf /etc/apache2/sites-available/000-default.conf
-ADD docker-entrypoint.sh /docker-entrypoint.sh
-ADD manage.sh /manage.sh
-
-ENTRYPOINT ["bash", "/docker-entrypoint.sh"]

docker/4_run/Dockerfile

@@ -0,0 +1,65 @@
+FROM armadillica/pillar_py:3.6
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+RUN set -ex; \
+    apt-get update; \
+    DEBIAN_FRONTEND=noninteractive apt-get install -qy \
+    -o APT::Install-Recommends=false -o APT::Install-Suggests=false \
+        git \
+        apache2 \
+        libapache2-mod-xsendfile \
+        libjpeg8 \
+        libtiff5 \
+        ffmpeg \
+        rsyslog logrotate \
+        nano vim-tiny curl; \
+    rm -rf /var/lib/apt/lists/*
+
+RUN ln -s /usr/bin/vim.tiny /usr/bin/vim
+
+ENV APACHE_RUN_USER www-data
+ENV APACHE_RUN_GROUP www-data
+ENV APACHE_LOG_DIR /var/log/apache2
+ENV APACHE_PID_FILE /var/run/apache2.pid
+ENV APACHE_RUN_DIR /var/run/apache2
+ENV APACHE_LOCK_DIR /var/lock/apache2
+
+RUN mkdir -p $APACHE_RUN_DIR $APACHE_LOCK_DIR $APACHE_LOG_DIR
+
+ADD wheelhouse /data/wheelhouse
+RUN pip3 install --no-index --find-links=/data/wheelhouse -r /data/wheelhouse/requirements.txt
+
+VOLUME /data/config
+VOLUME /data/storage
+VOLUME /var/log
+
+ENV USE_X_SENDFILE True
+
+EXPOSE 80
+EXPOSE 5000
+
+ADD apache/remoteip.conf /etc/apache2/mods-available/
+ADD apache/wsgi-py36.* /etc/apache2/mods-available/
+RUN a2enmod remoteip && a2enmod rewrite && a2enmod wsgi-py36
+
+ADD apache/apache2.conf /etc/apache2/apache2.conf
+ADD apache/000-default.conf /etc/apache2/sites-available/000-default.conf
+ADD apache/logrotate.conf /etc/logrotate.d/apache2
+ADD *.sh /
+
+# Remove some empty top-level directories we won't use anyway.
+RUN rmdir /media /home 2>/dev/null || true
+
+# This file includes some useful commands to have in the shell history
+# for easy access.
+ADD bash_history /root/.bash_history
+
+ENTRYPOINT /docker-entrypoint.sh
+
+# Add the most-changing files as last step for faster rebuilds.
+ADD config_local.py /data/git/blender-cloud/
+ADD staging /data/git
+RUN python3 -c "import re, secrets; \
+f = open('/data/git/blender-cloud/config_local.py', 'a'); \
+h = re.sub(r'[_.~-]', '', secrets.token_urlsafe())[:8]; \
+print(f'STATIC_FILE_HASH = {h!r}', file=f)"

docker/4_run/apache/000-default.conf

@@ -0,0 +1,56 @@
+<VirtualHost *:80>
+	XSendFile on
+	XSendFilePath /data/storage/pillar
+	XSendFilePath /data/git/pillar/pillar/web/static/
+	XSendFilePath /data/git/attract/attract/static/
+	XSendFilePath /data/git/flamenco/flamenco/static/
+	XsendFilePath /data/git/pillar-svnman/svnman/static/
+	XsendFilePath /data/git/blender-cloud/static/
+	XsendFilePath /data/git/blender-cloud/cloud/static/
+
+	ServerAdmin webmaster@localhost
+	DocumentRoot /var/www/html
+
+	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
+	# error, crit, alert, emerg.
+	# It is also possible to configure the loglevel for particular
+	# modules, e.g.
+	# LogLevel info ssl:warn
+
+	ErrorLog ${APACHE_LOG_DIR}/error.log
+	CustomLog ${APACHE_LOG_DIR}/access.log combined
+
+	WSGIDaemonProcess cloud processes=2 threads=64 maximum-requests=10000
+	WSGIPassAuthorization On
+
+	WSGIScriptAlias / /data/git/blender-cloud/runserver.wsgi \
+		process-group=cloud application-group=%{GLOBAL}
+
+	<Directory /data/git/blender-cloud>
+		<Files runserver.wsgi>
+			Require all granted
+		</Files>
+	</Directory>
+
+	# Temporary edit to remap the old cloudapi.blender.org to cloud.blender.org/api
+	RewriteEngine On
+	RewriteCond "%{HTTP_HOST}"   "^cloudapi\.blender\.org" [NC]
+	RewriteRule (.*) /api$1 [PT]
+
+	# Redirects for blender-cloud projects
+	RewriteRule "^/p/blender-cloud/?$"  "/blog" [R=301,L]
+	RewriteRule "^/agent327/?$"  "/p/agent-327" [R=301,L]
+	RewriteRule "^/caminandes/?$"  "/p/caminandes-3" [R=301,L]
+	RewriteRule "^/cf2/?$"  "/p/creature-factory-2" [R=301,L]
+	RewriteRule "^/characters/?$"  "/p/characters" [R=301,L]
+	RewriteRule "^/gallery/?$"  "/p/gallery" [R=301,L]
+	RewriteRule "^/hdri/?$"  "/p/hdri" [R=301,L]
+	RewriteRule "^/textures/?$"  "/p/textures" [R=301,L]
+	RewriteRule "^/training/?$"  "/courses" [R=301,L]
+	RewriteRule "^/spring/?$"  "/p/spring" [R=301,L]
+	RewriteRule "^/hero/?$"  "/p/hero" [R=301,L]
+	RewriteRule "^/coffee-run/?$"  "/p/coffee-run" [R=301,L]
+	RewriteRule "^/settlers/?$"  "/p/settlers" [R=301,L]
+	# Waking the forest was moved from the art gallery to its own workshop
+	RewriteRule "^/p/gallery/58cfec4f88ac8f1440aeb309/?$"  "/p/waking-the-forest" [R=301,L]
+</VirtualHost>

docker/4_run/apache/apache2.conf

@@ -133,9 +133,9 @@ AccessFileName .htaccess
 # Note that the use of %{X-Forwarded-For}i instead of %h is not recommended.
 # Use mod_remoteip instead.
 #
-LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
-LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
-LogFormat "%h %l %u %t \"%r\" %>s %O" common
+LogFormat "%v:%p %a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
+LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
+LogFormat "%a %l %u %t \"%r\" %>s %O" common
 LogFormat "%{Referer}i -> %U" referer
 LogFormat "%{User-agent}i" agent
 

docker/4_run/apache/logrotate.conf

@@ -0,0 +1,21 @@
+/var/log/apache2/*.log {
+    daily
+    missingok
+    rotate 14
+    size 100M
+    compress
+    delaycompress
+    notifempty
+    create 640 root adm
+    sharedscripts
+    postrotate
+        if /etc/init.d/apache2 status > /dev/null ; then \
+            /etc/init.d/apache2 reload > /dev/null; \
+        fi;
+    endscript
+    prerotate
+        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
+            run-parts /etc/logrotate.d/httpd-prerotate; \
+        fi; \
+    endscript
+}

docker/4_run/apache/remoteip.conf

@@ -0,0 +1,2 @@
+RemoteIPHeader X-Forwarded-For
+RemoteIPInternalProxy 172.16.0.0/12

docker/4_run/apache/wsgi-py36.conf

@@ -0,0 +1,122 @@
+<IfModule mod_wsgi.c>
+
+
+    #This config file is provided to give an overview of the directives,
+    #which are only allowed in the 'server config' context.
+    #For a detailed description of all avaiable directives please read
+    #http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives
+
+
+    #WSGISocketPrefix: Configure directory to use for daemon sockets.
+    #
+    #Apache's DEFAULT_REL_RUNTIMEDIR should be the proper place for WSGI's
+    #Socket. In case you want to mess with the permissions of the directory,
+    #you need to define WSGISocketPrefix to an alternative directory.
+    #See http://code.google.com/p/modwsgi/wiki/ConfigurationIssues for more
+    #information
+
+        #WSGISocketPrefix /var/run/apache2/wsgi
+
+
+    #WSGIPythonOptimize: Enables basic Python optimisation features.
+    #
+    #Sets the level of Python compiler optimisations. The default is '0'
+    #which means no optimisations are applied.
+    #Setting the optimisation level to '1' or above will have the effect
+    #of enabling basic Python optimisations and changes the filename
+    #extension for compiled (bytecode) files from .pyc to .pyo.
+    #When the optimisation level is set to '2', doc strings will not be
+    #generated and retained. This will result in a smaller memory footprint,
+    #but may cause some Python packages which interrogate doc strings in some
+    #way to fail.
+
+        #WSGIPythonOptimize 0
+
+
+    #WSGIPythonPath: Additional directories to search for Python modules,
+    #                overriding the PYTHONPATH environment variable.
+    #
+    #Used to specify additional directories to search for Python modules.
+    #If multiple directories are specified they should be separated by a ':'.
+
+        WSGIPythonPath /opt/python/lib/python3.6/site-packages
+
+    #WSGIPythonEggs: Directory to use for Python eggs cache.
+    #
+    #Used to specify the directory to be used as the Python eggs cache
+    #directory for all sub interpreters created within embedded mode.
+    #This directive achieves the same affect as having set the
+    #PYTHON_EGG_CACHE environment variable.
+    #Note that the directory specified must exist and be writable by the user
+    #that the Apache child processes run as. The directive only applies to
+    #mod_wsgi embedded mode. To set the Python eggs cache directory for
+    #mod_wsgi daemon processes, use the 'python-eggs' option to the
+    #WSGIDaemonProcess directive instead.
+
+        #WSGIPythonEggs directory
+
+
+
+    #WSGIRestrictEmbedded: Enable restrictions on use of embedded mode.
+    #
+    #The WSGIRestrictEmbedded directive determines whether mod_wsgi embedded
+    #mode is enabled or not. If set to 'On' and the restriction on embedded
+    #mode is therefore enabled, any attempt to make a request against a
+    #WSGI application which hasn't been properly configured so as to be
+    #delegated to a daemon mode process will fail with a HTTP internal server
+    #error response.
+
+        #WSGIRestrictEmbedded On|Off
+
+
+
+    #WSGIRestrictStdin: Enable restrictions on use of STDIN.
+    #WSGIRestrictStdout: Enable restrictions on use of STDOUT.
+    #WSGIRestrictSignal: Enable restrictions on use of signal().
+    #
+    #Well behaved WSGI applications neither should try to read/write from/to
+    #STDIN/STDOUT, nor should they try to register signal handlers. If your
+    #application needs an exception from this rule, you can disable the
+    #restrictions here.
+
+        #WSGIRestrictStdin On
+        #WSGIRestrictStdout On
+        #WSGIRestrictSignal On
+
+
+
+    #WSGIAcceptMutex: Specify type of accept mutex used by daemon processes.
+    #
+    #The WSGIAcceptMutex directive sets the method that mod_wsgi will use to
+    #serialize multiple daemon processes in a process group accepting requests
+    #on a socket connection from the Apache child processes. If this directive
+    #is not defined then the same type of mutex mechanism as used by Apache for
+    #the main Apache child processes when accepting connections from a client
+    #will be used. If set the method types are the same as for the Apache
+    #AcceptMutex directive.
+
+        #WSGIAcceptMutex default
+
+
+
+    #WSGIImportScript: Specify a script file to be loaded on process start.
+    #
+    #The WSGIImportScript directive can be used to specify a script file to be
+    #loaded when a process starts. Options must be provided to indicate the
+    #name of the process group and the application group into which the script
+    #will be loaded.
+
+        #WSGIImportScript process-group=name application-group=name
+
+
+    #WSGILazyInitialization: Enable/disable lazy initialisation of Python.
+    #
+    #The WSGILazyInitialization directives sets whether or not the Python
+    #interpreter is preinitialised within the Apache parent process or whether
+    #lazy initialisation is performed, and the Python interpreter only
+    #initialised in the Apache server processes or mod_wsgi daemon processes
+    #after they have forked from the Apache parent process.
+
+        #WSGILazyInitialization On|Off
+
+</IfModule>

docker/4_run/apache/wsgi-py36.load

@@ -0,0 +1 @@
+LoadModule wsgi_module /opt/python/mod-wsgi/mod_wsgi.so

docker/4_run/bash_history

@@ -0,0 +1,9 @@
+bash docker-entrypoint.sh
+env | sort
+apache2ctl start
+apache2ctl graceful
+/manage.sh operations worker -- -C
+celery status --broker amqp://guest:guest@rabbit:5672//
+celery events --broker amqp://guest:guest@rabbit:5672//
+tail -n 40 -f /var/log/apache2/access.log
+tail -n 40 -f /var/log/apache2/error.log

docker/4_run/build.sh

@@ -0,0 +1,5 @@
+#!/bin/bash -e
+
+docker build -t armadillica/blender_cloud:latest .
+
+echo "Done, built armadillica/blender_cloud:latest"

docker/4_run/celery-beat.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+source /install_scripts.sh
+source /manage.sh celery beat -- \
+    --schedule /data/storage/pillar/celerybeat-schedule.db \
+    --pid /data/storage/pillar/celerybeat.pid

docker/4_run/celery-worker.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+
+source /install_scripts.sh
+source /manage.sh celery worker -- -C

docker/4_run/config_local.py

@@ -0,0 +1,123 @@
+import os
+from collections import defaultdict
+
+DEBUG = False
+
+SCHEME = 'https'
+PREFERRED_URL_SCHEME = 'https'
+SERVER_NAME = 'cloud.blender.org'
+
+# os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = 'true'
+os.environ['PILLAR_MONGO_DBNAME'] = 'cloud'
+os.environ['PILLAR_MONGO_PORT'] = '27017'
+os.environ['PILLAR_MONGO_HOST'] = 'mongo'
+
+USE_X_SENDFILE = True
+
+STORAGE_BACKEND = 'gcs'
+
+CDN_SERVICE_DOMAIN = 'blendercloud-pro.r.worldssl.net'
+CDN_CONTENT_SUBFOLDER = ''
+CDN_STORAGE_ADDRESS = 'push-11.cdnsun.com'
+
+CACHE_TYPE = 'redis'  # null
+CACHE_KEY_PREFIX = 'pw_'
+CACHE_REDIS_HOST = 'redis'
+CACHE_REDIS_PORT = '6379'
+CACHE_REDIS_URL = 'redis://redis:6379'
+
+PILLAR_SERVER_ENDPOINT = 'https://cloud.blender.org/api/'
+
+BLENDER_ID_ENDPOINT = 'https://www.blender.org/id/'
+
+GCLOUD_APP_CREDENTIALS = '/data/config/google_app.json'
+GCLOUD_PROJECT = 'blender-cloud'
+
+MAIN_PROJECT_ID = '563a9c8cf0e722006ce97b03'
+# MAIN_PROJECT_ID = '57aa07c088bef606e89078bd'
+
+ALGOLIA_INDEX_USERS = 'pro_Users'
+ALGOLIA_INDEX_NODES = 'pro_Nodes'
+
+ZENCODER_NOTIFICATIONS_URL = 'https://cloud.blender.org/api/encoding/zencoder/notifications'
+
+FILE_LINK_VALIDITY = defaultdict(
+    lambda: 3600 * 24 * 30,  # default of 1 month.
+    gcs=3600 * 23,  # 23 hours for Google Cloud Storage.
+    cdnsun=3600 * 23
+)
+
+LOGGING = {
+    'version': 1,
+    'formatters': {
+        'default': {'format': '%(levelname)8s %(name)s %(message)s'}
+    },
+    'handlers': {
+        'console': {
+            'class': 'logging.StreamHandler',
+            'formatter': 'default',
+            'stream': 'ext://sys.stderr',
+        }
+    },
+    'loggers': {
+        'pillar': {'level': 'INFO'},
+        # 'pillar.auth': {'level': 'DEBUG'},
+        # 'pillar.api.blender_id': {'level': 'DEBUG'},
+        # 'pillar.api.blender_cloud.subscription': {'level': 'DEBUG'},
+        'bcloud': {'level': 'INFO'},
+        'cloud': {'level': 'INFO'},
+        'attract': {'level': 'INFO'},
+        'flamenco': {'level': 'INFO'},
+        # 'pillar.api.file_storage': {'level': 'DEBUG'},
+        # 'pillar.api.file_storage.ensure_valid_link': {'level': 'INFO'},
+        'pillar.api.file_storage.refresh_links_for_backend': {'level': 'DEBUG'},
+        'werkzeug': {'level': 'DEBUG'},
+        'eve': {'level': 'WARNING'},
+        # 'elasticsearch': {'level': 'DEBUG'},
+    },
+    'root': {
+        'level': 'WARNING',
+        'handlers': [
+            'console',
+        ],
+    }
+}
+
+# Latest version of the add-on.
+BLENDER_CLOUD_ADDON_VERSION = '1.9.0'
+
+REDIRECTS = {
+    # For old links, refer to the services page (hopefully it refreshes then)
+    'downloads/blender_cloud-latest-bundle.zip': 'https://cloud.blender.org/services#blender-addon',
+
+    # Latest Blender Cloud add-on.
+    'downloads/blender_cloud-latest-addon.zip':
+        f'https://storage.googleapis.com/institute-storage/addons/'
+        f'blender_cloud-{BLENDER_CLOUD_ADDON_VERSION}.addon.zip',
+
+    # Redirect old Grafista endpoint to /stats
+    '/stats/': '/stats',
+}
+
+UTM_LINKS = {
+    'cartoon_brew': {
+        'image': 'https://imgur.com/13nQTi3.png',
+        'link': 'https://store.blender.org/product/membership/'
+    }
+}
+
+SVNMAN_REPO_URL = 'https://svn.blender.cloud/repo/'
+SVNMAN_API_URL = 'https://svn.blender.cloud/api/'
+
+# Mail options, see pillar.celery.email_tasks.
+SMTP_HOST = 'proog.blender.org'
+SMTP_PORT = 25
+SMTP_USERNAME = 'server@blender.cloud'
+SMTP_TIMEOUT = 30  # timeout in seconds, https://docs.python.org/3/library/smtplib.html#smtplib.SMTP
+MAIL_RETRY = 180  # in seconds, delay until trying to send an email again.
+MAIL_DEFAULT_FROM_NAME = 'Blender Cloud'
+MAIL_DEFAULT_FROM_ADDR = 'cloudsupport@blender.org'
+
+# MUST be 8 characters long, see pillar.flask_extra.HashedPathConverter
+# STATIC_FILE_HASH = '12345678'
+# The value used in production is appended from Dockerfile.

docker/4_run/docker-entrypoint.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+source /install_scripts.sh
+
+# Make sure that log rotation works.
+mkdir -p ${APACHE_LOG_DIR}
+service cron start
+
+if [ "$DEV" = "true" ]; then
+    echo "Running in development mode"
+    cd /data/git/blender-cloud
+    exec bash /manage.sh runserver --host='0.0.0.0'
+else
+    exec /usr/sbin/apache2ctl -D FOREGROUND
+fi

docker/4_run/install_scripts.sh

@@ -0,0 +1,21 @@
+#!/bin/sh
+
+if [ -f /installed ]; then
+    return
+fi
+
+SITEPKG=$(echo /opt/python/lib/python3.*/site-packages)
+echo "Installing Blender Cloud packages into $SITEPKG"
+
+# TODO: 'pip3 install -e' runs 'setup.py develop', which runs 'setup.py egg_info',
+# which can't write the egg info to the read-only /data/git volume. This is why
+# we manually install the links.
+for SUBPROJ in /data/git/{pillar,pillar-python-sdk,attract,flamenco,pillar-svnman}; do
+    NAME=$(python3 $SUBPROJ/setup.py --name)
+    echo "... $NAME"
+    echo $SUBPROJ >> $SITEPKG/easy-install.pth
+    echo $SUBPROJ > $SITEPKG/$NAME.egg-link
+done
+echo "All packages installed."
+
+touch /installed

docker/4_run/manage.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+
+set -e
+cd /data/git/blender-cloud
+exec python manage.py "$@"

docker/README.md

@@ -0,0 +1,96 @@
+# Setting up a production machine
+
+To get the docker stack up and running, we use the following, on an Ubuntu 16.10 machine.
+
+## 0. Basic stuff
+
+Install the machine, use `locale-gen nl_NL.UTF-8` or similar commands to generate locale
+definitions. Set up automatic security updates and backups, the usual.
+
+## 1. Install Docker
+
+Install Docker itself, as described in the
+   [Docker CE for Ubuntu manual](https://store.docker.com/editions/community/docker-ce-server-ubuntu?tab=description):
+
+    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+    add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+        $(lsb_release -cs) stable"
+    apt-get update
+    apt-get install docker-ce
+
+## 2. Configure Docker to use "overlay"
+
+Configure Docker to use "overlay" instead of "aufs" for the images. This prevents
+[segfaults in auplink](https://bugs.launchpad.net/ubuntu/+source/aufs-tools/+bug/1442568).
+
+1. Set `DOCKER_OPTS="-s overlay"` in `/etc/defaults/docker`
+2. Copy `/lib/systemd/system/docker.service` to `/etc/systemd/system/docker.service`.
+   This allows later upgrading of docker without overwriting the changes we're about to do.
+2. Edit the `[Service]` section of `/etc/systemd/system/docker.service`:
+    1. Add `EnvironmentFile=/etc/default/docker`
+    2. Append ` $DOCKER_OPTS` to the `ExecStart` line
+3. Run `systemctl daemon-reload`
+4. Remove all your containers and images.
+5. Restart Docker: `systemctl restart docker`
+
+## 3. Pull the Blender Cloud docker image
+
+`docker pull armadillica/blender_cloud:latest`
+
+## 4. Get docker-compose + our repositories
+
+See the [Quick setup](../README.md) on how to get those. Then run:
+
+    cd /data/git/blender-cloud/docker
+    docker-compose up -d
+
+
+Set up permissions for Docker volumes; the following should be writable by
+
+- `/data/storage/pillar`: writable by `www-data` and `root` (do a `chown root:www-data`
+  and `chmod 2770`).
+- `/data/storage/db`: writable by uid 999.
+
+
+## 5. Set up TLS
+
+Place TLS certificates in `/data/certs/{cloud,cloudapi}.blender.org.pem`.
+They should contain (in order) the private key, the host certificate, and the
+CA certificate.
+
+
+## 6. Create a local config
+
+Blender Cloud expects the following files to exist:
+
+- `/data/git/blender_cloud/config_local.py` with machine-local configuration overrides
+- `/data/config/google_app.json` with Google Cloud Storage credentials.
+
+When run from Docker, the `docker/4_run/config_local.py` file will be used. Overrides for that file
+can be placed in `/data/config/config_secrets.py`.
+
+
+## 7. ElasticSearch & kibana
+
+ElasticSearch and Kibana run in our self-rolled images. This is needed because by default
+
+- ElasticSearch uses up to 2 GB of RAM, which is too much for our droplet, and
+- the Docker images contain the proprietary X-Pack plugin, which we don't want.
+
+This also gives us the opportunity to let Kibana do its optimization when we build the image, rather
+than every time the container is recreated.
+
+`/data/storage/elasticsearch` needs to be writable by UID 1000, GID 1000.
+
+Kibana connects to [ElasticProxy](https://github.com/armadillica/elasticproxy), which only allows
+GET, HEAD, and some specific POST requests. This ensures that the public-facing Kibana cannot be
+used to change the ElasticSearch database.
+
+Production Kibana can be placed in read-only mode, but this is not necessary now that we use
+ElasticProxy. However, I've left this in here as reference.
+
+`curl -XPUT 'localhost:9200/.kibana/_settings' -d '{ "index.blocks.read_only" : true }'`
+
+If editing is desired, temporarily turn off read-only mode:
+
+`curl -XPUT 'localhost:9200/.kibana/_settings' -d '{ "index.blocks.read_only" : false }'`

docker/build.sh

@@ -1,16 +0,0 @@
-#!/usr/bin/env bash
-
-set -x;
-set -e;
-
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-cd $DIR;
-
-cd 1_base/;
-bash build.sh;
-
-cd ../2_build/;
-bash build.sh;
-
-cd ../3_run/;
-bash build.sh;

docker/docker-compose.yml

@@ -1,68 +1,183 @@
-mongo:
-  image: mongo
-  container_name: mongo
-  restart: always
-  volumes:
-   - /data/storage/db:/data/db
-  ports:
-   - "127.0.0.1:27017:27017"
-redis:
-  image: redis
-  container_name: redis
-  restart: always
-blender_cloud:
-  image: armadillica/blender_cloud
-  container_name: blender_cloud
-  restart: always
-  environment:
-   VIRTUAL_HOST: http://cloudapi.blender.org/*,https://cloudapi.blender.org/*,http://cloud.blender.org/*,https://cloud.blender.org/*,http://pillar-web/*
-   VIRTUAL_HOST_WEIGHT: 10
-   FORCE_SSL: "true"
-  volumes:
-   - /data/git/blender-cloud:/data/git/blender-cloud:ro
-   - /data/git/attract:/data/git/attract:ro
-   - /data/git/flamenco:/data/git/flamenco:ro
-   - /data/git/pillar:/data/git/pillar:ro
-   - /data/git/pillar-python-sdk:/data/git/pillar-python-sdk:ro
-   - /data/config:/data/config:ro
-   - /data/storage/pillar:/data/storage/pillar
-  links:
-   - mongo
-   - redis
-# notifserv:
-#   container_name: notifserv
-#   image: armadillica/pillar-notifserv:cd8fa678436563ac3b800b2721e36830c32e4656
-#   restart: always
-#   links:
-#    - mongo
-#   environment:
-#    VIRTUAL_HOST: https://cloud.blender.org/notifications*,http://pillar-web/notifications*
-#    VIRTUAL_HOST_WEIGHT: 20
-#    FORCE_SSL: true
-grafista:
-  image: armadillica/grafista
-  container_name: grafista
-  restart: always
-  environment:
-    VIRTUAL_HOST: http://cloud.blender.org/stats/*,https://cloud.blender.org/stats/*,http://blender-cloud/stats/*
-    VIRTUAL_HOST_WEIGHT: 20
-    FORCE_SSL: "true"
-  volumes:
-    - /data/git/grafista:/data/git/grafista:ro
-    - /data/storage/grafista:/data/storage
-haproxy:
-  image: dockercloud/haproxy
-  container_name: haproxy
-  restart: always
-  ports:
-   - "443:443"
-   - "80:80"
-  environment:
-   - CERT_FOLDER=/certs/
-   - TIMEOUT=connect 5s, client 5m, server 10m
-  links:
-   - blender_cloud
-   - grafista
-   # - notifserv
-  volumes:
-   - '/data/certs:/certs'
+version:                     '3.4'
+services:
+  mongo:
+    image:                   mongo:3.4
+    container_name:          mongo
+    restart:                 always
+    volumes:
+      - /data/storage/db:/data/db
+      - /data/storage/db-bak:/data/db-bak # for backing up stuff etc.
+    ports:
+      - "127.0.0.1:27017:27017"
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  # Databases in use:
+  # 0: Flask Cache
+  # 1: Celery (backend)
+  # 2: Celery (broker)
+  redis:
+    image:                   redis:5.0
+    container_name:          redis
+    restart:                 always
+    ports:
+      - "127.0.0.1:6379:6379"
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  elastic:
+    # This image is defined in blender-cloud/docker/elastic
+    image:                   armadillica/elasticsearch:6.1.1
+    container_name:          elastic
+    restart:                 always
+    volumes:
+      #  NOTE:               this path must be writable by UID=1000 GID=1000.
+      - /data/storage/elastic:/usr/share/elasticsearch/data
+    ports:
+      - "127.0.0.1:9200:9200"
+    environment:
+      ES_JAVA_OPTS:          "-Xms256m -Xmx256m"
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  elasticproxy:
+    image:                   armadillica/elasticproxy:1.2
+    container_name:          elasticproxy
+    restart:                 always
+    command:                 /elasticproxy -elastic http://elastic:9200/
+    depends_on:
+      - elastic
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  kibana:
+    # This image is defined in blender-cloud/docker/elastic
+    image:                   armadillica/kibana:6.1.1
+    container_name:          kibana
+    restart:                 always
+    environment:
+      SERVER_NAME:           "stats.cloud.blender.org"
+      ELASTICSEARCH_URL:     http://elasticproxy:9200
+      CONSOLE_ENABLED:       'false'
+      VIRTUAL_HOST:          http://stats.cloud.blender.org/*,https://stats.cloud.blender.org/*,http://stats.cloud.local/*,https://stats.cloud.local/*
+      VIRTUAL_HOST_WEIGHT:   20
+      FORCE_SSL:             "true"
+
+      # See https://github.com/elastic/kibana/issues/5170#issuecomment-163042525
+      NODE_OPTIONS:          "--max-old-space-size=200"
+    depends_on:
+      - elasticproxy
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  blender_cloud:
+    image:                   armadillica/blender_cloud:latest
+    container_name:          blender_cloud
+    restart:                 always
+    environment:
+      VIRTUAL_HOST:          http://cloud.blender.org/*,https://cloud.blender.org/*,http://cloud.local/*,https://cloud.local/*
+      VIRTUAL_HOST_WEIGHT:   10
+      FORCE_SSL:             "true"
+      GZIP_COMPRESSION_TYPE: "text/html text/plain text/css application/javascript"
+      PILLAR_CONFIG:         /data/config/config_secrets.py
+    volumes:
+      # format:              HOST:CONTAINER
+      - /data/config:/data/config:ro
+      - /data/storage/pillar:/data/storage/pillar
+      - /data/log:/var/log
+    depends_on:
+      - mongo
+      - redis
+
+  celery_worker:
+    image:                   armadillica/blender_cloud:latest
+    entrypoint:              /celery-worker.sh
+    container_name:          celery_worker
+    restart:                 always
+    environment:
+      PILLAR_CONFIG:         /data/config/config_secrets.py
+    volumes:
+      # format:              HOST:CONTAINER
+      - /data/config:/data/config:ro
+      - /data/storage/pillar:/data/storage/pillar
+      - /data/log:/var/log
+    depends_on:
+      - mongo
+      - redis
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  celery_beat:
+    image:                   armadillica/blender_cloud:latest
+    entrypoint:              /celery-beat.sh
+    container_name:          celery_beat
+    restart:                 always
+    environment:
+      PILLAR_CONFIG:         /data/config/config_secrets.py
+    volumes:
+      # format:              HOST:CONTAINER
+      - /data/config:/data/config:ro
+      - /data/storage/pillar:/data/storage/pillar
+      - /data/log:/var/log
+    depends_on:
+      - mongo
+      - redis
+      - celery_worker
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  letsencrypt:
+    image:                   armadillica/picohttp:1.0
+    container_name:          letsencrypt
+    restart:                 always
+    environment:
+      WEBROOT:               /data/letsencrypt
+      LISTEN:                '[::]:80'
+      VIRTUAL_HOST:          http://cloud.blender.org/.well-known/*, http://stats.cloud.blender.org/.well-known/*
+      VIRTUAL_HOST_WEIGHT:   30
+    volumes:
+      - /data/letsencrypt:/data/letsencrypt
+
+  haproxy:
+    # This image is defined in blender-cloud/docker/haproxy
+    image:                   armadillica/haproxy:1.6.7
+    container_name:          haproxy
+    restart:                 always
+    ports:
+      - "443:443"
+      - "80:80"
+    environment:
+      - ADDITIONAL_SERVICES=docker:blender_cloud,docker:letsencrypt,docker:kibana
+      - CERT_FOLDER=/certs/
+      - TIMEOUT=connect 5s, client 5m, server 10m
+      - SSL_BIND_CIPHERS=ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
+      - SSL_BIND_OPTIONS=no-sslv3
+      - EXTRA_GLOBAL_SETTINGS=tune.ssl.default-dh-param 2048
+    depends_on:
+      - blender_cloud
+      - letsencrypt
+      - kibana
+    volumes:
+      - '/data/certs:/certs'
+      - /var/run/docker.sock:/var/run/docker.sock

docker/elastic/Dockerfile-elastic

@@ -0,0 +1,10 @@
+FROM docker.elastic.co/elasticsearch/elasticsearch:6.1.1
+LABEL maintainer Sybren A. Stüvel <sybren@blender.studio>
+
+RUN elasticsearch-plugin remove --purge  x-pack
+
+ADD elasticsearch.yml jvm.options /usr/share/elasticsearch/config/
+
+USER root
+RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/config/
+USER elasticsearch

docker/elastic/Dockerfile-kibana

@@ -0,0 +1,6 @@
+FROM docker.elastic.co/kibana/kibana:6.1.1
+LABEL maintainer Sybren A. Stüvel <sybren@blender.studio>
+
+RUN bin/kibana-plugin remove x-pack
+ADD kibana.yml /usr/share/kibana/config/kibana.yml
+RUN kibana 2>&1 | grep -m 1 "Optimization of .* complete"

docker/elastic/build.sh

@@ -0,0 +1,15 @@
+#!/bin/bash -e
+
+# When updating this, also update the versions in Dockerfile-*, and make sure that
+# it matches the versions of the elasticsearch and elasticsearch_dsl packages
+# used in Pillar. Those don't have to match exactly, but the major version should.
+VERSION=6.1.1
+
+docker build -t armadillica/elasticsearch:${VERSION} -f Dockerfile-elastic .
+docker build -t armadillica/kibana:${VERSION} -f Dockerfile-kibana .
+
+docker tag armadillica/elasticsearch:${VERSION} armadillica/elasticsearch:latest
+docker tag armadillica/kibana:${VERSION} armadillica/kibana:latest
+
+echo "Done, built armadillica/elasticsearch:${VERSION} and armadillica/kibana:${VERSION}"
+echo "Also tagged as armadillica/elasticsearch:latest and armadillica/kibana:latest"

docker/elastic/elasticsearch.yml

@@ -0,0 +1,7 @@
+cluster.name: "blender-cloud"
+network.host: 0.0.0.0
+
+# minimum_master_nodes need to be explicitly set when bound on a public IP
+# set to 1 to allow single node clusters
+# Details: https://github.com/elastic/elasticsearch/pull/17288
+discovery.zen.minimum_master_nodes: 1

docker/elastic/jvm.options

@@ -0,0 +1,112 @@
+## JVM configuration
+
+################################################################
+## IMPORTANT: JVM heap size
+################################################################
+##
+## You should always set the min and max JVM heap
+## size to the same value. For example, to set
+## the heap to 4 GB, set:
+##
+## -Xms4g
+## -Xmx4g
+##
+## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
+## for more information
+##
+################################################################
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+# Sybren: uncommented so that we can set those options using the ES_JAVA_OPTS environment variable.
+#-Xms512m
+#-Xmx512m
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+## GC configuration
+-XX:+UseConcMarkSweepGC
+-XX:CMSInitiatingOccupancyFraction=75
+-XX:+UseCMSInitiatingOccupancyOnly
+
+## optimizations
+
+# pre-touch memory pages used by the JVM during initialization
+-XX:+AlwaysPreTouch
+
+## basic
+
+# force the server VM (remove on 32-bit client JVMs)
+-server
+
+# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
+-Xss1m
+
+# set to headless, just in case
+-Djava.awt.headless=true
+
+# ensure UTF-8 encoding by default (e.g. filenames)
+-Dfile.encoding=UTF-8
+
+# use our provided JNA always versus the system one
+-Djna.nosys=true
+
+# use old-style file permissions on JDK9
+-Djdk.io.permissionsUseCanonicalPath=true
+
+# flags to configure Netty
+-Dio.netty.noUnsafe=true
+-Dio.netty.noKeySetOptimization=true
+-Dio.netty.recycler.maxCapacityPerThread=0
+
+# log4j 2
+-Dlog4j.shutdownHookEnabled=false
+-Dlog4j2.disable.jmx=true
+-Dlog4j.skipJansi=true
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+-XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps
+# ensure the directory exists and has sufficient space
+#-XX:HeapDumpPath=${heap.dump.path}
+
+## GC logging
+
+#-XX:+PrintGCDetails
+#-XX:+PrintGCTimeStamps
+#-XX:+PrintGCDateStamps
+#-XX:+PrintClassHistogram
+#-XX:+PrintTenuringDistribution
+#-XX:+PrintGCApplicationStoppedTime
+
+# log GC status to a file with time stamps
+# ensure the directory exists
+#-Xloggc:${loggc}
+
+# By default, the GC log file will not rotate.
+# By uncommenting the lines below, the GC log file
+# will be rotated every 128MB at most 32 times.
+#-XX:+UseGCLogFileRotation
+#-XX:NumberOfGCLogFiles=32
+#-XX:GCLogFileSize=128M
+
+# Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.
+# If documents were already indexed with unquoted fields in a previous version
+# of Elasticsearch, some operations may throw errors.
+#
+# WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided
+# only for migration purposes.
+#-Delasticsearch.json.allow_unquoted_field_names=true

docker/elastic/kibana.yml

@@ -0,0 +1,8 @@
+---
+
+server.name: kibana
+server.host: "0"
+elasticsearch.url: http://elasticsearch:9200
+
+# Hide dev tools
+console.enabled: false

docker/full_rebuild.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+
+set -xe
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+cd $DIR/1_base
+bash build.sh
+
+cd $DIR/2_buildpy
+bash build.sh
+
+cd $DIR/3_buildwheels
+bash build.sh
+
+cd $DIR/4_run
+bash build.sh

docker/haproxy/Dockerfile

@@ -0,0 +1,5 @@
+FROM dockercloud/haproxy:1.6.7
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+# Fix https://talosintelligence.com/vulnerability_reports/TALOS-2019-0782
+RUN sed 's/root::/root:!:/' -i /etc/shadow

docker/haproxy/build.sh

@@ -0,0 +1,10 @@
+#!/bin/bash -e
+
+# When updating this, also update the version in Dockerfile
+VERSION=1.6.7
+
+docker build -t armadillica/haproxy:${VERSION} .
+docker tag armadillica/haproxy:${VERSION} armadillica/haproxy:latest
+
+echo "Done, built armadillica/haproxy:${VERSION}"
+echo "Also tagged as armadillica/haproxy:latest"

docker/mongo-backup.cron

@@ -0,0 +1,5 @@
+# Change to suit your needs, then place in /etc/cron.d/mongo-backup
+# (so remove the .cron from the name)
+
+MAILTO=yourname@youraddress.org
+30 5 * * * root /root/docker/mongo-backup.sh

docker/mongo-backup.sh

@@ -0,0 +1,28 @@
+#!/bin/bash -e
+
+BACKUPDIR=/data/storage/db-bak
+DATE=$(date +'%Y-%m-%d-%H%M')
+ARCHIVE=$BACKUPDIR/mongo-live-$DATE.tar.xz
+
+# Just a sanity check before we give it to 'rm -rf'
+if [ -z "$DATE" ]; then
+    echo "Empty string found where the date should be, aborting."
+    exit 1
+fi
+
+
+# /data/db-bak in Docker is /data/storage/db-bak on the host.
+docker exec mongo mongodump -d cloud \
+    --out /data/db-bak/dump-$DATE \
+    --excludeCollection tokens \
+    --excludeCollection flamenco_task_logs \
+    --quiet
+
+cd $BACKUPDIR
+tar -Jcf $ARCHIVE dump-$DATE/
+rm -rf dump-$DATE
+
+TO_DELETE="$(ls $BACKUPDIR/mongo-live-*.tar.xz | head -n -7)"
+[ -z "$TO_DELETE" ] || rm "$TO_DELETE"
+
+rsync -a $BACKUPDIR/mongo-live-*.tar.xz cloud-backup@swami-direct.blender.cloud:

docker/renew-letsencrypt.sh

@@ -0,0 +1,27 @@
+#!/bin/bash -e
+
+# First time creating a certificate for a domain, use:
+# certbot certonly --webroot -w /data/letsencrypt -d $DOMAINNAME
+
+cd /data/letsencrypt
+
+certbot renew
+
+echo
+echo "Recreating HAProxy certificates"
+
+for certdir in /etc/letsencrypt/live/*; do
+    domain=$(basename $certdir)
+    echo "  - $domain"
+
+    cat $certdir/privkey.pem $certdir/fullchain.pem > $domain.pem
+    mv $domain.pem /data/certs/
+done
+
+
+echo
+echo -n "Restarting "
+docker restart haproxy
+
+echo "Certificate renewal completed."
+

gulp

@@ -0,0 +1,33 @@
+#!/bin/bash -ex
+
+GULP=./node_modules/.bin/gulp
+
+function install() {
+    npm install
+    touch $GULP  # installer doesn't always touch this after a build, so we do.
+}
+
+# Rebuild Gulp if missing or outdated.
+[ -e $GULP ] || install
+[ gulpfile.js -nt $GULP ] && install
+
+if [ "$1" == "watch" ]; then
+    # Treat "gulp watch" as "gulp && gulp watch"
+    $GULP
+elif [ "$1" == "all" ]; then
+    pushd .
+    # This is useful when building the Blender Cloud project for the first time.
+    # Run ./gulp in all depending projects (pillar, attract, flamenco, pillar-svnman)
+    declare -a repos=("pillar" "attract" "flamenco" "pillar-svnman")
+    for r in "${repos[@]}"
+    do
+        cd ../$r
+        ./gulp
+    done
+    popd
+    # Run "gulp" once inside the repo
+    $GULP
+    exit 1
+fi
+
+exec $GULP "$@"

gulpfile.js

@@ -0,0 +1,130 @@
+let argv         = require('minimist')(process.argv.slice(2));
+let autoprefixer = require('gulp-autoprefixer');
+let cache        = require('gulp-cached');
+let chmod        = require('gulp-chmod');
+let concat       = require('gulp-concat');
+let git          = require('gulp-git');
+let gulp         = require('gulp');
+let gulpif       = require('gulp-if');
+let pug          = require('gulp-pug');
+let plumber      = require('gulp-plumber');
+let rename       = require('gulp-rename');
+let sass         = require('gulp-sass');
+let sourcemaps   = require('gulp-sourcemaps');
+let uglify       = require('gulp-uglify-es').default;
+
+let enabled = {
+    uglify: argv.production,
+    maps: !argv.production,
+    failCheck: !argv.production,
+    prettyPug: !argv.production,
+    cachify: !argv.production,
+    cleanup: argv.production,
+    chmod: argv.production,
+};
+
+let destination = {
+    css: 'cloud/static/assets/css',
+    pug: 'cloud/templates',
+    js: 'cloud/static/assets/js',
+}
+
+let source = {
+    pillar: '../pillar/'
+}
+
+
+/* CSS */
+gulp.task('styles', function(done) {
+    gulp.src('src/styles/**/*.sass')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.maps, sourcemaps.init()))
+        .pipe(sass({
+            outputStyle: 'compressed'}
+            ))
+        .pipe(autoprefixer("last 3 versions"))
+        .pipe(gulpif(enabled.maps, sourcemaps.write(".")))
+        .pipe(gulp.dest(destination.css));
+    done();
+});
+
+
+/* Templates - Pug */
+gulp.task('templates', function(done) {
+    gulp.src('src/templates/**/*.pug')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.cachify, cache('templating')))
+        .pipe(pug({
+            pretty: enabled.prettyPug
+        }))
+        .pipe(gulp.dest(destination.pug));
+    // TODO(venomgfx): please check why 'gulp watch' doesn't pick up on .txt changes.
+    gulp.src('src/templates/**/*.txt')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.cachify, cache('templating')))
+        .pipe(gulp.dest(destination.pug));
+    done();
+});
+
+
+/* Tutti gets built by Pillar. See gulpfile.js in pillar.*/
+
+
+/* Individual Uglified Scripts */
+gulp.task('scripts', function(done) {
+    gulp.src('src/scripts/*.js')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.cachify, cache('scripting')))
+        .pipe(gulpif(enabled.maps, sourcemaps.init()))
+        .pipe(gulpif(enabled.uglify, uglify()))
+        .pipe(rename({suffix: '.min'}))
+        .pipe(gulpif(enabled.maps, sourcemaps.write(".")))
+        .pipe(gulpif(enabled.chmod, chmod(0o644)))
+        .pipe(gulp.dest(destination.js));
+    done();
+});
+
+
+// While developing, run 'gulp watch'
+gulp.task('watch',function(done) {
+    let watchStyles = [
+        'src/styles/**/*.sass',
+        source.pillar + 'src/styles/**/*.sass',
+    ];
+
+    let watchScripts = [
+        'src/scripts/**/*.js',
+        source.pillar + 'src/scripts/**/*.js',
+    ];
+
+    let watchTemplates = [
+        'src/templates/**/*.pug',
+        source.pillar + 'src/templates/**/*.pug',
+    ];
+
+    gulp.watch(watchStyles, gulp.series('styles'));
+    gulp.watch(watchScripts, gulp.series('scripts'));
+    gulp.watch(watchTemplates, gulp.series('templates'));
+    done();
+});
+
+
+// Erases all generated files in output directories.
+gulp.task('cleanup', function(done) {
+    let paths = [];
+    for (attr in destination) {
+        paths.push(destination[attr]);
+    }
+
+    git.clean({ args: '-f -X ' + paths.join(' ') }, function (err) {
+        if(err) throw err;
+    });
+    done();
+});
+
+
+// Run 'gulp' to build everything at once
+let tasks = [];
+if (enabled.cleanup) tasks.push('cleanup');
+
+gulp.task('default', gulp.parallel(tasks.concat(['styles', 'templates', 'scripts'])));

jwtkeys/README.md

@@ -0,0 +1,6 @@
+# Flamenco Server JWT keys
+
+To generate a keypair for `ES256`:
+
+    openssl ecparam -genkey -name prime256v1 -noout -out es256-private.pem
+    openssl ec -in es256-private.pem -pubout -out es256-public.pem

manage.py

@@ -1,40 +1,7 @@
 #!/usr/bin/env python
 
-from __future__ import print_function
-
-import logging
-from flask import current_app
 from pillar import cli
-from pillar.cli import manager_maintenance
-from cloud import app
-
-log = logging.getLogger(__name__)
-
-
-@manager_maintenance.command
-def reconcile_subscribers():
-    """For every user, check their subscription status with the store."""
-    from pillar.auth.subscriptions import fetch_user
-
-    users_coll = current_app.data.driver.db['users']
-    unsubscribed_users = []
-    for user in users_coll.find({'roles': 'subscriber'}):
-        print('Processing %s' % user['email'])
-        print('  Checking subscription')
-        user_store = fetch_user(user['email'])
-        if user_store['cloud_access'] == 0:
-            print('    Removing subscriber role')
-            users_coll.update(
-                {'_id': user['_id']},
-                {'$pull': {'roles': 'subscriber'}})
-            unsubscribed_users.append(user['email'])
-
-    if not unsubscribed_users:
-        return
-
-    print('The following users have been unsubscribed')
-    for user in unsubscribed_users:
-        print(user)
+from runserver import app
 
 cli.manager.app = app
 cli.manager.run()

package.json

@@ -0,0 +1,32 @@
+{
+	"name": "blender-cloud",
+	"license": "GPL-2.0+",
+	"author": "Blender Institute",
+	"repository": {
+		"type": "git",
+		"url": "git://git.blender.org/blender-cloud.git"
+	},
+	"devDependencies": {
+		"gulp": "~4.0",
+		"gulp-autoprefixer": "~6.0.0",
+		"gulp-cached": "~1.1.1",
+		"gulp-chmod": "~2.0.0",
+		"gulp-concat": "~2.6.1",
+		"gulp-if": "^2.0.2",
+		"gulp-git": "~2.8.0",
+		"gulp-plumber": "~1.2.0",
+		"gulp-pug": "~4.0.1",
+		"gulp-rename": "~1.4.0",
+		"gulp-sass": "~4.1.0",
+		"gulp-sourcemaps": "~2.6.4",
+		"gulp-uglify-es": "^1.0.4",
+		"minimist": "^1.2.0"
+	},
+	"dependencies": {
+		"bootstrap": "^4.1.3",
+		"jquery": "^3.3.1",
+		"natives": "^1.1.6",
+		"popper.js": "^1.14.4",
+		"video.js": "^7.2.2"
+	}
+}