Start of "production videos", a.k.a. tagged assets overview

Tagged assets are shown in a list per tag. The list is dynamically
loaded with JavaScript.

.gitignore

@@ -3,6 +3,12 @@
 .coverage
 *.pyc
 __pycache__
+*.js.map
+*.css.map
+
+/cloud/templates/
+/cloud/static/assets/
+node_modules/
 
 /config_local.py
 
@@ -12,4 +18,9 @@ __pycache__
 /.eggs/
 /dump/
 /google_app*.json
-/docker/3_run/wheelhouse/
+/docker/2_buildpy/python/
+/docker/4_run/wheelhouse/
+/docker/4_run/deploy/
+/celerybeat-schedule.bak
+/celerybeat-schedule.dat
+/celerybeat-schedule.dir

README.md

@@ -1,43 +1,97 @@
 # Blender Cloud
 
-Welcome to the [Blender Cloud](https://cloud.blender.org) code repo!
-Blender Cloud runs on the [Pillar](https://pillarframework.org) framework.
+Welcome to the [Blender Cloud](https://cloud.blender.org/) code repo!
+Blender Cloud runs on the [Pillar](https://pillarframework.org/) framework.
 
-## Quick setup
-Set up a node with these commands. Note that that script is already outdated...
+## Development setup
+Jumpstart Blender Cloud development with this simple guide.
+
+### System setup
+Blender Cloud relies on a number of services in order to run. Check out the [Pillar system setup](
+https://pillarframework.org/development/system_setup/#step-by-step-setup) to set this up.
+
+Add `cloud.local` to the `/etc/hosts` file on localhost. This is a development convention.
+
+### Check out the code
+Go to the local development directory and check out the following repositories, next to each other.
 
 ```
-#!/usr/bin/env bash
-
-mkdir -p /data/git
-mkdir -p /data/storage
-mkdir -p /data/config
-mkdir -p /data/certs
-
-sudo apt-get update
-sudo apt-get -y install python-pip
-pip install docker-compose
-
-cd /data/git
+cd /home/guest/Developer
 git clone git://git.blender.org/pillar-python-sdk.git
-git clone git://git.blender.org/pillar-server.git pillar
+git clone git://git.blender.org/pillar.git
 git clone git://git.blender.org/attract.git
 git clone git://git.blender.org/flamenco.git
+git clone git://git.blender.org/pillar-svnman.git
 git clone git://git.blender.org/blender-cloud.git
 ```
 
-## Deploying to production server
+### Initial setup and configuration
 
-First of all, add those aliases to the `[alias]` section of your `~/.gitconfig`
+Create a virtualenv for the project and install the requirements:
+
+```
+cd blender-cloud
+mkvirtualenv blender-cloud -p python3.6
+pip install -r requirements-dev.txt
+```
+
+Build assets and templates for all Blender Cloud dependencies using Gulp.
+
+```
+./gulp all
+```
+
+Make a copy of the config_local example, which will be further edited as the application is
+configured.
+
+```
+cp config_local.example.py config_local.py
+```
+
+Setup the database with the initial collections and the admin user.
+
+```
+./manage.py setup setup_db your_email
+```
+
+The command will return the following message:
+
+```
+Created project <project_id> for user <user_id>
+```
+
+Copy the value of `<project_id>` and assign it as value for `MAIN_PROJECT_ID`.
+
+Run the application:
+
+```
+./manage.py runserver
+```
+
+
+## Development
+
+When ready to commit, change the remotes to work over SSH. For example:
+
+`git remote set-url origin git@git.blender.org:blender-cloud.git`
+
+For more information, check out [this guide](https://wiki.blender.org/wiki/Tools/Git#Commit_Access).
+
+
+## Preparing the production branch for deployment
+
+All revisions to deploy to production should be on the `production` branches of all the relevant
+repositories.
+
+Make sure you have these aliases in the `[alias]` section of your `~/.gitconfig`:
 
 ```
 prod = "!git checkout production && git fetch origin production && gitk --all"
 ff = "merge --ff-only"
-pp = "!git push && if [ -e deploy.sh ]; then ./deploy.sh; fi && git checkout master"
 ```
 
-The following commands should be executed for each subproject; specifically for
-Pillar and Attract:
+The following commands should be executed for each (sub)project; specifically for
+the current repository, Pillar, Attract, Flamenco, and Pillar-SVNMan:
 
 ```
 cd $projectdir
@@ -60,35 +114,19 @@ git ff master
 # Run tests again
 py.test
 
-# Push the production branch and run dummy deploy script.
-git pp  # pp = "Push to Production"
-
-# The above alias waits for [ENTER] until all deploys are done.
-# Let it wait, perform the other commands in another terminal.
+# Push the production branch.
+git push
 ```
 
-Now follow the above receipe on the Blender Cloud project as well.
-Contrary to the subprojects, `git pp` will actually perform the deploy
-for real.
+## Deploying to production server
 
-Now you can press `[ENTER]` in the Pillar and Attract terminals that
-were still waiting for it.
+```
+workon blender-cloud  # activate your virtualenv
+cd $projectdir/deploy
+./2docker.sh
+./build-all.sh  # or ./build-quick.sh
+./2server.sh servername
+```
 
-After everything is done, your (sub)projects should all be back on
-the master branch.
-
-
-## Updating dependencies via Docker images
-
-To update dependencies that need compiling, you need the `2_build` docker
-container. To rebuild the lot, run `docker/build.sh`.
-
-Follow these steps to deploy the new container on production:
-
-1. run `docker/build.sh`
-2. `docker push armadillica/blender_cloud`
-
-On the production machine:
-
-1. `docker pull armadillica/blender_cloud`
-2. `docker-compose up -d` (from the `/data/git/blender-cloud/docker` directory)
+To deploy another branch than `production`, do `export DEPLOY_BRANCH=otherbranch` before starting
+the above commands.

cloud/__init__.py

@@ -0,0 +1,103 @@
+import logging
+
+import flask
+from werkzeug.local import LocalProxy
+
+from pillar.api.utils import authorization
+from pillar.extension import PillarExtension
+
+EXTENSION_NAME = 'cloud'
+
+
+class CloudExtension(PillarExtension):
+    has_context_processor = True
+    user_roles = {'subscriber-pro', 'has_subscription'}
+    user_roles_indexable = {'subscriber-pro', 'has_subscription'}
+
+    user_caps = {
+        'has_subscription': {'can-renew-subscription'},
+    }
+
+    def __init__(self):
+        self._log = logging.getLogger('%s.CloudExtension' % __name__)
+
+    @property
+    def name(self):
+        return EXTENSION_NAME
+
+    def flask_config(self):
+        """Returns extension-specific defaults for the Flask configuration.
+
+        Use this to set sensible default values for configuration settings
+        introduced by the extension.
+
+        :rtype: dict
+        """
+
+        # Just so that it registers the management commands.
+        from . import cli
+
+        return {
+            'EXTERNAL_SUBSCRIPTIONS_MANAGEMENT_SERVER': 'https://store.blender.org/api/',
+            'EXTERNAL_SUBSCRIPTIONS_TIMEOUT_SECS': 10,
+            'BLENDER_ID_WEBHOOK_USER_CHANGED_SECRET': 'oos9wah1Zoa0Yau6ahThohleiChephoi',
+            'NODE_TAGS': ['animation', 'modelling', 'rigging'],
+        }
+
+    def eve_settings(self):
+        """Returns extensions to the Eve settings.
+
+        Currently only the DOMAIN key is used to insert new resources into
+        Eve's configuration.
+
+        :rtype: dict
+        """
+
+        return {}
+
+    def blueprints(self):
+        """Returns the list of top-level blueprints for the extension.
+
+        These blueprints will be mounted at the url prefix given to
+        app.load_extension().
+
+        :rtype: list of flask.Blueprint objects.
+        """
+        from . import routes
+        import cloud.stats.routes
+        return [
+            routes.blueprint,
+            cloud.stats.routes.blueprint,
+        ]
+
+    @property
+    def template_path(self):
+        import os.path
+        return os.path.join(os.path.dirname(__file__), 'templates')
+
+    @property
+    def static_path(self):
+        import os.path
+        return os.path.join(os.path.dirname(__file__), 'static')
+
+    def context_processor(self):
+        return {
+            'current_user_is_subscriber': authorization.user_has_cap('subscriber')
+        }
+
+    def setup_app(self, app):
+        from . import routes, webhooks, eve_hooks, email
+
+        routes.setup_app(app)
+        app.register_api_blueprint(webhooks.blueprint, '/webhooks')
+        eve_hooks.setup_app(app)
+        email.setup_app(app)
+
+
+def _get_current_cloud():
+    """Returns the Cloud extension of the current application."""
+    return flask.current_app.pillar_extensions[EXTENSION_NAME]
+
+
+current_cloud = LocalProxy(_get_current_cloud)
+"""Cloud extension of the current app."""

cloud/cli.py

@@ -0,0 +1,129 @@
+#!/usr/bin/env python
+
+import logging
+from urllib.parse import urljoin
+
+from flask import current_app
+from flask_script import Manager
+import requests
+
+from pillar.cli import manager
+from pillar.api import service
+
+log = logging.getLogger(__name__)
+
+manager_cloud = Manager(
+    current_app, usage="Blender Cloud scripts")
+
+
+@manager_cloud.command
+def create_groups():
+    """Creates the admin/demo/subscriber groups."""
+
+    import pprint
+
+    group_ids = {}
+    groups_coll = current_app.db('groups')
+
+    for group_name in ['admin', 'demo', 'subscriber']:
+        if groups_coll.find({'name': group_name}).count():
+            log.info('Group %s already exists, skipping', group_name)
+            continue
+        result = groups_coll.insert_one({'name': group_name})
+        group_ids[group_name] = result.inserted_id
+
+    service.fetch_role_to_group_id_map()
+    log.info('Created groups:\n%s', pprint.pformat(group_ids))
+
+
+@manager_cloud.command
+def reconcile_subscribers():
+    """For every user, check their subscription status with the store."""
+
+    import threading
+    import concurrent.futures
+
+    from pillar.auth import UserClass
+    from pillar.api import blender_id
+    from pillar.api.blender_cloud.subscription import do_update_subscription
+
+    sessions = threading.local()
+
+    service.fetch_role_to_group_id_map()
+
+    users_coll = current_app.data.driver.db['users']
+    found = users_coll.find({'auth.provider': 'blender-id'})
+    count_users = found.count()
+    count_skipped = count_processed = 0
+    log.info('Processing %i users', count_users)
+
+    lock = threading.Lock()
+
+    real_current_app = current_app._get_current_object()
+
+    api_token = real_current_app.config['BLENDER_ID_USER_INFO_TOKEN']
+    api_url = real_current_app.config['BLENDER_ID_USER_INFO_API']
+
+    def do_user(idx, user):
+        nonlocal count_skipped, count_processed
+
+        log.info('Processing %i/%i %s', idx + 1, count_users, user['email'])
+
+        # Get the Requests session for this thread.
+        try:
+            sess = sessions.session
+        except AttributeError:
+            sess = sessions.session = requests.Session()
+
+        # Get the info from Blender ID
+        bid_user_id = blender_id.get_user_blenderid(user)
+        if not bid_user_id:
+            with lock:
+                count_skipped += 1
+            return
+
+        url = urljoin(api_url, bid_user_id)
+        resp = sess.get(url, headers={'Authorization': f'Bearer {api_token}'})
+
+        if resp.status_code == 404:
+            log.info('User %s with Blender ID %s not found, skipping', user['email'], bid_user_id)
+            with lock:
+                count_skipped += 1
+            return
+
+        if resp.status_code != 200:
+            log.error('Unable to reach Blender ID (code %d), aborting', resp.status_code)
+            with lock:
+                count_skipped += 1
+            return
+
+        bid_user = resp.json()
+        if not bid_user:
+            log.error('Unable to parse response for user %s, aborting', user['email'])
+            with lock:
+                count_skipped += 1
+            return
+
+        # Actually update the user, and do it thread-safe just to be sure.
+        with real_current_app.app_context():
+            local_user = UserClass.construct('', user)
+            with lock:
+                do_update_subscription(local_user, bid_user)
+                count_processed += 1
+
+    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
+        future_to_user = {executor.submit(do_user, idx, user): user
+                          for idx, user in enumerate(found)}
+        for future in concurrent.futures.as_completed(future_to_user):
+            user = future_to_user[future]
+            try:
+                future.result()
+            except Exception as ex:
+                log.exception('Error updating user %s', user)
+
+    log.info('Done reconciling %d subscribers', count_users)
+    log.info('    processed: %d', count_processed)
+    log.info('    skipped  : %d', count_skipped)
+
+
+manager.add_command("cloud", manager_cloud)

cloud/email.py

@@ -0,0 +1,33 @@
+import functools
+import logging
+
+import flask
+
+from pillar.auth import UserClass
+
+log = logging.getLogger(__name__)
+
+
+def queue_welcome_mail(user: UserClass):
+    """Queue a welcome email for execution by Celery."""
+    assert user.email
+    log.info('queueing welcome email to %s', user.email)
+
+    subject = 'Welcome to Blender Cloud'
+    render = functools.partial(flask.render_template, subject=subject, user=user)
+    text = render('emails/welcome.txt')
+    html = render('emails/welcome.html')
+
+    from pillar.celery import email_tasks
+    email_tasks.send_email.delay(user.full_name, user.email, subject, text, html)
+
+
+def user_subscription_changed(user: UserClass, *, grant_roles: set, revoke_roles: set):
+    if user.has_cap('subscriber') and 'has_subscription' in grant_roles:
+        log.info('user %s just got a new subscription', user.email)
+        queue_welcome_mail(user)
+
+
+def setup_app(app):
+    from pillar.api.blender_cloud import subscription
+    subscription.user_subscription_updated.connect(user_subscription_changed)

cloud/eve_hooks.py

@@ -0,0 +1,39 @@
+import logging
+import typing
+
+from pillar.auth import UserClass
+
+from . import email
+
+log = logging.getLogger(__name__)
+
+
+def welcome_new_user(user_doc: dict):
+    """Sends a welcome email to a new user."""
+
+    user_email = user_doc.get('email')
+    if not user_email:
+        log.warning('user %s has no email address', user_doc.get('_id', '-no-id-'))
+        return
+
+    # Only send mail to new users when they actually are subscribers.
+    user = UserClass.construct('', user_doc)
+    if not (user.has_cap('subscriber') or user.has_cap('can-renew-subscription')):
+        log.debug('user %s is new, but not a subscriber, so no email for them.', user_email)
+        return
+
+    email.queue_welcome_mail(user)
+
+
+def welcome_new_users(user_docs: typing.List[dict]):
+    """Sends a welcome email to new users."""
+
+    for user_doc in user_docs:
+        try:
+            welcome_new_user(user_doc)
+        except Exception:
+            log.exception('error sending welcome mail to user %s', user_doc)
+
+
+def setup_app(app):
+    app.on_inserted_users += welcome_new_users

cloud/routes.py

@@ -0,0 +1,473 @@
+import functools
+import json
+import logging
+import typing
+
+from flask_login import current_user, login_required
+import flask
+from flask import Blueprint, render_template, redirect, session, url_for, abort, flash
+from pillarsdk import Node, Project, User, exceptions as sdk_exceptions, Group
+from pillarsdk.exceptions import ResourceNotFound
+
+from pillar import current_app
+import pillar.api
+from pillar.web.users import forms
+from pillar.web.utils import system_util, get_file, current_user_is_authenticated
+from pillar.web.utils import attach_project_pictures
+from pillar.web.settings import blueprint as blueprint_settings
+from pillar.web.nodes.routes import url_for_node
+from pillar.web.nodes.custom.comments import render_comments_for_node
+from pillar.web.projects.routes import render_project
+from pillar.web.projects.routes import find_project_or_404
+
+
+blueprint = Blueprint('cloud', __name__)
+log = logging.getLogger(__name__)
+
+
+@blueprint.route('/')
+def homepage():
+    if current_user.is_anonymous:
+        return redirect(url_for('cloud.welcome'))
+
+    return render_template(
+        'homepage.html',
+        api=system_util.pillar_api(),
+        **_homepage_context(),
+    )
+
+
+def _homepage_context() -> dict:
+    """Returns homepage template context variables."""
+
+    # Get latest blog posts
+    api = system_util.pillar_api()
+    latest_posts = Node.all({
+        'projection': {
+            'name': 1,
+            'project': 1,
+            'node_type': 1,
+            'picture': 1,
+            'properties.url': 1,
+            'properties.content': 1,
+            'properties.attachments': 1
+        },
+
+        'where': {'node_type': 'post', 'properties.status': 'published'},
+        'embedded': {'project': 1},
+        'sort': '-_created',
+        'max_results': '3'
+    }, api=api)
+
+    # Append picture Files to last_posts
+    for post in latest_posts._items:
+        post.picture = get_file(post.picture, api=api)
+        post.url = url_for_node(node=post)
+
+    # Get latest assets added to any project
+    latest_assets = Node.latest('assets', api=api)
+
+    # Append picture Files to latest_assets
+    for asset in latest_assets._items:
+        asset.picture = get_file(asset.picture, api=api)
+        asset.url = url_for_node(node=asset)
+
+    # Get latest comments to any node
+    latest_comments = Node.latest('comments', api=api)
+
+    # Get a list of random featured assets
+    random_featured = get_random_featured_nodes()
+
+    # Parse results for replies
+    to_remove = []
+
+    @functools.lru_cache()
+    def _find_parent(parent_node_id) -> Node:
+        return Node.find(parent_node_id,
+                         {'projection': {
+                             '_id': 1,
+                             'name': 1,
+                             'node_type': 1,
+                             'project': 1,
+                             'properties.url': 1,
+                         }},
+                         api=api)
+
+    for idx, comment in enumerate(latest_comments._items):
+        if comment.properties.is_reply:
+            try:
+                comment.attached_to = _find_parent(comment.parent.parent)
+            except ResourceNotFound:
+                # Remove this comment
+                to_remove.append(idx)
+        else:
+            comment.attached_to = comment.parent
+
+    for idx in reversed(to_remove):
+        del latest_comments._items[idx]
+
+    for comment in latest_comments._items:
+        if not comment.attached_to:
+            continue
+        comment.attached_to.url = url_for_node(node=comment.attached_to)
+        comment.url = url_for_node(node=comment)
+
+    main_project = Project.find(current_app.config['MAIN_PROJECT_ID'], api=api)
+    main_project.picture_header = get_file(main_project.picture_header, api=api)
+
+    # Merge latest assets and comments into one activity stream.
+    def sort_key(item):
+        return item._created
+
+    activity_stream = sorted(latest_assets._items, key=sort_key, reverse=True)
+
+    for node in activity_stream:
+        node.url = url_for_node(node=node)
+
+    return dict(
+        main_project=main_project,
+        latest_posts=latest_posts._items,
+        latest_comments=latest_comments._items,
+        activity_stream=activity_stream,
+        random_featured=random_featured)
+
+
+@blueprint.route('/login')
+def login():
+    from flask import request
+
+    if request.args.get('force'):
+        log.debug('Forcing logout of user before rendering login page.')
+        pillar.auth.logout_user()
+
+    next_after_login = request.args.get('next')
+
+    if not next_after_login:
+        next_after_login = request.referrer
+
+    session['next_after_login'] = next_after_login
+    return redirect(url_for('users.oauth_authorize', provider='blender-id'))
+
+
+@blueprint.route('/welcome')
+def welcome():
+    # Workaround to cache rendering of a page if user not logged in
+    @current_app.cache.cached(timeout=3600, unless=current_user_is_authenticated)
+    def render_page():
+        return render_template('welcome.html')
+
+    return render_page()
+
+
+@blueprint.route('/about')
+def about():
+    return render_template('about.html')
+
+
+@blueprint.route('/services')
+def services():
+    return render_template('services.html')
+
+
+@blueprint.route('/stats')
+def stats():
+    return render_template('stats.html')
+
+
+@blueprint.route('/join')
+def join():
+    """Join page"""
+    return redirect('https://store.blender.org/product/membership/')
+
+
+@blueprint.route('/renew')
+def renew_subscription():
+    return render_template('renew_subscription.html')
+
+
+def get_projects(category):
+    """Utility to get projects based on category. Should be moved on the API
+    and improved with more extensive filtering capabilities.
+    """
+    api = system_util.pillar_api()
+    projects = Project.all({
+        'where': {
+            'category': category,
+            'is_private': False},
+        'sort': '-_created',
+    }, api=api)
+    for project in projects._items:
+        attach_project_pictures(project, api)
+    return projects
+
+
+@blueprint.route('/courses')
+def courses():
+    @current_app.cache.cached(timeout=3600, unless=current_user_is_authenticated)
+    def render_page():
+        projects = get_projects('course')
+        return render_template(
+            'projects_index_collection.html',
+            title='courses',
+            projects=projects._items,
+            api=system_util.pillar_api())
+
+    return render_page()
+
+
+@blueprint.route('/open-projects')
+def open_projects():
+    @current_app.cache.cached(timeout=3600, unless=current_user_is_authenticated)
+    def render_page():
+        projects = get_projects('film')
+        return render_template(
+            'projects_index_collection.html',
+            title='open-projects',
+            projects=projects._items,
+            api=system_util.pillar_api())
+
+    return render_page()
+
+
+@blueprint.route('/workshops')
+def workshops():
+    @current_app.cache.cached(timeout=3600, unless=current_user_is_authenticated)
+    def render_page():
+        projects = get_projects('workshop')
+        return render_template(
+            'projects_index_collection.html',
+            title='workshops',
+            projects=projects._items,
+            api=system_util.pillar_api())
+
+    return render_page()
+
+
+def get_random_featured_nodes() -> typing.List[dict]:
+    """Returns a list of project/node combinations for featured nodes.
+
+    A random subset of 3 featured nodes from all public projects is returned.
+    Assumes that the user actually has access to the public projects' nodes.
+
+    The dict is a node, with a 'project' key that contains a projected project.
+    """
+
+    proj_coll = current_app.db('projects')
+    featured_nodes = proj_coll.aggregate([
+        {'$match': {'is_private': False}},
+        {'$project': {'nodes_featured': True,
+                      'url': True,
+                      'name': True,
+                      'summary': True,
+                      'picture_square': True}},
+        {'$unwind': {'path': '$nodes_featured'}},
+        {'$sample': {'size': 3}},
+        {'$lookup': {'from': 'nodes',
+                     'localField': 'nodes_featured',
+                     'foreignField': '_id',
+                     'as': 'node'}},
+        {'$unwind': {'path': '$node'}},
+        {'$project': {'url': True,
+                      'name': True,
+                      'summary': True,
+                      'picture_square': True,
+                      'node._id': True,
+                      'node.name': True,
+                      'node.permissions': True,
+                      'node.picture': True,
+                      'node.properties.content_type': True,
+                      'node.properties.url': True}},
+    ])
+
+    featured_node_documents = []
+    api = system_util.pillar_api()
+    for node_info in featured_nodes:
+        # Turn the project-with-node doc into a node-with-project doc.
+        node_document = node_info.pop('node')
+        node_document['project'] = node_info
+
+        node = Node(node_document)
+        node.picture = get_file(node.picture, api=api)
+        node.url = url_for_node(node=node)
+        node.project.url = url_for('projects.view', project_url=node.project.url)
+        node.project.picture_square = get_file(node.project.picture_square, api=api)
+        featured_node_documents.append(node)
+
+    return featured_node_documents
+
+
+@blueprint_settings.route('/emails', methods=['GET', 'POST'])
+@login_required
+def emails():
+    """Main email settings.
+    """
+    if current_user.has_role('protected'):
+        return abort(404)  # TODO: make this 403, handle template properly
+    api = system_util.pillar_api()
+    user = User.find(current_user.objectid, api=api)
+
+    # Force creation of settings for the user (safely remove this code once
+    # implemented on account creation level, and after adding settings to all
+    # existing users)
+    if not user.settings:
+        user.settings = dict(email_communications=1)
+        user.update(api=api)
+
+    if user.settings.email_communications is None:
+        user.settings.email_communications = 1
+        user.update(api=api)
+
+    # Generate form
+    form = forms.UserSettingsEmailsForm(
+        email_communications=user.settings.email_communications)
+
+    if form.validate_on_submit():
+        try:
+            user.settings.email_communications = form.email_communications.data
+            user.update(api=api)
+            flash("Profile updated", 'success')
+        except sdk_exceptions.ResourceInvalid as e:
+            message = json.loads(e.content)
+            flash(message)
+
+    return render_template('users/settings/emails.html', form=form, title='emails')
+
+
+@blueprint_settings.route('/billing')
+@login_required
+def billing():
+    """View the subscription status of a user
+    """
+    from . import store
+
+    log.debug('START OF REQUEST')
+
+    if current_user.has_role('protected'):
+        return abort(404)  # TODO: make this 403, handle template properly
+
+    expiration_date = 'No subscription to expire'
+
+    # Classify the user based on their roles and capabilities
+    cap_subs = current_user.has_cap('subscriber')
+    if current_user.has_role('demo'):
+        user_cls = 'demo'
+    elif not cap_subs and current_user.has_cap('can-renew-subscription'):
+        # This user has an inactive but renewable subscription.
+        user_cls = 'subscriber-expired'
+    elif cap_subs:
+        if current_user.has_role('subscriber'):
+            # This user pays for their own subscription. Only in this case do we need to fetch
+            # the expiration date from the Store.
+            user_cls = 'subscriber'
+            store_user = store.fetch_subscription_info(current_user.email)
+            if store_user is None:
+                expiration_date = 'Unable to reach Blender Store to check'
+            else:
+                expiration_date = store_user['expiration_date'][:10]
+
+        elif current_user.has_role('org-subscriber'):
+            # An organisation pays for this subscription.
+            user_cls = 'subscriber-org'
+        else:
+            # This user gets the subscription cap from somewhere else (like an organisation).
+            user_cls = 'subscriber-other'
+    else:
+        user_cls = 'outsider'
+
+    return render_template(
+        'users/settings/billing.html',
+        user_cls=user_cls,
+        expiration_date=expiration_date,
+        title='billing')
+
+
+@blueprint.route('/terms-and-conditions')
+def terms_and_conditions():
+    return render_template('terms_and_conditions.html')
+
+
+@blueprint.route('/privacy')
+def privacy():
+    return render_template('privacy.html')
+
+
+@blueprint.route('/production')
+def production():
+    return render_template('production.html')
+
+
+@blueprint.route('/emails/welcome.send')
+@login_required
+def emails_welcome_send():
+    from cloud import email
+    email.queue_welcome_mail(current_user)
+    return f'queued mail to {current_user.email}'
+
+
+@blueprint.route('/emails/welcome.html')
+@login_required
+def emails_welcome_html():
+    return render_template('emails/welcome.html',
+                           subject='Welcome to Blender Cloud',
+                           user=current_user)
+
+
+@blueprint.route('/emails/welcome.txt')
+@login_required
+def emails_welcome_txt():
+    txt = render_template('emails/welcome.txt',
+                          subject='Welcome to Blender Cloud',
+                          user=current_user)
+    return flask.Response(txt, content_type='text/plain; charset=utf-8')
+
+
+@blueprint.route('/nodes/<string(length=24):node_id>/comments')
+def comments_for_node(node_id):
+    """Overrides the default render_comments_for_node.
+
+    This is done in order to extend can_post_comments by requiring the
+    subscriber capability.
+    """
+
+    api = system_util.pillar_api()
+
+    node = Node.find(node_id, api=api)
+    project = Project({'_id': node.project})
+    can_post_comments = project.node_type_has_method('comment', 'POST', api=api)
+    can_comment_override = flask.request.args.get('can_comment', 'True') == 'True'
+    can_post_comments = can_post_comments and can_comment_override and current_user.has_cap(
+        'subscriber')
+
+    return render_comments_for_node(node_id, can_post_comments=can_post_comments)
+
+
+@blueprint.route('/p/hero')
+def project_hero():
+    api = system_util.pillar_api()
+    project = find_project_or_404('hero',
+                                  embedded={'header_node': 1},
+                                  api=api)
+    # Load the header video file, if there is any.
+    header_video_file = None
+    header_video_node = None
+    if project.header_node and project.header_node.node_type == 'asset' and \
+            project.header_node.properties.content_type == 'video':
+        header_video_node = project.header_node
+        header_video_file = get_file(project.header_node.properties.file)
+        header_video_node.picture = get_file(header_video_node.picture)
+
+    pages = Node.all({
+        'where': {'project': project._id, 'node_type': 'page'},
+        'projection': {'name': 1}}, api=api)
+
+    return render_project(project, api,
+                          extra_context={'header_video_file': header_video_file,
+                                         'header_video_node': header_video_node,
+                                         'pages': pages._items,},
+                          template_name='projects/landing.html')
+
+
+def setup_app(app):
+    global _homepage_context
+    cached = app.cache.cached(timeout=300)
+    _homepage_context = cached(_homepage_context)

cloud/stats/__init__.py

@@ -0,0 +1,89 @@
+"""Interesting usage metrics"""
+
+from flask import current_app
+
+
+def count_nodes(query=None) -> int:
+    pipeline = [
+        {'$match': {'_deleted': {'$ne': 'true'}}},
+        {
+            '$lookup':
+                {
+                    'from': "projects",
+                    'localField': "project",
+                    'foreignField': "_id",
+                    'as': "project",
+                }
+        },
+        {
+            '$unwind':
+                {
+                    'path': '$project',
+                }
+        },
+        {
+            '$project':
+                {
+                    'project.is_private': 1,
+                }
+        },
+        {'$match': {'project.is_private': False}},
+        {'$count': 'tot'}
+    ]
+    c = current_app.db()['nodes']
+    # If we provide a query, we extend the first $match step in the aggregation pipeline with
+    # with the extra parameters (for example node_type)
+    if query:
+        pipeline[0]['$match'].update(query)
+    # Return either a list with one item or an empty list
+    r = list(c.aggregate(pipeline=pipeline))
+    count = 0 if not r else r[0]['tot']
+    return count
+
+
+def count_users(query=None) -> int:
+    u = current_app.db()['users']
+    return u.count(query)
+
+
+def count_blender_sync(query=None) -> int:
+    pipeline = [
+        # 0 Find all startups.blend that are not deleted
+        {
+            '$match': {
+                '_deleted': {'$ne': 'true'},
+                'name': 'startup.blend',
+            }
+        },
+        # 1 Group them per project (drops any duplicates)
+        {'$group': {'_id': '$project'}},
+        # 2 Join the project info
+        {
+            '$lookup':
+                {
+                    'from': "projects",
+                    'localField': "_id",
+                    'foreignField': "_id",
+                    'as': "project",
+                }
+        },
+        # 3 Unwind the project list (there is always only one project)
+        {
+            '$unwind':
+                {
+                    'path': '$project',
+                }
+        },
+        # 4 Find all home projects
+        {'$match': {'project.category': 'home'}},
+        {'$count': 'tot'}
+    ]
+    c = current_app.db()['nodes']
+    # If we provide a query, we extend the first $match step in the aggregation pipeline with
+    # with the extra parameters (for example _created)
+    if query:
+        pipeline[0]['$match'].update(query)
+    # Return either a list with one item or an empty list
+    r = list(c.aggregate(pipeline=pipeline))
+    count = 0 if not r else r[0]['tot']
+    return count

cloud/stats/routes.py

@@ -0,0 +1,57 @@
+import logging
+import datetime
+import functools
+
+from flask import Blueprint, jsonify
+from cloud.stats import count_nodes, count_users, count_blender_sync
+
+blueprint = Blueprint('cloud.stats', __name__, url_prefix='/s')
+
+log = logging.getLogger(__name__)
+
+
+@functools.lru_cache()
+def get_stats(before: datetime.datetime):
+    query_comments = {'node_type': 'comment'}
+    query_assets = {'node_type': 'asset'}
+
+    date_query = {}
+
+    if before:
+        date_query = {'_created': {'$lt': before}}
+        query_comments.update(date_query)
+        query_assets.update(date_query)
+
+    stats = {
+        'comments': count_nodes(query_comments),
+        'assets': count_nodes(query_assets),
+        'users_total': count_users(date_query),
+        'users_blender_sync': count_blender_sync(date_query),
+    }
+    return stats
+
+
+@blueprint.route('/')
+@blueprint.route('/before/<int:before>')
+def index(before: int=0):
+    """
+    This endpoint is queried on a daily basis by grafista to retrieve cloud usage
+    stats. For assets and comments we take into considerations only those who belong
+    to public projects.
+
+    These is the data we retrieve
+
+    - Comments count
+    - Assets count (video, images and files)
+    - Users count (subscribers count goes via store)
+    - Blender Sync users
+    """
+
+    # TODO: Implement project-level metrics (and update ad every child update)
+    if before:
+        before = datetime.datetime.strptime(str(before), '%Y%m%d')
+    else:
+        today = datetime.date.today()
+        before = datetime.datetime(today.year, today.month, today.day)
+
+    return jsonify(get_stats(before))

cloud/store.py

@@ -0,0 +1,72 @@
+"""Blender Store interface."""
+
+import logging
+import typing
+
+from pillar import current_app
+
+log = logging.getLogger(__name__)
+
+
+def fetch_subscription_info(email: str) -> typing.Optional[dict]:
+    """Returns the user info dict from the external subscriptions management server.
+
+    :returns: the store user info, or None if the user can't be found or there
+        was an error communicating. A dict like this is returned:
+        {
+            "shop_id": 700,
+            "cloud_access": 1,
+            "paid_balance": 314.75,
+            "balance_currency": "EUR",
+            "start_date": "2014-08-25 17:05:46",
+            "expiration_date": "2016-08-24 13:38:45",
+            "subscription_status": "wc-active",
+            "expiration_date_approximate": true
+        }
+    """
+
+    from requests.adapters import HTTPAdapter
+    import requests.exceptions
+
+    external_subscriptions_server = current_app.config['EXTERNAL_SUBSCRIPTIONS_MANAGEMENT_SERVER']
+
+    if log.isEnabledFor(logging.DEBUG):
+        import urllib.parse
+
+        log_email = urllib.parse.quote(email)
+        log.debug('Connecting to store at %s?blenderid=%s',
+                  external_subscriptions_server, log_email)
+
+    # Retry a few times when contacting the store.
+    s = requests.Session()
+    s.mount(external_subscriptions_server, HTTPAdapter(max_retries=5))
+
+    try:
+        r = s.get(external_subscriptions_server,
+                  params={'blenderid': email},
+                  verify=current_app.config['TLS_CERT_FILE'],
+                  timeout=current_app.config.get('EXTERNAL_SUBSCRIPTIONS_TIMEOUT_SECS', 10))
+    except requests.exceptions.ConnectionError as ex:
+        log.error('Error connecting to %s: %s', external_subscriptions_server, ex)
+        return None
+    except requests.exceptions.Timeout as ex:
+        log.error('Timeout communicating with %s: %s', external_subscriptions_server, ex)
+        return None
+    except requests.exceptions.RequestException as ex:
+        log.error('Some error communicating with %s: %s', external_subscriptions_server, ex)
+        return None
+
+    if r.status_code != 200:
+        log.warning("Error communicating with %s, code=%i, unable to check "
+                    "subscription status of user %s",
+                    external_subscriptions_server, r.status_code, email)
+        return None
+
+    store_user = r.json()
+
+    if log.isEnabledFor(logging.DEBUG):
+        import json
+        log.debug('Received JSON from store API: %s',
+                  json.dumps(store_user, sort_keys=False, indent=4))
+
+    return store_user

cloud/tagged/__init__.py

@@ -0,0 +1,3 @@
+"""Routes for fetching tagged assets."""
+
+

cloud/tagged/routes.py

@@ -0,0 +1,16 @@
+import logging
+import datetime
+import functools
+
+from flask import Blueprint, jsonify
+
+blueprint = Blueprint('cloud.tagged', __name__, url_prefix='/tagged')
+
+log = logging.getLogger(__name__)
+
+
+@blueprint.route('/')
+def index():
+    """Return all tagged assets as JSON, grouped by tag."""
+
+

cloud/webhooks.py

@@ -0,0 +1,215 @@
+"""Blender ID webhooks."""
+
+import functools
+import hashlib
+import hmac
+import json
+import logging
+import typing
+
+from flask import Blueprint, request
+import werkzeug.exceptions as wz_exceptions
+
+from pillar import current_app
+from pillar.api.blender_cloud import subscription
+from pillar.api.utils.authentication import create_new_user_document, make_unique_username
+from pillar.auth import UserClass
+
+blueprint = Blueprint('cloud-webhooks', __name__)
+log = logging.getLogger(__name__)
+WEBHOOK_MAX_BODY_SIZE = 1024 * 10  # 10 kB is large enough for
+
+
+def webhook_payload(hmac_secret: str) -> dict:
+    """Obtains the webhook payload from the request, verifying its HMAC.
+
+    :returns the webhook payload as dictionary.
+    """
+    # Check the content type
+    if request.content_type != 'application/json':
+        log.info('request from %s to %s had bad content type %s',
+                 request.remote_addr, request, request.content_type)
+        raise wz_exceptions.BadRequest('Content type not supported')
+
+    # Check the length of the body
+    if request.content_length > WEBHOOK_MAX_BODY_SIZE:
+        raise wz_exceptions.BadRequest('Request too large')
+    body = request.get_data()
+    if len(body) > request.content_length:
+        raise wz_exceptions.BadRequest('Larger body than Content-Length header')
+
+    # Validate the request
+    mac = hmac.new(hmac_secret.encode(), body, hashlib.sha256)
+    req_hmac = request.headers.get('X-Webhook-HMAC', '')
+    our_hmac = mac.hexdigest()
+    if not hmac.compare_digest(req_hmac, our_hmac):
+        log.info('request from %s to %s had bad HMAC %r, expected %r',
+                 request.remote_addr, request, req_hmac, our_hmac)
+        raise wz_exceptions.BadRequest('Bad HMAC')
+
+    try:
+        return json.loads(body)
+    except json.JSONDecodeError as ex:
+        log.warning('request from %s to %s had bad JSON: %s',
+                    request.remote_addr, request, ex)
+        raise wz_exceptions.BadRequest('Bad JSON')
+
+
+def score(wh_payload: dict, user: dict) -> int:
+    """Determine how likely it is that this is the correct user to modify.
+
+    :param wh_payload: the info we received from Blender ID;
+        see user_modified()
+    :param user: the user in our database
+    :return: the score for this user
+    """
+
+    bid_str = str(wh_payload['id'])
+    try:
+        match_on_bid = any((auth['provider'] == 'blender-id' and auth['user_id'] == bid_str)
+                           for auth in user['auth'])
+    except KeyError:
+        match_on_bid = False
+
+    match_on_old_email = user.get('email', 'none') == wh_payload.get('old_email', 'nothere')
+    match_on_new_email = user.get('email', 'none') == wh_payload.get('email', 'nothere')
+    return match_on_bid * 10 + match_on_old_email + match_on_new_email * 2
+
+
+def insert_or_fetch_user(wh_payload: dict) -> typing.Optional[dict]:
+    """Fetch the user from the DB or create it.
+
+    Only creates it if the webhook payload indicates they could actually use
+    Blender Cloud (i.e. demo or subscriber). This prevents us from creating
+    Cloud accounts for Blender Network users.
+
+    :returns the user document, or None when not created.
+    """
+
+    users_coll = current_app.db('users')
+    my_log = log.getChild('insert_or_fetch_user')
+
+    bid_str = str(wh_payload['id'])
+    email = wh_payload['email']
+
+    # Find the user by their Blender ID, or any of their email addresses.
+    # We use one query to find all matching users. This is done as a
+    # consistency check; if more than one user is returned, we know the
+    # database is inconsistent with Blender ID and can emit a warning
+    # about this.
+    query = {'$or': [
+        {'auth.provider': 'blender-id', 'auth.user_id': bid_str},
+        {'email': {'$in': [wh_payload['old_email'], email]}},
+    ]}
+    db_users = users_coll.find(query)
+    user_count = db_users.count()
+    if user_count > 1:
+        # Now we have to pay the price for finding users in one query; we
+        # have to prioritise them and return the one we think is most reliable.
+        calc_score = functools.partial(score, wh_payload)
+        best_score = max(db_users, key=calc_score)
+
+        my_log.error('%d users found for query %s, picking user %s (%s)',
+                     user_count, query, best_score['_id'], best_score['email'])
+        return best_score
+    if user_count:
+        db_user = db_users[0]
+        my_log.debug('found user %s', db_user['email'])
+        return db_user
+
+    # Pretend to create the user, so that we can inspect the resulting
+    # capabilities. This is more future-proof than looking at the list
+    # of roles in the webhook payload.
+    username = make_unique_username(email)
+    user_doc = create_new_user_document(email, bid_str, username,
+                                        provider='blender-id',
+                                        full_name=wh_payload['full_name'])
+
+    # Figure out the user's eventual roles. These aren't stored in the document yet,
+    # because that's handled by the badger service.
+    eventual_roles = [subscription.ROLES_BID_TO_PILLAR[r]
+                      for r in wh_payload.get('roles', [])
+                      if r in subscription.ROLES_BID_TO_PILLAR]
+    user_ob = UserClass.construct('', user_doc)
+    user_ob.roles = eventual_roles
+    user_ob.collect_capabilities()
+    create = (user_ob.has_cap('subscriber') or
+              user_ob.has_cap('can-renew-subscription') or
+              current_app.org_manager.user_is_unknown_member(email))
+    if not create:
+        my_log.info('Received update for unknown user %r without Cloud access (caps=%s)',
+                    wh_payload['old_email'], user_ob.capabilities)
+        return None
+
+    # Actually create the user in the database.
+    r, _, _, status = current_app.post_internal('users', user_doc)
+    if status != 201:
+        my_log.error('unable to create user %s: : %r %r', email, status, r)
+        raise wz_exceptions.InternalServerError('unable to create user')
+
+    user_doc.update(r)
+    my_log.info('created user %r = %s to allow immediate Cloud access', email, user_doc['_id'])
+    return user_doc
+
+
+@blueprint.route('/user-modified', methods=['POST'])
+def user_modified():
+    """Update the local user based on the info from Blender ID.
+
+    If the payload indicates the user has access to Blender Cloud (or at least
+    a renewable subscription), create the user if not already in our DB.
+
+    The payload we expect is a dictionary like:
+    {'id': 12345,  # the user's ID in Blender ID
+     'old_email': 'old@example.com',
+     'full_name': 'Harry',
+     'email': 'new@example'com,
+     'roles': ['role1', 'role2', …]}
+    """
+    my_log = log.getChild('user_modified')
+    my_log.debug('Received request from %s', request.remote_addr)
+
+    hmac_secret = current_app.config['BLENDER_ID_WEBHOOK_USER_CHANGED_SECRET']
+    payload = webhook_payload(hmac_secret)
+
+    my_log.info('payload: %s', payload)
+
+    # Update the user
+    db_user = insert_or_fetch_user(payload)
+    if not db_user:
+        my_log.info('Received update for unknown user %r', payload['old_email'])
+        return '', 204
+
+    # Use direct database updates to change the email and full name.
+    # Also updates the db_user dict so that local_user below will have
+    # the updated information.
+    updates = {}
+    if db_user['email'] != payload['email']:
+        my_log.info('User changed email from %s to %s', payload['old_email'], payload['email'])
+        updates['email'] = payload['email']
+        db_user['email'] = payload['email']
+
+    if db_user['full_name'] != payload['full_name']:
+        my_log.info('User changed full name from %r to %r',
+                    db_user['full_name'], payload['full_name'])
+        if payload['full_name']:
+            updates['full_name'] = payload['full_name']
+        else:
+            # Fall back to the username when the full name was erased.
+            updates['full_name'] = db_user['username']
+        db_user['full_name'] = updates['full_name']
+
+    if updates:
+        users_coll = current_app.db('users')
+        update_res = users_coll.update_one({'_id': db_user['_id']},
+                                           {'$set': updates})
+        if update_res.matched_count != 1:
+            my_log.error('Unable to find user %s to update, even though '
+                         'we found them by email address %s',
+                         db_user['_id'], payload['old_email'])
+
+    # Defer to Pillar to do the role updates.
+    local_user = UserClass.construct('', db_user)
+    subscription.do_update_subscription(local_user, payload)
+
+    return '', 204

cloud_share_img.py

@@ -0,0 +1,296 @@
+#!/usr/bin/env python3
+
+from __future__ import print_function
+
+"""CLI command for sharing an image via Blender Cloud.
+
+Assumes that you are logged in on Blender ID with the Blender ID Add-on.
+
+The user_config_dir and user_data_dir functions come from
+ https://github.com/ActiveState/appdirs/blob/master/appdirs.py and
+ are licensed under the MIT license.
+"""
+
+import argparse
+import json
+import mimetypes
+import os.path
+import pprint
+import sys
+import webbrowser
+
+from urllib.parse import urljoin
+
+import requests
+
+cli = argparse.Namespace()  # CLI args from argparser
+sess = requests.Session()
+IMAGE_SHARING_GROUP_NODE_NAME = 'Image sharing'
+
+if sys.platform.startswith('java'):
+    import platform
+
+    os_name = platform.java_ver()[3][0]
+    if os_name.startswith('Windows'):  # "Windows XP", "Windows 7", etc.
+        system = 'win32'
+    elif os_name.startswith('Mac'):  # "Mac OS X", etc.
+        system = 'darwin'
+    else:  # "Linux", "SunOS", "FreeBSD", etc.
+        # Setting this to "linux2" is not ideal, but only Windows or Mac
+        # are actually checked for and the rest of the module expects
+        # *sys.platform* style strings.
+        system = 'linux2'
+else:
+    system = sys.platform
+
+
+def request(method: str, rel_url: str, **kwargs) -> requests.Response:
+    kwargs.setdefault('auth', (cli.token, ''))
+    url = urljoin(cli.server_url, rel_url)
+    return sess.request(method, url, **kwargs)
+
+
+def get(rel_url: str, **kwargs) -> requests.Response:
+    return request('GET', rel_url, **kwargs)
+
+
+def post(rel_url: str, **kwargs) -> requests.Response:
+    return request('POST', rel_url, **kwargs)
+
+
+def find_user_id() -> str:
+    """Returns the current user ID."""
+
+    print(15 * '=', 'User info', 15 * '=')
+    resp = get('/api/users/me')
+    resp.raise_for_status()
+
+    user_info = resp.json()
+    print('You are logged in as %(full_name)s (%(_id)s)' % user_info)
+
+    return user_info['_id']
+
+
+def find_home_project_id() -> dict:
+    resp = get('/api/bcloud/home-project')
+    resp.raise_for_status()
+
+    proj = resp.json()
+    proj_id = proj['_id']
+    print('Your home project ID is %s' % proj_id)
+    return proj_id
+
+
+def find_image_sharing_group_id(home_project_id, user_id) -> str:
+    """Find the top-level image sharing group node."""
+
+    node_doc = {
+        'project': home_project_id,
+        'node_type': 'group',
+        'name': IMAGE_SHARING_GROUP_NODE_NAME,
+        'user': user_id,
+    }
+
+    resp = get('/api/nodes', params={'where': json.dumps(node_doc)})
+    resp.raise_for_status()
+    items = resp.json()['_items']
+
+    if not items:
+        print('Share group not found, creating one.')
+        node_doc.update({
+            'properties': {},
+        })
+        resp = post('/api/nodes', json=node_doc)
+        resp.raise_for_status()
+        share_group = resp.json()
+    else:
+        share_group = items[0]
+
+    # print('Share group:', share_group)
+    return share_group['_id']
+
+
+def upload_image():
+    user_id = find_user_id()
+    home_project_id = find_home_project_id()
+    group_id = find_image_sharing_group_id(home_project_id, user_id)
+    basename = os.path.basename(cli.imgfile)
+    print('Sharing group ID is %s' % group_id)
+
+    # Upload the image to the project.
+    print('Uploading %r' % cli.imgfile)
+    mimetype, _ = mimetypes.guess_type(cli.imgfile, strict=False)
+    with open(cli.imgfile, mode='rb') as infile:
+        resp = post('api/storage/stream/%s' % home_project_id,
+                    files={'file': (basename, infile, mimetype)})
+    resp.raise_for_status()
+    file_upload_resp = resp.json()
+    file_upload_status = file_upload_resp.get('_status') or file_upload_resp.get('status')
+    if file_upload_status != 'ok':
+        raise ValueError('Received bad status %s from Pillar: %s' %
+                         (file_upload_status, json.dumps(file_upload_resp)))
+    file_id = file_upload_resp['file_id']
+    print('File ID is', file_id)
+
+    # Create the asset node
+    asset_node = {
+        'project': home_project_id,
+        'node_type': 'asset',
+        'name': basename,
+        'parent': group_id,
+        'properties': {
+            'content_type': mimetype,
+            'file': file_id,
+        },
+    }
+    resp = post('api/nodes', json=asset_node)
+    resp.raise_for_status()
+    node_info = resp.json()
+    node_id = node_info['_id']
+    print('Created asset node', node_id)
+
+    # Share the node to get a public URL.
+    resp = post('api/nodes/%s/share' % node_id)
+    resp.raise_for_status()
+    share_info = resp.json()
+    print(json.dumps(share_info, indent=4))
+
+    url = share_info.get('short_link')
+    print('Opening %s in a browser' % url)
+    webbrowser.open_new_tab(url)
+
+
+def find_credentials():
+    """Finds BlenderID credentials.
+
+    :rtype: str
+    :returns: the authentication token to use.
+    """
+    import glob
+
+    # Find BlenderID profile file.
+    configpath = user_config_dir('blender', 'Blender Foundation', roaming=True)
+    found = glob.glob(os.path.join(configpath, '*'))
+    for confpath in reversed(sorted(found)):
+        profiles_path = os.path.join(confpath, 'config', 'blender_id', 'profiles.json')
+        if not os.path.exists(profiles_path):
+            continue
+
+        print('Reading credentials from %s' % profiles_path)
+        with open(profiles_path) as infile:
+            profiles = json.load(infile, encoding='utf8')
+        if profiles:
+            break
+    else:
+        print('Unable to find Blender ID credentials. Log in with the Blender ID add-on in '
+              'Blender first.')
+        raise SystemExit()
+
+    active_profile = profiles[u'active_profile']
+    profile = profiles[u'profiles'][active_profile]
+    print('Logging in as %s' % profile[u'username'])
+
+    return profile[u'token']
+
+
+def main():
+    global cli
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument('imgfile', help='The image file to share.')
+    parser.add_argument('-u', '--server-url', default='https://cloud.blender.org/',
+                        help='URL of the Flamenco server.')
+    parser.add_argument('-t', '--token',
+                        help='Authentication token to use. If not given, your token from the '
+                             'Blender ID add-on is used.')
+
+    cli = parser.parse_args()
+    if not cli.token:
+        cli.token = find_credentials()
+
+    upload_image()
+
+
+def user_config_dir(appname=None, appauthor=None, version=None, roaming=False):
+    r"""Return full path to the user-specific config dir for this application.
+        "appname" is the name of application.
+            If None, just the system directory is returned.
+        "appauthor" (only used on Windows) is the name of the
+            appauthor or distributing body for this application. Typically
+            it is the owning company name. This falls back to appname. You may
+            pass False to disable it.
+        "version" is an optional version path element to append to the
+            path. You might want to use this if you want multiple versions
+            of your app to be able to run independently. If used, this
+            would typically be "<major>.<minor>".
+            Only applied when appname is present.
+        "roaming" (boolean, default False) can be set True to use the Windows
+            roaming appdata directory. That means that for users on a Windows
+            network setup for roaming profiles, this user data will be
+            sync'd on login. See
+            <http://technet.microsoft.com/en-us/library/cc766489(WS.10).aspx>
+            for a discussion of issues.
+    Typical user config directories are:
+        Mac OS X:               same as user_data_dir
+        Unix:                   ~/.config/<AppName>     # or in $XDG_CONFIG_HOME, if defined
+        Win *:                  same as user_data_dir
+    For Unix, we follow the XDG spec and support $XDG_CONFIG_HOME.
+    That means, by default "~/.config/<AppName>".
+    """
+    if system in {"win32", "darwin"}:
+        path = user_data_dir(appname, appauthor, None, roaming)
+    else:
+        path = os.getenv('XDG_CONFIG_HOME', os.path.expanduser("~/.config"))
+        if appname:
+            path = os.path.join(path, appname)
+    if appname and version:
+        path = os.path.join(path, version)
+    return path
+
+
+def user_data_dir(appname=None, appauthor=None, version=None, roaming=False):
+    r"""Return full path to the user-specific data dir for this application.
+        "appname" is the name of application.
+            If None, just the system directory is returned.
+        "appauthor" (only used on Windows) is the name of the
+            appauthor or distributing body for this application. Typically
+            it is the owning company name. This falls back to appname. You may
+            pass False to disable it.
+        "version" is an optional version path element to append to the
+            path. You might want to use this if you want multiple versions
+            of your app to be able to run independently. If used, this
+            would typically be "<major>.<minor>".
+            Only applied when appname is present.
+        "roaming" (boolean, default False) can be set True to use the Windows
+            roaming appdata directory. That means that for users on a Windows
+            network setup for roaming profiles, this user data will be
+            sync'd on login. See
+            <http://technet.microsoft.com/en-us/library/cc766489(WS.10).aspx>
+            for a discussion of issues.
+    Typical user data directories are:
+        Mac OS X:               ~/Library/Application Support/<AppName>
+        Unix:                   ~/.local/share/<AppName>    # or in $XDG_DATA_HOME, if defined
+        Win XP (not roaming):   C:\Documents and Settings\<username>\Application Data\<AppAuthor>\<AppName>
+        Win XP (roaming):       C:\Documents and Settings\<username>\Local Settings\Application Data\<AppAuthor>\<AppName>
+        Win 7  (not roaming):   C:\Users\<username>\AppData\Local\<AppAuthor>\<AppName>
+        Win 7  (roaming):       C:\Users\<username>\AppData\Roaming\<AppAuthor>\<AppName>
+    For Unix, we follow the XDG spec and support $XDG_DATA_HOME.
+    That means, by default "~/.local/share/<AppName>".
+    """
+    if system == "win32":
+        raise RuntimeError("Sorry, Windows is not supported for now.")
+    elif system == 'darwin':
+        path = os.path.expanduser('~/Library/Application Support/')
+        if appname:
+            path = os.path.join(path, appname)
+    else:
+        path = os.getenv('XDG_DATA_HOME', os.path.expanduser("~/.local/share"))
+        if appname:
+            path = os.path.join(path, appname)
+    if appname and version:
+        path = os.path.join(path, version)
+    return path
+
+
+if __name__ == '__main__':
+    main()

config_local.example.py

@@ -0,0 +1,32 @@
+import os
+
+DEBUG = True
+
+BLENDER_ID_ENDPOINT = 'http://id.local:8000/'
+
+SERVER_NAME = 'cloud.local:5001'
+SCHEME = 'http'
+PILLAR_SERVER_ENDPOINT = f'{SCHEME}://{SERVER_NAME}/api/'
+
+os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = 'true'
+os.environ['PILLAR_MONGO_DBNAME'] = 'cloud'
+os.environ['PILLAR_MONGO_PORT'] = '27017'
+os.environ['PILLAR_MONGO_HOST'] = 'mongo'
+
+os.environ['PILLAR_SERVER_ENDPOINT'] = PILLAR_SERVER_ENDPOINT
+
+SECRET_KEY = '##DEFINE##'
+
+OAUTH_CREDENTIALS = {
+    'blender-id': {
+        'id': 'CLOUD-OF-SNOWFLAKES-42',
+        'secret': '##DEFINE##',
+    }
+}
+
+MAIN_PROJECT_ID = '##DEFINE##'
+URLER_SERVICE_AUTH_TOKEN = '##DEFINE##'
+
+ZENCODER_API_KEY = '##DEFINE##'
+ZENCODER_NOTIFICATIONS_SECRET = '##DEFINE##'
+ZENCODER_NOTIFICATIONS_URL = 'http://zencoderfetcher/'

deploy.sh

@@ -1,124 +0,0 @@
-#!/bin/bash -e
-
-# Deploys the current production branch to the production machine.
-PROJECT_NAME="blender-cloud"
-DOCKER_NAME="blender_cloud"
-REMOTE_ROOT="/data/git/${PROJECT_NAME}"
-
-SSH="ssh -o ClearAllForwardings=yes cloud.blender.org"
-
-# macOS does not support readlink -f, so we use greadlink instead
-if [[ `uname` == 'Darwin' ]]; then
-    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
-    readlink='greadlink'
-else
-    readlink='readlink'
-fi
-
-ROOT="$(dirname "$($readlink -f "$0")")"
-cd ${ROOT}
-
-# Check that we're on production branch.
-if [ $(git rev-parse --abbrev-ref HEAD) != "production" ]; then
-    echo "You are NOT on the production branch, refusing to deploy." >&2
-    exit 1
-fi
-
-# Check that production branch has been pushed.
-if [ -n "$(git log origin/production..production --oneline)" ]; then
-    echo "WARNING: not all changes to the production branch have been pushed."
-    echo "Press [ENTER] to continue deploying current origin/production, CTRL+C to abort."
-    read dummy
-fi
-
-function find_module()
-{
-    MODULE_NAME=$1
-    MODULE_DIR=$(python <<EOT
-from __future__ import print_function
-import os.path
-try:
-    import ${MODULE_NAME}
-except ImportError:
-    raise SystemExit('${MODULE_NAME} not found on Python path. Are you in the correct venv?')
-
-print(os.path.dirname(os.path.dirname(${MODULE_NAME}.__file__)))
-EOT
-)
-
-    if [ $(git -C $MODULE_DIR rev-parse --abbrev-ref HEAD) != "production" ]; then
-        echo "${MODULE_NAME}: ($MODULE_DIR) NOT on the production branch, refusing to deploy." >&2
-        exit 1
-    fi
-
-    echo $MODULE_DIR
-}
-
-# Find our modules
-PILLAR_DIR=$(find_module pillar)
-ATTRACT_DIR=$(find_module attract)
-FLAMENCO_DIR=$(find_module flamenco)
-
-echo "Pillar  : $PILLAR_DIR"
-echo "Attract : $ATTRACT_DIR"
-echo "Flamenco: $FLAMENCO_DIR"
-
-if [ -z "$PILLAR_DIR" -o -z "$ATTRACT_DIR" -o -z "$FLAMENCO_DIR" ];
-then
-    exit 1
-fi
-
-# SSH to cloud to pull all files in
-function git_pull() {
-    PROJECT_NAME="$1"
-    BRANCH="$2"
-    REMOTE_ROOT="/data/git/${PROJECT_NAME}"
-
-    echo "==================================================================="
-    echo "UPDATING FILES ON ${PROJECT_NAME}"
-    ${SSH} git -C ${REMOTE_ROOT} fetch origin ${BRANCH}
-    ${SSH} git -C ${REMOTE_ROOT} log origin/${BRANCH}..${BRANCH} --oneline
-    ${SSH} git -C ${REMOTE_ROOT} merge --ff-only origin/${BRANCH}
-}
-
-git_pull pillar-python-sdk master
-git_pull pillar production
-git_pull attract production
-git_pull flamenco production
-git_pull blender-cloud production
-
-# Update the virtualenv
-#${SSH} -t docker exec ${DOCKER_NAME} /data/venv/bin/pip install -U -r ${REMOTE_ROOT}/requirements.txt --exists-action w
-
-# RSync the world
-$ATTRACT_DIR/rsync_ui.sh
-$FLAMENCO_DIR/rsync_ui.sh
-./rsync_ui.sh
-
-# Notify Bugsnag of this new deploy.
-echo
-echo "==================================================================="
-GIT_REVISION=$(${SSH} git -C ${REMOTE_ROOT} describe --always)
-echo "Notifying Bugsnag of this new deploy of revision ${GIT_REVISION}."
-BUGSNAG_API_KEY=$(${SSH} python -c "\"import sys; sys.path.append('${REMOTE_ROOT}'); import config_local; print(config_local.BUGSNAG_API_KEY)\"")
-curl --data "apiKey=${BUGSNAG_API_KEY}&revision=${GIT_REVISION}" https://notify.bugsnag.com/deploy
-echo
-
-# Wait for [ENTER] to restart the server
-echo
-echo "==================================================================="
-echo "NOTE: If you want to edit config_local.py on the server, do so now."
-echo "NOTE: Press [ENTER] to continue and restart the server process."
-read dummy
-${SSH} docker exec ${DOCKER_NAME} apache2ctl graceful
-echo "Server process restarted"
-
-echo
-echo "==================================================================="
-echo "Clearing front page from Redis cache."
-${SSH} docker exec redis redis-cli DEL pwview//
-
-echo
-echo "==================================================================="
-echo "Deploy of ${PROJECT_NAME} is done."
-echo "==================================================================="

deploy/2docker.sh

@@ -0,0 +1,116 @@
+#!/bin/bash -e
+
+DEPLOY_BRANCH=${DEPLOY_BRANCH:-production}
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [[ `uname` == 'Darwin' ]]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+
+ROOT="$(dirname "$(dirname "$($readlink -f "$0")")")"
+DEPLOYDIR="$ROOT/docker/4_run/deploy"
+PROJECT_NAME="$(basename $ROOT)"
+
+if [ -e $DEPLOYDIR ]; then
+    echo "$DEPLOYDIR already exists, press [ENTER] to DESTROY AND DEPLOY, Ctrl+C to abort."
+    read dummy
+    rm -rf $DEPLOYDIR
+else
+    echo -n "Deploying to $DEPLOYDIR… "
+    echo "press [ENTER] to continue, Ctrl+C to abort."
+    read dummy
+fi
+
+cd ${ROOT}
+mkdir -p $DEPLOYDIR
+REMOTE_ROOT="$DEPLOYDIR/$PROJECT_NAME"
+
+if [ -z "$SKIP_BRANCH_CHECK" ]; then
+    # Check that we're on production branch.
+    if [ $(git rev-parse --abbrev-ref HEAD) != "$DEPLOY_BRANCH" ]; then
+        echo "You are NOT on the $DEPLOY_BRANCH branch, refusing to deploy." >&2
+        exit 1
+    fi
+
+    # Check that production branch has been pushed.
+    if [ -n "$(git log origin/$DEPLOY_BRANCH..$DEPLOY_BRANCH --oneline)" ]; then
+        echo "WARNING: not all changes to the $DEPLOY_BRANCH branch have been pushed."
+        echo "Press [ENTER] to continue deploying current origin/$DEPLOY_BRANCH, CTRL+C to abort."
+        read dummy
+    fi
+fi
+
+function find_module()
+{
+    MODULE_NAME=$1
+    MODULE_DIR=$(python <<EOT
+from __future__ import print_function
+import os.path
+try:
+    import ${MODULE_NAME}
+except ImportError:
+    raise SystemExit('${MODULE_NAME} not found on Python path. Are you in the correct venv?')
+
+print(os.path.dirname(os.path.dirname(${MODULE_NAME}.__file__)))
+EOT
+)
+    echo $MODULE_DIR
+}
+
+# Find our modules
+echo "==================================================================="
+echo "LOCAL MODULE LOCATIONS"
+PILLAR_DIR=$(find_module pillar)
+ATTRACT_DIR=$(find_module attract)
+FLAMENCO_DIR=$(find_module flamenco)
+SVNMAN_DIR=$(find_module svnman)
+SDK_DIR=$(find_module pillarsdk)
+
+echo "Pillar  : $PILLAR_DIR"
+echo "Attract : $ATTRACT_DIR"
+echo "Flamenco: $FLAMENCO_DIR"
+echo "SVNMan  : $SVNMAN_DIR"
+echo "SDK     : $SDK_DIR"
+
+if [ -z "$PILLAR_DIR" -o -z "$ATTRACT_DIR" -o -z "$FLAMENCO_DIR" -o -z "$SVNMAN_DIR" -o -z "$SDK_DIR" ];
+then
+    exit 1
+fi
+
+function git_clone() {
+    PROJECT_NAME="$1"
+    BRANCH="$2"
+    LOCAL_ROOT="$3"
+
+    echo "==================================================================="
+    echo "CLONING REPO ON $PROJECT_NAME @$BRANCH"
+    URL=$(git -C $LOCAL_ROOT remote get-url origin)
+    git -C $DEPLOYDIR clone --depth 1 --branch $BRANCH $URL $PROJECT_NAME
+}
+
+git_clone pillar-python-sdk master $SDK_DIR
+git_clone pillar $DEPLOY_BRANCH $PILLAR_DIR
+git_clone attract $DEPLOY_BRANCH $ATTRACT_DIR
+git_clone flamenco $DEPLOY_BRANCH $FLAMENCO_DIR
+git_clone pillar-svnman $DEPLOY_BRANCH $SVNMAN_DIR
+git_clone blender-cloud $DEPLOY_BRANCH $ROOT
+
+# Gulp everywhere
+GULP=$ROOT/node_modules/.bin/gulp
+if [ ! -e $GULP -o gulpfile.js -nt $GULP ]; then
+    npm install
+    touch $GULP  # installer doesn't always touch this after a build, so we do.
+fi
+$GULP --cwd $DEPLOYDIR/pillar --production
+$GULP --cwd $DEPLOYDIR/attract --production
+$GULP --cwd $DEPLOYDIR/flamenco --production
+$GULP --cwd $DEPLOYDIR/pillar-svnman --production
+$GULP --cwd $DEPLOYDIR/blender-cloud --production
+
+echo
+echo "==================================================================="
+echo "Deploy of ${PROJECT_NAME} is ready for dockerisation."
+echo "==================================================================="

deploy/2server.sh

@@ -0,0 +1,81 @@
+#!/bin/bash -e
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [[ `uname` == 'Darwin' ]]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+ROOT="$(dirname "$(dirname "$($readlink -f "$0")")")"
+PROJECT_NAME="$(basename $ROOT)"
+DOCKER_DEPLOYDIR="$ROOT/docker/4_run/deploy"
+DOCKER_IMAGE="armadillica/blender_cloud:latest"
+REMOTE_SECRET_CONFIG_DIR="/data/config"
+REMOTE_DOCKER_COMPOSE_DIR="/root/docker"
+
+#################################################################################
+case $1 in
+    cloud*)
+        DEPLOYHOST="$1"
+        ;;
+    *)
+        echo "Use $0 cloud{nr}|cloud.blender.org" >&2
+        exit 1
+esac
+SSH_OPTS="-o ClearAllForwardings=yes -o PermitLocalCommand=no"
+SSH="ssh $SSH_OPTS $DEPLOYHOST"
+SCP="scp $SSH_OPTS"
+
+echo -n "Deploying to $DEPLOYHOST… "
+
+if ! ping $DEPLOYHOST -q -c 1 -W 2 >/dev/null; then
+    echo "host $DEPLOYHOST cannot be pinged, refusing to deploy." >&2
+    exit 2
+fi
+
+cat <<EOT
+[ping OK]
+
+Make sure that you have pushed the $DOCKER_IMAGE
+docker image to Docker Hub.
+
+press [ENTER] to continue, Ctrl+C to abort.
+EOT
+read dummy
+
+#################################################################################
+echo "==================================================================="
+echo "Bringing remote Docker up to date…"
+$SSH mkdir -p $REMOTE_DOCKER_COMPOSE_DIR
+$SCP \
+    $ROOT/docker/{docker-compose.yml,renew-letsencrypt.sh,mongo-backup.{cron,sh}} \
+    $DEPLOYHOST:$REMOTE_DOCKER_COMPOSE_DIR
+$SSH -T <<EOT
+set -e
+cd $REMOTE_DOCKER_COMPOSE_DIR
+docker pull $DOCKER_IMAGE
+docker-compose up -d
+
+echo
+echo "==================================================================="
+echo "Clearing front page from Redis cache."
+docker exec redis redis-cli DEL pwview//
+EOT
+
+# Notify Sentry of this new deploy.
+# See https://sentry.io/blender-institute/blender-cloud/settings/release-tracking/
+# and https://docs.sentry.io/api/releases/post-organization-releases/
+# and https://sentry.io/api/
+echo
+echo "==================================================================="
+REVISION=$(date +'%Y%m%d.%H%M%S.%Z')
+echo "Notifying Sentry of this new deploy of revision $REVISION."
+SENTRY_RELEASE_URL="$($SSH env PYTHONPATH="$REMOTE_SECRET_CONFIG_DIR" python3 -c "\"import config_secrets; print(config_secrets.SENTRY_RELEASE_URL)\"")"
+curl -s "$SENTRY_RELEASE_URL" -XPOST -H 'Content-Type: application/json' -d "{\"version\": \"$REVISION\"}" | json_pp
+echo
+
+echo
+echo "==================================================================="
+echo "Deploy to $DEPLOYHOST done."
+echo "==================================================================="

deploy/build-all.sh

@@ -0,0 +1 @@
+build-quick.sh

deploy/build-quick.sh

@@ -0,0 +1,34 @@
+#!/bin/bash -e
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [[ `uname` == 'Darwin' ]]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+ROOT="$(dirname "$(dirname "$($readlink -f "$0")")")"
+DOCKERDIR="$ROOT/docker/4_run"
+
+case "$(basename "$0")" in
+    build-quick.sh)
+        pushd "$ROOT/docker/4_run"
+        ./build.sh
+        ;;
+    build-all.sh)
+        pushd "$ROOT/docker"
+        ./full_rebuild.sh
+        ;;
+    *)
+        echo "Unknown script $0, aborting" >&2
+        exit 1
+esac
+
+popd
+echo
+echo "Press [ENTER] to push the new Docker image."
+read dummy
+docker push armadillica/blender_cloud:latest
+echo
+echo "Build is done, ready to update the server."
+

docker/1_base/Dockerfile

@@ -0,0 +1,10 @@
+FROM ubuntu:18.04
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+RUN set -ex; \
+    apt-get update; \
+    DEBIAN_FRONTEND=noninteractive apt-get install \
+      -qyy -o APT::Install-Recommends=false -o APT::Install-Suggests=false \
+      tzdata openssl ca-certificates locales; \
+    locale-gen en_US.UTF-8 en_GB.UTF-8 nl_NL.UTF-8
+ENV LANG en_US.UTF-8

docker/1_base/base.docker

@@ -1,16 +0,0 @@
-FROM ubuntu:16.04
-MAINTAINER Francesco Siddi <francesco@blender.org>
-
-RUN apt-get update && apt-get install -qyy \
-    -o APT::Install-Recommends=false -o APT::Install-Suggests=false \
-    python-pip libffi6 openssl ffmpeg rsyslog logrotate
-
-RUN mkdir -p /data/git/pillar \
-&& mkdir -p /data/storage \
-&& mkdir -p /data/config \
-&& mkdir -p /data/venv \
-&& mkdir -p /data/wheelhouse
-
-RUN pip install virtualenv
-RUN virtualenv /data/venv
-RUN . /data/venv/bin/activate && pip install -U pip && pip install wheel

docker/1_base/build.sh

@@ -1,3 +1,4 @@
 #!/usr/bin/env bash
 
-docker build -t pillar_base -f base.docker .;
+# Uses --no-cache to always get the latest upstream (security) upgrades.
+exec docker build --no-cache "$@" -t pillar_base .

docker/2_build/build-wheels.sh

@@ -1,3 +0,0 @@
-#!/usr/bin/env bash
-
-. /data/venv/bin/activate && pip wheel --wheel-dir=/data/wheelhouse -r /requirements.txt

docker/2_build/build.docker

@@ -1,26 +0,0 @@
-FROM pillar_base
-MAINTAINER Francesco Siddi <francesco@blender.org>
-
-RUN apt-get update && apt-get install -qy \
-git \
-gcc \
-libffi-dev \
-libssl-dev \
-pypy-dev \
-python-dev \
-python-imaging \
-zlib1g-dev \
-libjpeg-dev \
-libtiff-dev \
-python-crypto \
-python-openssl
-
-ENV WHEELHOUSE=/data/wheelhouse
-ENV PIP_WHEEL_DIR=/data/wheelhouse
-ENV PIP_FIND_LINKS=/data/wheelhouse
-
-VOLUME /data/wheelhouse
-
-ADD requirements.txt /requirements.txt
-ADD build-wheels.sh /build-wheels.sh
-ENTRYPOINT ["bash", "build-wheels.sh"]

docker/2_build/build.sh

@@ -1,11 +0,0 @@
-#!/usr/bin/env bash
-
-mkdir -p ../3_run/wheelhouse;
-cp ../../requirements.txt .;
-
-docker build -t pillar_build -f build.docker .;
-docker run --rm \
-       -v "$(pwd)"/../3_run/wheelhouse:/data/wheelhouse \
-       pillar_build;
-
-rm requirements.txt;

docker/2_buildpy/Python-3.6.6.tar.xz.md5

@@ -0,0 +1 @@
+c3f30a0aff425dda77d19e02f420d6ba  Python-3.6.6.tar.xz

docker/2_buildpy/build.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+
+set -e
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [ $(uname) == 'Darwin' ]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+
+PYTHONTARGET=$($readlink -f ./python)
+
+mkdir -p "$PYTHONTARGET"
+echo "Python will be built to $PYTHONTARGET"
+
+docker build -t pillar_build -f buildpy.docker .
+
+# Use the docker image to build Python 3.6 and mod-wsgi
+GID=$(id -g)
+docker run --rm -i \
+    -v "$PYTHONTARGET:/opt/python" \
+    pillar_build <<EOT
+set -e
+cd \$PYTHONSOURCE
+./configure \
+    --prefix=/opt/python \
+    --enable-ipv6 \
+    --enable-shared \
+    --with-ensurepip=upgrade
+make -j8 install
+
+# Make sure we can run Python
+ldconfig
+
+# Upgrade pip
+/opt/python/bin/python3 -m pip install -U pip
+
+# Build mod-wsgi-py3 for Python 3.6
+cd /dpkg/mod-wsgi-*
+./configure --with-python=/opt/python/bin/python3
+make -j8 install
+mkdir -p /opt/python/mod-wsgi
+cp /usr/lib/apache2/modules/mod_wsgi.so /opt/python/mod-wsgi
+
+chown -R $UID:$GID /opt/python/*
+EOT
+
+# Strip some stuff we don't need from the Python install.
+rm -rf $PYTHONTARGET/lib/python3.*/test
+rm -rf $PYTHONTARGET/lib/python3.*/config-3.*/libpython3.*.a
+find $PYTHONTARGET/lib -name '*.so.*' -o -name '*.so' | while read libname; do
+    chmod u+w "$libname"
+    strip "$libname"
+done
+
+# Create another docker image which contains the actual Python.
+# This one will serve as base for the Wheel builder and the
+# production image.
+docker build -t armadillica/pillar_py:3.6 -f includepy.docker .

docker/2_buildpy/buildpy.docker

@@ -0,0 +1,35 @@
+FROM pillar_base
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+RUN sed -i 's/^# deb-src/deb-src/' /etc/apt/sources.list && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -qy \
+        build-essential \
+        apache2-dev \
+        checkinstall \
+        curl
+
+RUN apt-get build-dep -y python3.6
+
+ADD Python-3.6.6.tar.xz.md5 /Python-3.6.6.tar.xz.md5
+
+# Install Python sources
+RUN curl -O https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz && \
+    md5sum -c Python-3.6.6.tar.xz.md5 && \
+    tar xf Python-3.6.6.tar.xz && \
+    rm -v Python-3.6.6.tar.xz
+
+# Install mod-wsgi sources
+RUN mkdir -p /dpkg && cd /dpkg && apt-get source libapache2-mod-wsgi-py3
+
+# To be able to install Python outside the docker.
+VOLUME /opt/python
+
+# To be able to run Python; after building, ldconfig has to be re-run to do this.
+# This makes it easier to use Python right after building (for example to build
+# mod-wsgi for Python 3.6).
+RUN echo /opt/python/lib > /etc/ld.so.conf.d/python.conf
+RUN ldconfig
+ENV PATH=/opt/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
+ENV PYTHONSOURCE=/Python-3.6.6

docker/2_buildpy/includepy.docker

@@ -0,0 +1,13 @@
+FROM pillar_base
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+ADD python /opt/python
+
+RUN echo /opt/python/lib > /etc/ld.so.conf.d/python.conf
+RUN ldconfig
+
+RUN echo Python is installed in /opt/python/ > README.python
+ENV PATH=/opt/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
+RUN cd /opt/python/bin && \
+    ln -s python3 python

docker/3_buildwheels/Dockerfile

@@ -0,0 +1,20 @@
+FROM armadillica/pillar_py:3.6
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+RUN set -ex; \
+    apt-get update; \
+    DEBIAN_FRONTEND=noninteractive apt-get install -qy \
+    git \
+    build-essential \
+    checkinstall \
+    libffi-dev \
+    libssl-dev \
+    libjpeg-dev \
+    zlib1g-dev
+
+ENV WHEELHOUSE=/data/wheelhouse
+ENV PIP_WHEEL_DIR=/data/wheelhouse
+ENV PIP_FIND_LINKS=/data/wheelhouse
+RUN mkdir -p $WHEELHOUSE
+
+VOLUME /data/wheelhouse

docker/3_buildwheels/build.sh

@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+set -e
+
+# macOS does not support readlink -f, so we use greadlink instead
+if [ $(uname) == 'Darwin' ]; then
+    command -v greadlink 2>/dev/null 2>&1 || { echo >&2 "Install greadlink using brew."; exit 1; }
+    readlink='greadlink'
+else
+    readlink='readlink'
+fi
+
+TOPDEVDIR="$($readlink -f ../../..)"
+echo "Top-level development dir is $TOPDEVDIR"
+
+WHEELHOUSE="$($readlink -f ../4_run/wheelhouse)"
+if [ -z "$WHEELHOUSE" ]; then
+    echo "Error, ../4_run might not exist." >&2
+    exit 2
+fi
+
+echo "Wheelhouse is $WHEELHOUSE"
+mkdir -p "$WHEELHOUSE"
+
+docker build -t pillar_wheelbuilder .
+
+GID=$(id -g)
+docker run --rm -i \
+    -v "$WHEELHOUSE:/data/wheelhouse" \
+    -v "$TOPDEVDIR:/data/topdev" \
+    pillar_wheelbuilder <<EOT
+set -e
+# Build wheels for all dependencies.
+cd /data/topdev/blender-cloud
+pip3 install wheel
+pip3 wheel --wheel-dir=/data/wheelhouse -r requirements.txt
+chown -R $UID:$GID /data/wheelhouse
+
+# Install the dependencies so that we can get a full freeze.
+pip3 install --no-index --find-links=/data/wheelhouse -r requirements.txt
+pip3 freeze | grep -v '^-[ef] ' > /data/wheelhouse/requirements.txt
+EOT
+
+# Remove our own projects, they shouldn't be installed as wheel (for now).
+rm -f $WHEELHOUSE/{attract,flamenco,pillar,pillarsdk}*.whl

docker/3_run/000-default.conf

@@ -1,39 +0,0 @@
-<VirtualHost *:80>
-	# EnableSendfile on
-	XSendFile on
-	XSendFilePath /data/storage/pillar
-	XSendFilePath /data/git/pillar
-	XSendFilePath /data/venv/lib/python2.7/site-packages/attract/static/
-	XSendFilePath /data/venv/lib/python2.7/site-packages/flamenco/static/
-	XsendFilePath /data/git/blender-cloud
-
-	ServerAdmin webmaster@localhost
-	DocumentRoot /var/www/html
-
-	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
-	# error, crit, alert, emerg.
-	# It is also possible to configure the loglevel for particular
-	# modules, e.g.
-	# LogLevel info ssl:warn
-
-	ErrorLog ${APACHE_LOG_DIR}/error.log
-	CustomLog ${APACHE_LOG_DIR}/access.log combined
-
-	WSGIDaemonProcess cloud processes=4 threads=1 maximum-requests=10000
-	WSGIPassAuthorization On
-
-	WSGIScriptAlias / /data/git/blender-cloud/runserver.wsgi \
-		process-group=cloud application-group=%{GLOBAL}
-
-	<Directory /data/git/blender-cloud>
-		<Files runserver.wsgi>
-			Require all granted
-		</Files>
-	</Directory>
-
-	# Temporary edit to remap the old cloudapi.blender.org to cloud.blender.org/api
-	RewriteEngine On
-	RewriteCond "%{HTTP_HOST}"   "^cloudapi\.blender\.org" [NC]
-	RewriteRule (.*) /api$1 [PT]
-
-</VirtualHost>

docker/3_run/build.sh

@@ -1,5 +0,0 @@
-#!/usr/bin/env bash
-
-cp ../../requirements.txt .;
-docker build -t armadillica/blender_cloud -f run.docker .;
-rm requirements.txt;

docker/3_run/docker-entrypoint.sh

@@ -1,25 +0,0 @@
-#!/usr/bin/env bash
-
-if [ ! -f /installed ]; then
-    echo "Installing pillar and pillar-sdk"
-    # TODO: curretly doing pip install -e takes a long time, so we symlink
-    # . /data/venv/bin/activate && pip install -e /data/git/pillar
-    ln -s /data/git/pillar/pillar /data/venv/lib/python2.7/site-packages/pillar
-    # . /data/venv/bin/activate && pip install -e /data/git/attract
-    ln -s /data/git/attract/attract /data/venv/lib/python2.7/site-packages/attract
-    # . /data/venv/bin/activate && pip install -e /data/git/flamenco/packages/flamenco
-    ln -s /data/git/flamenco/packages/flamenco/flamenco/ /data/venv/lib/python2.7/site-packages/flamenco
-    # . /data/venv/bin/activate && pip install -e /data/git/pillar-python-sdk
-    ln -s /data/git/pillar-python-sdk/pillarsdk /data/venv/lib/python2.7/site-packages/pillarsdk
-    touch installed
-fi
-
-if [ "$DEV" = "true" ]; then
-    echo "Running in development mode"
-    cd /data/git/blender-cloud
-    bash /manage.sh runserver --host='0.0.0.0'
-else
-    # Run Apache
-    a2enmod rewrite
-    /usr/sbin/apache2ctl -D FOREGROUND
-fi

docker/3_run/manage.sh

@@ -1,5 +0,0 @@
-#!/usr/bin/env bash -e
-
-. /data/venv/bin/activate
-cd /data/git/blender-cloud
-python manage.py "$@"

docker/3_run/run.docker

@@ -1,46 +0,0 @@
-FROM pillar_base
-
-RUN apt-get update && apt-get install -qyy \
--o APT::Install-Recommends=true -o APT::Install-Suggests=false \
-git \
-apache2 \
-libapache2-mod-wsgi \
-libapache2-mod-xsendfile \
-libjpeg8 \
-libtiff5 \
-nano vim curl \
-&& rm -rf /var/lib/apt/lists/*
-
-ENV APACHE_RUN_USER www-data
-ENV APACHE_RUN_GROUP www-data
-ENV APACHE_LOG_DIR /var/log/apache2
-ENV APACHE_PID_FILE /var/run/apache2.pid
-ENV APACHE_RUN_DIR /var/run/apache2
-ENV APACHE_LOCK_DIR /var/lock/apache2
-
-RUN mkdir -p $APACHE_RUN_DIR $APACHE_LOCK_DIR $APACHE_LOG_DIR
-
-ADD requirements.txt /requirements.txt
-ADD wheelhouse /data/wheelhouse
-
-RUN . /data/venv/bin/activate \
-&& pip install --no-index --find-links=/data/wheelhouse -r requirements.txt \
-&& rm /requirements.txt
-
-VOLUME /data/git/blender-cloud
-VOLUME /data/git/pillar
-VOLUME /data/git/pillar-python-sdk
-VOLUME /data/config
-VOLUME /data/storage
-
-ENV USE_X_SENDFILE True
-
-EXPOSE 80
-EXPOSE 5000
-
-ADD apache2.conf /etc/apache2/apache2.conf
-ADD 000-default.conf /etc/apache2/sites-available/000-default.conf
-ADD docker-entrypoint.sh /docker-entrypoint.sh
-ADD manage.sh /manage.sh
-
-ENTRYPOINT ["bash", "/docker-entrypoint.sh"]

docker/4_run/Dockerfile

@@ -0,0 +1,64 @@
+FROM armadillica/pillar_py:3.6
+LABEL maintainer="Sybren A. Stüvel <sybren@blender.studio>"
+
+RUN set -ex; \
+    apt-get update; \
+    DEBIAN_FRONTEND=noninteractive apt-get install -qy \
+    -o APT::Install-Recommends=false -o APT::Install-Suggests=false \
+        git \
+        apache2 \
+        libapache2-mod-xsendfile \
+        libjpeg8 \
+        libtiff5 \
+        ffmpeg \
+        rsyslog logrotate \
+        nano vim-tiny curl; \
+    rm -rf /var/lib/apt/lists/*
+
+RUN ln -s /usr/bin/vim.tiny /usr/bin/vim
+
+ENV APACHE_RUN_USER www-data
+ENV APACHE_RUN_GROUP www-data
+ENV APACHE_LOG_DIR /var/log/apache2
+ENV APACHE_PID_FILE /var/run/apache2.pid
+ENV APACHE_RUN_DIR /var/run/apache2
+ENV APACHE_LOCK_DIR /var/lock/apache2
+
+RUN mkdir -p $APACHE_RUN_DIR $APACHE_LOCK_DIR $APACHE_LOG_DIR
+
+ADD wheelhouse /data/wheelhouse
+RUN pip3 install --no-index --find-links=/data/wheelhouse -r /data/wheelhouse/requirements.txt
+
+VOLUME /data/config
+VOLUME /data/storage
+VOLUME /var/log
+
+ENV USE_X_SENDFILE True
+
+EXPOSE 80
+EXPOSE 5000
+
+ADD apache/wsgi-py36.* /etc/apache2/mods-available/
+RUN a2enmod rewrite && a2enmod wsgi-py36
+
+ADD apache/apache2.conf /etc/apache2/apache2.conf
+ADD apache/000-default.conf /etc/apache2/sites-available/000-default.conf
+ADD apache/logrotate.conf /etc/logrotate.d/apache2
+ADD *.sh /
+
+# Remove some empty top-level directories we won't use anyway.
+RUN rmdir /media /home 2>/dev/null || true
+
+# This file includes some useful commands to have in the shell history
+# for easy access.
+ADD bash_history /root/.bash_history
+
+ENTRYPOINT /docker-entrypoint.sh
+
+# Add the most-changing files as last step for faster rebuilds.
+ADD config_local.py /data/git/blender-cloud/
+ADD deploy /data/git
+RUN python3 -c "import re, secrets; \
+f = open('/data/git/blender-cloud/config_local.py', 'a'); \
+h = re.sub(r'[_.~-]', '', secrets.token_urlsafe())[:8]; \
+print(f'STATIC_FILE_HASH = {h!r}', file=f)"

docker/4_run/apache/000-default.conf

@@ -0,0 +1,54 @@
+<VirtualHost *:80>
+	XSendFile on
+	XSendFilePath /data/storage/pillar
+	XSendFilePath /data/git/pillar/pillar/web/static/
+	XSendFilePath /data/git/attract/attract/static/
+	XSendFilePath /data/git/flamenco/flamenco/static/
+	XsendFilePath /data/git/pillar-svnman/svnman/static/
+	XsendFilePath /data/git/blender-cloud/static/
+	XsendFilePath /data/git/blender-cloud/cloud/static/
+
+	ServerAdmin webmaster@localhost
+	DocumentRoot /var/www/html
+
+	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
+	# error, crit, alert, emerg.
+	# It is also possible to configure the loglevel for particular
+	# modules, e.g.
+	# LogLevel info ssl:warn
+
+	ErrorLog ${APACHE_LOG_DIR}/error.log
+	CustomLog ${APACHE_LOG_DIR}/access.log combined
+
+	WSGIDaemonProcess cloud processes=2 threads=64 maximum-requests=10000
+	WSGIPassAuthorization On
+
+	WSGIScriptAlias / /data/git/blender-cloud/runserver.wsgi \
+		process-group=cloud application-group=%{GLOBAL}
+
+	<Directory /data/git/blender-cloud>
+		<Files runserver.wsgi>
+			Require all granted
+		</Files>
+	</Directory>
+
+	# Temporary edit to remap the old cloudapi.blender.org to cloud.blender.org/api
+	RewriteEngine On
+	RewriteCond "%{HTTP_HOST}"   "^cloudapi\.blender\.org" [NC]
+	RewriteRule (.*) /api$1 [PT]
+
+	# Redirects for blender-cloud projects
+	RewriteRule "^/p/blender-cloud/?$"  "/blog" [R=301,L]
+	RewriteRule "^/agent327/?$"  "/p/agent-327" [R=301,L]
+	RewriteRule "^/caminandes/?$"  "/p/caminandes-3" [R=301,L]
+	RewriteRule "^/cf2/?$"  "/p/creature-factory-2" [R=301,L]
+	RewriteRule "^/characters/?$"  "/p/characters" [R=301,L]
+	RewriteRule "^/gallery/?$"  "/p/gallery" [R=301,L]
+	RewriteRule "^/hdri/?$"  "/p/hdri" [R=301,L]
+	RewriteRule "^/textures/?$"  "/p/textures" [R=301,L]
+	RewriteRule "^/training/?$"  "/courses" [R=301,L]
+	RewriteRule "^/spring/?$"  "/p/spring" [R=301,L]
+	RewriteRule "^/hero/?$"  "/p/hero" [R=301,L]
+	# Waking the forest was moved from the art gallery to its own workshop
+	RewriteRule "^/p/gallery/58cfec4f88ac8f1440aeb309/?$"  "/p/waking-the-forest" [R=301,L]
+</VirtualHost>

docker/4_run/apache/logrotate.conf

@@ -0,0 +1,21 @@
+/var/log/apache2/*.log {
+    daily
+    missingok
+    rotate 14
+    size 100M
+    compress
+    delaycompress
+    notifempty
+    create 640 root adm
+    sharedscripts
+    postrotate
+        if /etc/init.d/apache2 status > /dev/null ; then \
+            /etc/init.d/apache2 reload > /dev/null; \
+        fi;
+    endscript
+    prerotate
+        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
+            run-parts /etc/logrotate.d/httpd-prerotate; \
+        fi; \
+    endscript
+}

docker/4_run/apache/wsgi-py36.conf

@@ -0,0 +1,122 @@
+<IfModule mod_wsgi.c>
+
+
+    #This config file is provided to give an overview of the directives,
+    #which are only allowed in the 'server config' context.
+    #For a detailed description of all avaiable directives please read
+    #http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives
+
+
+    #WSGISocketPrefix: Configure directory to use for daemon sockets.
+    #
+    #Apache's DEFAULT_REL_RUNTIMEDIR should be the proper place for WSGI's
+    #Socket. In case you want to mess with the permissions of the directory,
+    #you need to define WSGISocketPrefix to an alternative directory.
+    #See http://code.google.com/p/modwsgi/wiki/ConfigurationIssues for more
+    #information
+
+        #WSGISocketPrefix /var/run/apache2/wsgi
+
+
+    #WSGIPythonOptimize: Enables basic Python optimisation features.
+    #
+    #Sets the level of Python compiler optimisations. The default is '0'
+    #which means no optimisations are applied.
+    #Setting the optimisation level to '1' or above will have the effect
+    #of enabling basic Python optimisations and changes the filename
+    #extension for compiled (bytecode) files from .pyc to .pyo.
+    #When the optimisation level is set to '2', doc strings will not be
+    #generated and retained. This will result in a smaller memory footprint,
+    #but may cause some Python packages which interrogate doc strings in some
+    #way to fail.
+
+        #WSGIPythonOptimize 0
+
+
+    #WSGIPythonPath: Additional directories to search for Python modules,
+    #                overriding the PYTHONPATH environment variable.
+    #
+    #Used to specify additional directories to search for Python modules.
+    #If multiple directories are specified they should be separated by a ':'.
+
+        WSGIPythonPath /opt/python/lib/python3.6/site-packages
+
+    #WSGIPythonEggs: Directory to use for Python eggs cache.
+    #
+    #Used to specify the directory to be used as the Python eggs cache
+    #directory for all sub interpreters created within embedded mode.
+    #This directive achieves the same affect as having set the
+    #PYTHON_EGG_CACHE environment variable.
+    #Note that the directory specified must exist and be writable by the user
+    #that the Apache child processes run as. The directive only applies to
+    #mod_wsgi embedded mode. To set the Python eggs cache directory for
+    #mod_wsgi daemon processes, use the 'python-eggs' option to the
+    #WSGIDaemonProcess directive instead.
+
+        #WSGIPythonEggs directory
+
+
+
+    #WSGIRestrictEmbedded: Enable restrictions on use of embedded mode.
+    #
+    #The WSGIRestrictEmbedded directive determines whether mod_wsgi embedded
+    #mode is enabled or not. If set to 'On' and the restriction on embedded
+    #mode is therefore enabled, any attempt to make a request against a
+    #WSGI application which hasn't been properly configured so as to be
+    #delegated to a daemon mode process will fail with a HTTP internal server
+    #error response.
+
+        #WSGIRestrictEmbedded On|Off
+
+
+
+    #WSGIRestrictStdin: Enable restrictions on use of STDIN.
+    #WSGIRestrictStdout: Enable restrictions on use of STDOUT.
+    #WSGIRestrictSignal: Enable restrictions on use of signal().
+    #
+    #Well behaved WSGI applications neither should try to read/write from/to
+    #STDIN/STDOUT, nor should they try to register signal handlers. If your
+    #application needs an exception from this rule, you can disable the
+    #restrictions here.
+
+        #WSGIRestrictStdin On
+        #WSGIRestrictStdout On
+        #WSGIRestrictSignal On
+
+
+
+    #WSGIAcceptMutex: Specify type of accept mutex used by daemon processes.
+    #
+    #The WSGIAcceptMutex directive sets the method that mod_wsgi will use to
+    #serialize multiple daemon processes in a process group accepting requests
+    #on a socket connection from the Apache child processes. If this directive
+    #is not defined then the same type of mutex mechanism as used by Apache for
+    #the main Apache child processes when accepting connections from a client
+    #will be used. If set the method types are the same as for the Apache
+    #AcceptMutex directive.
+
+        #WSGIAcceptMutex default
+
+
+
+    #WSGIImportScript: Specify a script file to be loaded on process start.
+    #
+    #The WSGIImportScript directive can be used to specify a script file to be
+    #loaded when a process starts. Options must be provided to indicate the
+    #name of the process group and the application group into which the script
+    #will be loaded.
+
+        #WSGIImportScript process-group=name application-group=name
+
+
+    #WSGILazyInitialization: Enable/disable lazy initialisation of Python.
+    #
+    #The WSGILazyInitialization directives sets whether or not the Python
+    #interpreter is preinitialised within the Apache parent process or whether
+    #lazy initialisation is performed, and the Python interpreter only
+    #initialised in the Apache server processes or mod_wsgi daemon processes
+    #after they have forked from the Apache parent process.
+
+        #WSGILazyInitialization On|Off
+
+</IfModule>

docker/4_run/apache/wsgi-py36.load

@@ -0,0 +1 @@
+LoadModule wsgi_module /opt/python/mod-wsgi/mod_wsgi.so

docker/4_run/bash_history

@@ -0,0 +1,9 @@
+bash docker-entrypoint.sh
+env | sort
+apache2ctl start
+apache2ctl graceful
+/manage.sh operations worker -- -C
+celery status --broker amqp://guest:guest@rabbit:5672//
+celery events --broker amqp://guest:guest@rabbit:5672//
+tail -n 40 -f /var/log/apache2/access.log
+tail -n 40 -f /var/log/apache2/error.log

docker/4_run/build.sh

@@ -0,0 +1,5 @@
+#!/bin/bash -e
+
+docker build -t armadillica/blender_cloud:latest .
+
+echo "Done, built armadillica/blender_cloud:latest"

docker/4_run/celery-beat.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+source /install_scripts.sh
+source /manage.sh celery beat -- \
+    --schedule /data/storage/pillar/celerybeat-schedule.db \
+    --pid /data/storage/pillar/celerybeat.pid

docker/4_run/celery-worker.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+
+source /install_scripts.sh
+source /manage.sh celery worker -- -C

docker/4_run/config_local.py

@@ -0,0 +1,132 @@
+import os
+from collections import defaultdict
+
+DEBUG = False
+
+SCHEME = 'https'
+PREFERRED_URL_SCHEME = 'https'
+SERVER_NAME = 'cloud.blender.org'
+
+# os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = 'true'
+os.environ['PILLAR_MONGO_DBNAME'] = 'cloud'
+os.environ['PILLAR_MONGO_PORT'] = '27017'
+os.environ['PILLAR_MONGO_HOST'] = 'mongo'
+
+USE_X_SENDFILE = True
+
+STORAGE_BACKEND = 'gcs'
+
+CDN_SERVICE_DOMAIN = 'blendercloud-pro.r.worldssl.net'
+CDN_CONTENT_SUBFOLDER = ''
+CDN_STORAGE_ADDRESS = 'push-11.cdnsun.com'
+
+CACHE_TYPE = 'redis'  # null
+CACHE_KEY_PREFIX = 'pw_'
+CACHE_REDIS_HOST = 'redis'
+CACHE_REDIS_PORT = '6379'
+CACHE_REDIS_URL = 'redis://redis:6379'
+
+PILLAR_SERVER_ENDPOINT = 'https://cloud.blender.org/api/'
+
+BLENDER_ID_ENDPOINT = 'https://www.blender.org/id/'
+
+GCLOUD_APP_CREDENTIALS = '/data/config/google_app.json'
+GCLOUD_PROJECT = 'blender-cloud'
+
+MAIN_PROJECT_ID = '563a9c8cf0e722006ce97b03'
+# MAIN_PROJECT_ID = '57aa07c088bef606e89078bd'
+
+ALGOLIA_INDEX_USERS = 'pro_Users'
+ALGOLIA_INDEX_NODES = 'pro_Nodes'
+
+ZENCODER_NOTIFICATIONS_URL = 'https://cloud.blender.org/api/encoding/zencoder/notifications'
+
+FILE_LINK_VALIDITY = defaultdict(
+    lambda: 3600 * 24 * 30,  # default of 1 month.
+    gcs=3600 * 23,  # 23 hours for Google Cloud Storage.
+    cdnsun=3600 * 23
+)
+
+LOGGING = {
+    'version': 1,
+    'formatters': {
+        'default': {'format': '%(levelname)8s %(name)s %(message)s'}
+    },
+    'handlers': {
+        'console': {
+            'class': 'logging.StreamHandler',
+            'formatter': 'default',
+            'stream': 'ext://sys.stderr',
+        }
+    },
+    'loggers': {
+        'pillar': {'level': 'INFO'},
+        # 'pillar.auth': {'level': 'DEBUG'},
+        # 'pillar.api.blender_id': {'level': 'DEBUG'},
+        # 'pillar.api.blender_cloud.subscription': {'level': 'DEBUG'},
+        'bcloud': {'level': 'INFO'},
+        'cloud': {'level': 'INFO'},
+        'attract': {'level': 'INFO'},
+        'flamenco': {'level': 'INFO'},
+        # 'pillar.api.file_storage': {'level': 'DEBUG'},
+        # 'pillar.api.file_storage.ensure_valid_link': {'level': 'INFO'},
+        'pillar.api.file_storage.refresh_links_for_backend': {'level': 'DEBUG'},
+        'werkzeug': {'level': 'DEBUG'},
+        'eve': {'level': 'WARNING'},
+        # 'elasticsearch': {'level': 'DEBUG'},
+    },
+    'root': {
+        'level': 'WARNING',
+        'handlers': [
+            'console',
+        ],
+    }
+}
+
+# Latest version of the add-on.
+BLENDER_CLOUD_ADDON_VERSION = '1.9.0'
+
+REDIRECTS = {
+    # For old links, refer to the services page (hopefully it refreshes then)
+    'downloads/blender_cloud-latest-bundle.zip': 'https://cloud.blender.org/services#blender-addon',
+
+    # Latest Blender Cloud add-on.
+    'downloads/blender_cloud-latest-addon.zip':
+        f'https://storage.googleapis.com/institute-storage/addons/'
+        f'blender_cloud-{BLENDER_CLOUD_ADDON_VERSION}.addon.zip',
+
+    # Redirect old Grafista endpoint to /stats
+    '/stats/': '/stats',
+}
+
+UTM_LINKS = {
+    'cartoon_brew': {
+        'image': 'https://imgur.com/13nQTi3.png',
+        'link': 'https://store.blender.org/product/membership/'
+    }
+}
+
+# Disabled until we have regenerated the majority of the links.
+CELERY_BEAT_SCHEDULE = {
+    'regenerate-expired-links': {
+        'task': 'pillar.celery.file_link_tasks.regenerate_all_expired_links',
+        'schedule': 600,  # every N seconds
+        'args': ('gcs', 500)
+    },
+}
+
+SVNMAN_REPO_URL = 'https://svn.blender.cloud/repo/'
+SVNMAN_API_URL = 'https://svn.blender.cloud/api/'
+
+# Mail options, see pillar.celery.email_tasks.
+SMTP_HOST = 'proog.blender.org'
+SMTP_PORT = 25
+SMTP_USERNAME = 'server@blender.cloud'
+SMTP_TIMEOUT = 30  # timeout in seconds, https://docs.python.org/3/library/smtplib.html#smtplib.SMTP
+MAIL_RETRY = 180  # in seconds, delay until trying to send an email again.
+MAIL_DEFAULT_FROM_NAME = 'Blender Cloud'
+MAIL_DEFAULT_FROM_ADDR = 'cloudsupport@blender.org'
+
+# MUST be 8 characters long, see pillar.flask_extra.HashedPathConverter
+# STATIC_FILE_HASH = '12345678'
+# The value used in production is appended from Dockerfile.

docker/4_run/docker-entrypoint.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+source /install_scripts.sh
+
+# Make sure that log rotation works.
+mkdir -p ${APACHE_LOG_DIR}
+service cron start
+
+if [ "$DEV" = "true" ]; then
+    echo "Running in development mode"
+    cd /data/git/blender-cloud
+    exec bash /manage.sh runserver --host='0.0.0.0'
+else
+    exec /usr/sbin/apache2ctl -D FOREGROUND
+fi

docker/4_run/install_scripts.sh

@@ -0,0 +1,21 @@
+#!/bin/sh
+
+if [ -f /installed ]; then
+    return
+fi
+
+SITEPKG=$(echo /opt/python/lib/python3.*/site-packages)
+echo "Installing Blender Cloud packages into $SITEPKG"
+
+# TODO: 'pip3 install -e' runs 'setup.py develop', which runs 'setup.py egg_info',
+# which can't write the egg info to the read-only /data/git volume. This is why
+# we manually install the links.
+for SUBPROJ in /data/git/{pillar,pillar-python-sdk,attract,flamenco,pillar-svnman}; do
+    NAME=$(python3 $SUBPROJ/setup.py --name)
+    echo "... $NAME"
+    echo $SUBPROJ >> $SITEPKG/easy-install.pth
+    echo $SUBPROJ > $SITEPKG/$NAME.egg-link
+done
+echo "All packages installed."
+
+touch /installed

docker/4_run/manage.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+
+set -e
+cd /data/git/blender-cloud
+exec python manage.py "$@"

docker/README.md

@@ -0,0 +1,96 @@
+# Setting up a production machine
+
+To get the docker stack up and running, we use the following, on an Ubuntu 16.10 machine.
+
+## 0. Basic stuff
+
+Install the machine, use `locale-gen nl_NL.UTF-8` or similar commands to generate locale
+definitions. Set up automatic security updates and backups, the usual.
+
+## 1. Install Docker
+
+Install Docker itself, as described in the
+   [Docker CE for Ubuntu manual](https://store.docker.com/editions/community/docker-ce-server-ubuntu?tab=description):
+
+    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+    add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+        $(lsb_release -cs) stable"
+    apt-get update
+    apt-get install docker-ce
+
+## 2. Configure Docker to use "overlay"
+
+Configure Docker to use "overlay" instead of "aufs" for the images. This prevents
+[segfaults in auplink](https://bugs.launchpad.net/ubuntu/+source/aufs-tools/+bug/1442568).
+
+1. Set `DOCKER_OPTS="-s overlay"` in `/etc/defaults/docker`
+2. Copy `/lib/systemd/system/docker.service` to `/etc/systemd/system/docker.service`.
+   This allows later upgrading of docker without overwriting the changes we're about to do.
+2. Edit the `[Service]` section of `/etc/systemd/system/docker.service`:
+    1. Add `EnvironmentFile=/etc/default/docker`
+    2. Append ` $DOCKER_OPTS` to the `ExecStart` line
+3. Run `systemctl daemon-reload`
+4. Remove all your containers and images.
+5. Restart Docker: `systemctl restart docker`
+
+## 3. Pull the Blender Cloud docker image
+
+`docker pull armadillica/blender_cloud:latest`
+
+## 4. Get docker-compose + our repositories
+
+See the [Quick setup](../README.md) on how to get those. Then run:
+
+    cd /data/git/blender-cloud/docker
+    docker-compose up -d
+
+
+Set up permissions for Docker volumes; the following should be writable by
+
+- `/data/storage/pillar`: writable by `www-data` and `root` (do a `chown root:www-data`
+  and `chmod 2770`).
+- `/data/storage/db`: writable by uid 999.
+
+
+## 5. Set up TLS
+
+Place TLS certificates in `/data/certs/{cloud,cloudapi}.blender.org.pem`.
+They should contain (in order) the private key, the host certificate, and the
+CA certificate.
+
+
+## 6. Create a local config
+
+Blender Cloud expects the following files to exist:
+
+- `/data/git/blender_cloud/config_local.py` with machine-local configuration overrides
+- `/data/config/google_app.json` with Google Cloud Storage credentials.
+
+When run from Docker, the `docker/4_run/config_local.py` file will be used. Overrides for that file
+can be placed in `/data/config/config_secrets.py`.
+
+
+## 7. ElasticSearch & kibana
+
+ElasticSearch and Kibana run in our self-rolled images. This is needed because by default
+
+- ElasticSearch uses up to 2 GB of RAM, which is too much for our droplet, and
+- the Docker images contain the proprietary X-Pack plugin, which we don't want.
+
+This also gives us the opportunity to let Kibana do its optimization when we build the image, rather
+than every time the container is recreated.
+
+`/data/storage/elasticsearch` needs to be writable by UID 1000, GID 1000.
+
+Kibana connects to [ElasticProxy](https://github.com/armadillica/elasticproxy), which only allows
+GET, HEAD, and some specific POST requests. This ensures that the public-facing Kibana cannot be
+used to change the ElasticSearch database.
+
+Production Kibana can be placed in read-only mode, but this is not necessary now that we use
+ElasticProxy. However, I've left this in here as reference.
+
+`curl -XPUT 'localhost:9200/.kibana/_settings' -d '{ "index.blocks.read_only" : true }'`
+
+If editing is desired, temporarily turn off read-only mode:
+
+`curl -XPUT 'localhost:9200/.kibana/_settings' -d '{ "index.blocks.read_only" : false }'`

docker/build.sh

@@ -1,16 +0,0 @@
-#!/usr/bin/env bash
-
-set -x;
-set -e;
-
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-cd $DIR;
-
-cd 1_base/;
-bash build.sh;
-
-cd ../2_build/;
-bash build.sh;
-
-cd ../3_run/;
-bash build.sh;

docker/docker-compose.yml

@@ -1,68 +1,191 @@
-mongo:
-  image: mongo
-  container_name: mongo
-  restart: always
-  volumes:
-   - /data/storage/db:/data/db
-  ports:
-   - "127.0.0.1:27017:27017"
-redis:
-  image: redis
-  container_name: redis
-  restart: always
-blender_cloud:
-  image: armadillica/blender_cloud
-  container_name: blender_cloud
-  restart: always
-  environment:
-   VIRTUAL_HOST: http://cloudapi.blender.org/*,https://cloudapi.blender.org/*,http://cloud.blender.org/*,https://cloud.blender.org/*,http://pillar-web/*
-   VIRTUAL_HOST_WEIGHT: 10
-   FORCE_SSL: "true"
-  volumes:
-   - /data/git/blender-cloud:/data/git/blender-cloud:ro
-   - /data/git/attract:/data/git/attract:ro
-   - /data/git/flamenco:/data/git/flamenco:ro
-   - /data/git/pillar:/data/git/pillar:ro
-   - /data/git/pillar-python-sdk:/data/git/pillar-python-sdk:ro
-   - /data/config:/data/config:ro
-   - /data/storage/pillar:/data/storage/pillar
-  links:
-   - mongo
-   - redis
-# notifserv:
-#   container_name: notifserv
-#   image: armadillica/pillar-notifserv:cd8fa678436563ac3b800b2721e36830c32e4656
-#   restart: always
-#   links:
-#    - mongo
-#   environment:
-#    VIRTUAL_HOST: https://cloud.blender.org/notifications*,http://pillar-web/notifications*
-#    VIRTUAL_HOST_WEIGHT: 20
-#    FORCE_SSL: true
-grafista:
-  image: armadillica/grafista
-  container_name: grafista
-  restart: always
-  environment:
-    VIRTUAL_HOST: http://cloud.blender.org/stats/*,https://cloud.blender.org/stats/*,http://blender-cloud/stats/*
-    VIRTUAL_HOST_WEIGHT: 20
-    FORCE_SSL: "true"
-  volumes:
-    - /data/git/grafista:/data/git/grafista:ro
-    - /data/storage/grafista:/data/storage
-haproxy:
-  image: dockercloud/haproxy
-  container_name: haproxy
-  restart: always
-  ports:
-   - "443:443"
-   - "80:80"
-  environment:
-   - CERT_FOLDER=/certs/
-   - TIMEOUT=connect 5s, client 5m, server 10m
-  links:
-   - blender_cloud
-   - grafista
-   # - notifserv
-  volumes:
-   - '/data/certs:/certs'
+version:                     '3.4'
+services:
+  mongo:
+    image:                   mongo:3.4.2
+    container_name:          mongo
+    restart:                 always
+    volumes:
+      - /data/storage/db:/data/db
+      - /data/storage/db-bak:/data/db-bak # for backing up stuff etc.
+    ports:
+      - "127.0.0.1:27017:27017"
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  redis:
+    image:                   redis:3.2.8
+    container_name:          redis
+    restart:                 always
+    ports:
+      - "127.0.0.1:6379:6379"
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  rabbit:
+    image:                   rabbitmq:3.6.10
+    container_name:          rabbit
+    restart:                 always
+    ports:
+      - "127.0.0.1:5672:5672"
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  elastic:
+    image:                   armadillica/elasticsearch:6.1.1
+    container_name:          elastic
+    restart:                 always
+    volumes:
+      #  NOTE:               this path must be writable by UID=1000 GID=1000.
+      - /data/storage/elastic:/usr/share/elasticsearch/data
+    ports:
+      - "127.0.0.1:9200:9200"
+    environment:
+      ES_JAVA_OPTS:          "-Xms256m -Xmx256m"
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  elasticproxy:
+    image:                   armadillica/elasticproxy:1.2
+    container_name:          elasticproxy
+    restart:                 always
+    command:                 /elasticproxy -elastic http://elastic:9200/
+    depends_on:
+      - elastic
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  kibana:
+    image:                   armadillica/kibana:6.1.1
+    container_name:          kibana
+    restart:                 always
+    environment:
+      SERVER_NAME:           "stats.cloud.blender.org"
+      ELASTICSEARCH_URL:     http://elasticproxy:9200
+      CONSOLE_ENABLED:       'false'
+      VIRTUAL_HOST:          http://stats.cloud.blender.org/*,https://stats.cloud.blender.org/*,http://stats.cloud.local/*,https://stats.cloud.local/*
+      VIRTUAL_HOST_WEIGHT:   20
+      FORCE_SSL:             "true"
+
+      # See https://github.com/elastic/kibana/issues/5170#issuecomment-163042525
+      NODE_OPTIONS:          "--max-old-space-size=200"
+    depends_on:
+      - elasticproxy
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  blender_cloud:
+    image:                   armadillica/blender_cloud:latest
+    container_name:          blender_cloud
+    restart:                 always
+    environment:
+      VIRTUAL_HOST:          http://cloud.blender.org/*,https://cloud.blender.org/*,http://cloud.local/*,https://cloud.local/*
+      VIRTUAL_HOST_WEIGHT:   10
+      FORCE_SSL:             "true"
+      GZIP_COMPRESSION_TYPE: "text/html text/plain text/css application/javascript"
+      PILLAR_CONFIG:         /data/config/config_secrets.py
+    volumes:
+      # format:              HOST:CONTAINER
+      - /data/config:/data/config:ro
+      - /data/storage/pillar:/data/storage/pillar
+      - /data/log:/var/log
+    depends_on:
+      - mongo
+      - redis
+      - rabbit
+
+  celery_worker:
+    image:                   armadillica/blender_cloud:latest
+    entrypoint:              /celery-worker.sh
+    container_name:          celery_worker
+    restart:                 always
+    environment:
+      PILLAR_CONFIG:         /data/config/config_secrets.py
+    volumes:
+      # format:              HOST:CONTAINER
+      - /data/config:/data/config:ro
+      - /data/storage/pillar:/data/storage/pillar
+      - /data/log:/var/log
+    depends_on:
+      - mongo
+      - redis
+      - rabbit
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  celery_beat:
+    image:                   armadillica/blender_cloud:latest
+    entrypoint:              /celery-beat.sh
+    container_name:          celery_beat
+    restart:                 always
+    environment:
+      PILLAR_CONFIG:         /data/config/config_secrets.py
+    volumes:
+      # format:              HOST:CONTAINER
+      - /data/config:/data/config:ro
+      - /data/storage/pillar:/data/storage/pillar
+      - /data/log:/var/log
+    depends_on:
+      - mongo
+      - redis
+      - rabbit
+      - celery_worker
+    logging:
+      driver:                "json-file"
+      options:
+        max-size:            "200k"
+        max-file:            "20"
+
+  letsencrypt:
+    image:                   armadillica/picohttp:1.0
+    container_name:          letsencrypt
+    restart:                 always
+    environment:
+      WEBROOT:               /data/letsencrypt
+      LISTEN:                '[::]:80'
+      VIRTUAL_HOST:          http://cloud.blender.org/.well-known/*, http://stats.cloud.blender.org/.well-known/*
+      VIRTUAL_HOST_WEIGHT:   30
+    volumes:
+      - /data/letsencrypt:/data/letsencrypt
+
+  haproxy:
+    image:                   dockercloud/haproxy:1.5.3
+    container_name:          haproxy
+    restart:                 always
+    ports:
+      - "443:443"
+      - "80:80"
+    environment:
+      - ADDITIONAL_SERVICES=docker:blender_cloud,docker:letsencrypt,docker:kibana
+      - CERT_FOLDER=/certs/
+      - TIMEOUT=connect 5s, client 5m, server 10m
+      - SSL_BIND_CIPHERS=ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
+      - SSL_BIND_OPTIONS=no-sslv3
+      - EXTRA_GLOBAL_SETTINGS=tune.ssl.default-dh-param 2048
+    depends_on:
+      - blender_cloud
+      - letsencrypt
+      - kibana
+    volumes:
+      - '/data/certs:/certs'
+      - /var/run/docker.sock:/var/run/docker.sock

docker/elastic/Dockerfile-elastic

@@ -0,0 +1,10 @@
+FROM docker.elastic.co/elasticsearch/elasticsearch:6.1.1
+LABEL maintainer Sybren A. Stüvel <sybren@blender.studio>
+
+RUN elasticsearch-plugin remove --purge  x-pack
+
+ADD elasticsearch.yml jvm.options /usr/share/elasticsearch/config/
+
+USER root
+RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/config/
+USER elasticsearch

docker/elastic/Dockerfile-kibana

@@ -0,0 +1,6 @@
+FROM docker.elastic.co/kibana/kibana:6.1.1
+LABEL maintainer Sybren A. Stüvel <sybren@blender.studio>
+
+RUN bin/kibana-plugin remove x-pack
+ADD kibana.yml /usr/share/kibana/config/kibana.yml
+RUN kibana 2>&1 | grep -m 1 "Optimization of .* complete"

docker/elastic/build.sh

@@ -0,0 +1,15 @@
+#!/bin/bash -e
+
+# When updating this, also update the versions in Dockerfile-*, and make sure that
+# it matches the versions of the elasticsearch and elasticsearch_dsl packages
+# used in Pillar. Those don't have to match exactly, but the major version should.
+VERSION=6.1.1
+
+docker build -t armadillica/elasticsearch:${VERSION} -f Dockerfile-elastic .
+docker build -t armadillica/kibana:${VERSION} -f Dockerfile-kibana .
+
+docker tag armadillica/elasticsearch:${VERSION} armadillica/elasticsearch:latest
+docker tag armadillica/kibana:${VERSION} armadillica/kibana:latest
+
+echo "Done, built armadillica/elasticsearch:${VERSION} and armadillica/kibana:${VERSION}"
+echo "Also tagged as armadillica/elasticsearch:latest and armadillica/kibana:latest"

docker/elastic/elasticsearch.yml

@@ -0,0 +1,7 @@
+cluster.name: "blender-cloud"
+network.host: 0.0.0.0
+
+# minimum_master_nodes need to be explicitly set when bound on a public IP
+# set to 1 to allow single node clusters
+# Details: https://github.com/elastic/elasticsearch/pull/17288
+discovery.zen.minimum_master_nodes: 1

docker/elastic/jvm.options

@@ -0,0 +1,112 @@
+## JVM configuration
+
+################################################################
+## IMPORTANT: JVM heap size
+################################################################
+##
+## You should always set the min and max JVM heap
+## size to the same value. For example, to set
+## the heap to 4 GB, set:
+##
+## -Xms4g
+## -Xmx4g
+##
+## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
+## for more information
+##
+################################################################
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+# Sybren: uncommented so that we can set those options using the ES_JAVA_OPTS environment variable.
+#-Xms512m
+#-Xmx512m
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+## GC configuration
+-XX:+UseConcMarkSweepGC
+-XX:CMSInitiatingOccupancyFraction=75
+-XX:+UseCMSInitiatingOccupancyOnly
+
+## optimizations
+
+# pre-touch memory pages used by the JVM during initialization
+-XX:+AlwaysPreTouch
+
+## basic
+
+# force the server VM (remove on 32-bit client JVMs)
+-server
+
+# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
+-Xss1m
+
+# set to headless, just in case
+-Djava.awt.headless=true
+
+# ensure UTF-8 encoding by default (e.g. filenames)
+-Dfile.encoding=UTF-8
+
+# use our provided JNA always versus the system one
+-Djna.nosys=true
+
+# use old-style file permissions on JDK9
+-Djdk.io.permissionsUseCanonicalPath=true
+
+# flags to configure Netty
+-Dio.netty.noUnsafe=true
+-Dio.netty.noKeySetOptimization=true
+-Dio.netty.recycler.maxCapacityPerThread=0
+
+# log4j 2
+-Dlog4j.shutdownHookEnabled=false
+-Dlog4j2.disable.jmx=true
+-Dlog4j.skipJansi=true
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+-XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps
+# ensure the directory exists and has sufficient space
+#-XX:HeapDumpPath=${heap.dump.path}
+
+## GC logging
+
+#-XX:+PrintGCDetails
+#-XX:+PrintGCTimeStamps
+#-XX:+PrintGCDateStamps
+#-XX:+PrintClassHistogram
+#-XX:+PrintTenuringDistribution
+#-XX:+PrintGCApplicationStoppedTime
+
+# log GC status to a file with time stamps
+# ensure the directory exists
+#-Xloggc:${loggc}
+
+# By default, the GC log file will not rotate.
+# By uncommenting the lines below, the GC log file
+# will be rotated every 128MB at most 32 times.
+#-XX:+UseGCLogFileRotation
+#-XX:NumberOfGCLogFiles=32
+#-XX:GCLogFileSize=128M
+
+# Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.
+# If documents were already indexed with unquoted fields in a previous version
+# of Elasticsearch, some operations may throw errors.
+#
+# WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided
+# only for migration purposes.
+#-Delasticsearch.json.allow_unquoted_field_names=true

docker/elastic/kibana.yml

@@ -0,0 +1,8 @@
+---
+
+server.name: kibana
+server.host: "0"
+elasticsearch.url: http://elasticsearch:9200
+
+# Hide dev tools
+console.enabled: false

docker/full_rebuild.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+
+set -xe
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+cd $DIR/1_base
+bash build.sh
+
+cd $DIR/2_buildpy
+bash build.sh
+
+cd $DIR/3_buildwheels
+bash build.sh
+
+cd $DIR/4_run
+bash build.sh

docker/mongo-backup.cron

@@ -0,0 +1,5 @@
+# Change to suit your needs, then place in /etc/cron.d/mongo-backup
+# (so remove the .cron from the name)
+
+MAILTO=yourname@youraddress.org
+30 5 * * * root /root/docker/mongo-backup.sh

docker/mongo-backup.sh

@@ -0,0 +1,28 @@
+#!/bin/bash -e
+
+BACKUPDIR=/data/storage/db-bak
+DATE=$(date +'%Y-%m-%d-%H%M')
+ARCHIVE=$BACKUPDIR/mongo-live-$DATE.tar.xz
+
+# Just a sanity check before we give it to 'rm -rf'
+if [ -z "$DATE" ]; then
+    echo "Empty string found where the date should be, aborting."
+    exit 1
+fi
+
+
+# /data/db-bak in Docker is /data/storage/db-bak on the host.
+docker exec mongo mongodump -d cloud \
+    --out /data/db-bak/dump-$DATE \
+    --excludeCollection tokens \
+    --excludeCollection flamenco_task_logs \
+    --quiet
+
+cd $BACKUPDIR
+tar -Jcf $ARCHIVE dump-$DATE/
+rm -rf dump-$DATE
+
+TO_DELETE="$(ls $BACKUPDIR/mongo-live-*.tar.xz | head -n -7)"
+[ -z "$TO_DELETE" ] || rm "$TO_DELETE"
+
+rsync -a $BACKUPDIR/mongo-live-*.tar.xz cloud-backup@swami-direct.blender.cloud:

docker/renew-letsencrypt.sh

@@ -0,0 +1,27 @@
+#!/bin/bash -e
+
+# First time creating a certificate for a domain, use:
+# certbot certonly --webroot -w /data/letsencrypt -d $DOMAINNAME
+
+cd /data/letsencrypt
+
+certbot renew
+
+echo
+echo "Recreating HAProxy certificates"
+
+for certdir in /etc/letsencrypt/live/*; do
+    domain=$(basename $certdir)
+    echo "  - $domain"
+
+    cat $certdir/privkey.pem $certdir/fullchain.pem > $domain.pem
+    mv $domain.pem /data/certs/
+done
+
+
+echo
+echo -n "Restarting "
+docker restart haproxy
+
+echo "Certificate renewal completed."
+

gulp

@@ -0,0 +1,31 @@
+#!/bin/bash -ex
+
+GULP=./node_modules/.bin/gulp
+
+function install() {
+    npm install
+    touch $GULP  # installer doesn't always touch this after a build, so we do.
+}
+
+# Rebuild Gulp if missing or outdated.
+[ -e $GULP ] || install
+[ gulpfile.js -nt $GULP ] && install
+
+if [ "$1" == "watch" ]; then
+    # Treat "gulp watch" as "gulp && gulp watch"
+    $GULP
+elif [ "$1" == "all" ]; then
+    # This is useful when building the Blender Cloud project for the first time.
+    # Run "gulp" once inside the repo
+    $GULP
+    # Run ./gulp in all depending projects (pillar, attract, flamenco, pillar-svnman)
+    declare -a repos=("pillar" "attract" "flamenco" "pillar-svnman")
+    for r in "${repos[@]}"
+    do
+        cd ../$r
+        ./gulp
+    done
+    exit 1
+fi
+
+exec $GULP "$@"

gulpfile.js

@@ -0,0 +1,160 @@
+var argv         = require('minimist')(process.argv.slice(2));
+var autoprefixer = require('gulp-autoprefixer');
+var cache        = require('gulp-cached');
+var chmod        = require('gulp-chmod');
+var concat       = require('gulp-concat');
+var git          = require('gulp-git');
+var gulp         = require('gulp');
+var gulpif       = require('gulp-if');
+var pug          = require('gulp-pug');
+var livereload   = require('gulp-livereload');
+var plumber      = require('gulp-plumber');
+var rename       = require('gulp-rename');
+var sass         = require('gulp-sass');
+var sourcemaps   = require('gulp-sourcemaps');
+var uglify       = require('gulp-uglify-es').default;
+
+var enabled = {
+    uglify: argv.production,
+    maps: !argv.production,
+    failCheck: !argv.production,
+    prettyPug: !argv.production,
+    cachify: !argv.production,
+    cleanup: argv.production,
+    chmod: argv.production,
+};
+
+var destination = {
+    css: 'cloud/static/assets/css',
+    pug: 'cloud/templates',
+    js: 'cloud/static/assets/js',
+}
+
+var source = {
+    pillar: '../pillar/',
+    bootstrap: 'node_modules/bootstrap/',
+    popper: 'node_modules/popper.js/'
+}
+
+
+/* CSS */
+gulp.task('styles', function() {
+    gulp.src('src/styles/**/*.sass')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.maps, sourcemaps.init()))
+        .pipe(sass({
+            outputStyle: 'compressed'}
+            ))
+        .pipe(autoprefixer("last 3 versions"))
+        .pipe(gulpif(enabled.maps, sourcemaps.write(".")))
+        .pipe(gulp.dest(destination.css))
+        .pipe(gulpif(argv.livereload, livereload()));
+});
+
+
+/* Templates - Pug */
+gulp.task('templates', function() {
+    gulp.src('src/templates/**/*.pug')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.cachify, cache('templating')))
+        .pipe(pug({
+            pretty: enabled.prettyPug
+        }))
+        .pipe(gulp.dest(destination.pug))
+        .pipe(gulpif(argv.livereload, livereload()));
+    // TODO(venomgfx): please check why 'gulp watch' doesn't pick up on .txt changes.
+    gulp.src('src/templates/**/*.txt')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.cachify, cache('templating')))
+        .pipe(gulp.dest(destination.pug))
+        .pipe(gulpif(argv.livereload, livereload()));
+});
+
+
+/* Individual Uglified Scripts */
+gulp.task('scripts', function() {
+    gulp.src('src/scripts/*.js')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.cachify, cache('scripting')))
+        .pipe(gulpif(enabled.maps, sourcemaps.init()))
+        .pipe(gulpif(enabled.uglify, uglify()))
+        .pipe(rename({suffix: '.min'}))
+        .pipe(gulpif(enabled.maps, sourcemaps.write(".")))
+        .pipe(gulpif(enabled.chmod, chmod(644)))
+        .pipe(gulp.dest(destination.js))
+        .pipe(gulpif(argv.livereload, livereload()));
+});
+
+
+/* Collection of scripts in src/scripts/tutti/ to merge into tutti.min.js */
+/* Since it's always loaded, it's only for functions that we want site-wide */
+gulp.task('scripts_concat_tutti', function() {
+    gulp.src('src/scripts/tutti/**/*.js')
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.maps, sourcemaps.init()))
+        .pipe(concat("tutti.min.js"))
+        .pipe(gulpif(enabled.uglify, uglify()))
+        .pipe(gulpif(enabled.maps, sourcemaps.write(".")))
+        .pipe(gulpif(enabled.chmod, chmod(644)))
+        .pipe(gulp.dest(destination.js))
+        .pipe(gulpif(argv.livereload, livereload()));
+});
+
+
+// Combine all needed Bootstrap JavaScript into a single file.
+gulp.task('scripts_concat_bootstrap', function() {
+
+    toUglify = [
+        source.popper    + 'dist/umd/popper.min.js',
+        source.bootstrap + 'js/dist/index.js',
+        source.bootstrap + 'js/dist/util.js',
+        source.bootstrap + 'js/dist/tooltip.js',
+        source.bootstrap + 'js/dist/dropdown.js',
+    ];
+
+    gulp.src(toUglify)
+        .pipe(gulpif(enabled.failCheck, plumber()))
+        .pipe(gulpif(enabled.maps, sourcemaps.init()))
+        .pipe(concat("bootstrap.min.js"))
+        .pipe(gulpif(enabled.uglify, uglify()))
+        .pipe(gulpif(enabled.maps, sourcemaps.write(".")))
+        .pipe(gulpif(enabled.chmod, chmod(644)))
+        .pipe(gulp.dest(destination.js))
+        .pipe(gulpif(argv.livereload, livereload()));
+});
+
+
+// While developing, run 'gulp watch'
+gulp.task('watch',function() {
+    // Only listen for live reloads if ran with --livereload
+    if (argv.livereload){
+        livereload.listen();
+    }
+
+    gulp.watch('src/styles/**/*.sass',['styles']);
+    gulp.watch(source.pillar + 'src/styles/**/*.sass',['styles']);
+
+    gulp.watch('src/templates/**/*.pug',['templates']);
+    gulp.watch('src/scripts/*.js',['scripts']);
+    gulp.watch('src/scripts/tutti/**/*.js',['scripts_concat_tutti']);
+});
+
+// Erases all generated files in output directories.
+gulp.task('cleanup', function() {
+    var paths = [];
+    for (attr in destination) {
+        paths.push(destination[attr]);
+    }
+
+    git.clean({ args: '-f -X ' + paths.join(' ') }, function (err) {
+        if(err) throw err;
+    });
+
+});
+
+
+// Run 'gulp' to build everything at once
+var tasks = [];
+if (enabled.cleanup) tasks.push('cleanup');
+
+gulp.task('default', tasks.concat(['styles', 'templates', 'scripts', 'scripts_concat_tutti', 'scripts_concat_bootstrap']));

manage.py

@@ -1,40 +1,7 @@
 #!/usr/bin/env python
 
-from __future__ import print_function
-
-import logging
-from flask import current_app
 from pillar import cli
-from pillar.cli import manager_maintenance
-from cloud import app
-
-log = logging.getLogger(__name__)
-
-
-@manager_maintenance.command
-def reconcile_subscribers():
-    """For every user, check their subscription status with the store."""
-    from pillar.auth.subscriptions import fetch_user
-
-    users_coll = current_app.data.driver.db['users']
-    unsubscribed_users = []
-    for user in users_coll.find({'roles': 'subscriber'}):
-        print('Processing %s' % user['email'])
-        print('  Checking subscription')
-        user_store = fetch_user(user['email'])
-        if user_store['cloud_access'] == 0:
-            print('    Removing subscriber role')
-            users_coll.update(
-                {'_id': user['_id']},
-                {'$pull': {'roles': 'subscriber'}})
-            unsubscribed_users.append(user['email'])
-
-    if not unsubscribed_users:
-        return
-
-    print('The following users have been unsubscribed')
-    for user in unsubscribed_users:
-        print(user)
+from runserver import app
 
 cli.manager.app = app
 cli.manager.run()

package.json

@@ -0,0 +1,31 @@
+{
+	"name": "blender-cloud",
+	"license": "GPL-2.0+",
+	"author": "Blender Institute",
+	"repository": {
+		"type": "git",
+		"url": "git://git.blender.org/blender-cloud.git"
+	},
+	"devDependencies": {
+		"gulp": "~3.9.1",
+		"gulp-autoprefixer": "~6.0.0",
+		"gulp-cached": "~1.1.1",
+		"gulp-chmod": "~2.0.0",
+		"gulp-concat": "~2.6.1",
+		"gulp-if": "^2.0.2",
+		"gulp-git": "~2.8.0",
+		"gulp-livereload": "~4.0.0",
+		"gulp-plumber": "~1.2.0",
+		"gulp-pug": "~4.0.1",
+		"gulp-rename": "~1.4.0",
+		"gulp-sass": "~4.0.1",
+		"gulp-sourcemaps": "~2.6.4",
+		"gulp-uglify-es": "^1.0.4",
+		"minimist": "^1.2.0"
+	},
+	"dependencies": {
+		"bootstrap": "^4.1.3",
+		"jquery": "^3.3.1",
+		"popper.js": "^1.14.4"
+	}
+}

requirements-dev.txt

@@ -0,0 +1,12 @@
+-r ../pillar-python-sdk/requirements-dev.txt
+-r ../pillar/requirements-dev.txt
+-r ../attract/requirements-dev.txt
+-r ../flamenco/requirements-dev.txt
+-r ../pillar-svnman/requirements-dev.txt
+
+-e ../pillar-python-sdk
+-e ../pillar
+-e ../attract
+-e ../flamenco
+-e ../pillar-svnman
+-e .

requirements.txt

@@ -1,67 +1,4 @@
-# Primary requirements
-# pillarsdk
-# pillar
-# attract
-# flamenco
-
-# Secondary requirements (i.e. pulled in from primary requirements)
-algoliasearch==1.8.0
-attrs==16.2.0
-bcrypt==2.0.0
-blinker==1.4
-bugsnag==2.3.1
-bleach==1.4.3
-Cerberus==0.9.2
-cffi==1.7.0
-commonmark==0.7.2
-cryptography==1.4
-enum34==1.1.6
-Eve==0.6.3
-Events==0.2.1
-Flask==0.10.1
-Flask-Cache==0.13.1
-Flask-Script==2.0.5
-Flask-Login==0.3.2
-Flask-OAuthlib==0.9.3
-Flask-PyMongo==0.4.1
-Flask-WTF==0.12
-flup==1.0.2
-future==0.15.2
-gcloud==0.12.0
-google-apitools==0.4.11
-googleapis-common-protos==1.2.0
-html5lib==0.9999999
-httplib2==0.9.2
-idna==2.0
-ipaddress==1.0.16
-itsdangerous==0.24
-Jinja2==2.8
-MarkupSafe==0.23
-markdown==2.6.7
-ndg-httpsclient==0.4.0
-oauth2client==3.0.0
-oauthlib==1.1.2
-pathlib2==2.2.1
-Pillow==2.8.1
-protobuf==3.0.0
-protorpc==0.11.1
-pyasn1==0.1.9
-pyasn1-modules==0.0.8
-pycparser==2.14
-pycrypto==2.6.1
-pylru==1.0.4
-pymongo==3.3.0
-pyOpenSSL==0.15.1
-python-dateutil==2.5.3
-redis==2.10.5
-requests==2.9.1
-requests-oauthlib==0.6.2
-rsa==3.4.2
-scandir==1.4
-simplejson==3.8.2
-six==1.10.0
-svn==0.3.43
-WebOb==1.5.0
-Werkzeug==0.11.10
-WTForms==2.1
-zencoder==0.6.5
+-r ../pillar/requirements.txt
+-r ../attract/requirements.txt
+-r ../flamenco/requirements.txt
+-r ../pillar-svnman/requirements.txt

rsync_ui.sh

@@ -1,42 +0,0 @@
-#!/usr/bin/env bash
-
-set -e  # error out when one of the commands in the script errors.
-
-PILLAR_DIR=$(python <<EOT
-from __future__ import print_function
-import os.path
-import pillar
-
-print(os.path.dirname(os.path.dirname(pillar.__file__)))
-EOT
-)
-
-ASSETS="$PILLAR_DIR/pillar/web/static/assets/"
-TEMPLATES="$PILLAR_DIR/pillar/web/templates/"
-
-if [ ! -d "$ASSETS" ]; then
-    echo "Unable to find assets dir $ASSETS"
-    exit 1
-fi
-
-cd $PILLAR_DIR
-if [ $(git rev-parse --abbrev-ref HEAD) != "production" ]; then
-    echo "You are NOT on the production branch, refusing to rsync_ui." >&2
-    exit 1
-fi
-
-echo
-echo "*** GULPA GULPA ***"
-if [ -x ./node_modules/.bin/gulp ]; then
-    ./node_modules/.bin/gulp --production
-else
-    gulp --production
-fi
-
-echo
-echo "*** SYNCING ASSETS ***"
-rsync -avh $ASSETS root@cloud.blender.org:/data/git/pillar/pillar/web/static/assets/
-
-echo
-echo "*** SYNCING TEMPLATES ***"
-rsync -avh $TEMPLATES root@cloud.blender.org:/data/git/pillar/pillar/web/templates/

runserver.py

@@ -3,13 +3,19 @@
 from pillar import PillarServer
 from attract import AttractExtension
 from flamenco import FlamencoExtension
+from svnman import SVNManExtension
+from cloud import CloudExtension
 
 attract = AttractExtension()
 flamenco = FlamencoExtension()
+svnman = SVNManExtension()
+cloud = CloudExtension()
 
 app = PillarServer('.')
 app.load_extension(attract, '/attract')
 app.load_extension(flamenco, '/flamenco')
+app.load_extension(svnman, '/svn')
+app.load_extension(cloud, None)
 app.process_extensions()
 
 if __name__ == '__main__':

runserver.wsgi

@@ -1,22 +1,23 @@
 from os.path import abspath, dirname
 import sys
 
-activate_this = '/data/venv/bin/activate_this.py'
-execfile(activate_this, dict(__file__=activate_this))
-from flup.server.fcgi import WSGIServer
+my_path = dirname(abspath(__file__))
+sys.path.append(my_path)
+
 from pillar import PillarServer
 from attract import AttractExtension
 from flamenco import FlamencoExtension
-
-sys.path.append('/data/git/blender-cloud/')
+from svnman import SVNManExtension
+from cloud import CloudExtension
 
 attract = AttractExtension()
 flamenco = FlamencoExtension()
+svnman = SVNManExtension()
+cloud = CloudExtension()
 
-application = PillarServer(dirname(abspath(__file__)))
+application = PillarServer(my_path)
 application.load_extension(attract, '/attract')
 application.load_extension(flamenco, '/flamenco')
+application.load_extension(svnman, '/svn')
+application.load_extension(cloud, None)
 application.process_extensions()
-
-if __name__ == '__main__':
-    WSGIServer(application).run()

setup.cfg

@@ -0,0 +1,5 @@
+[tool:pytest]
+addopts = -v --cov cloud --cov-report term-missing --ignore node_modules --ignore docker
+
+[pep8]
+max-line-length = 100

src/scripts/tagged_assets.js

@@ -0,0 +1,109 @@
+/**
+ * Support for fetching & rendering assets by tags.
+ */
+(function($) {
+    /* How many nodes to load initially, and when clicked on the 'Load Next' link. */
+    const LOAD_INITIAL_COUNT = 5;
+    const LOAD_NEXT_COUNT = 3;
+
+    /* Renders a node as a <li> element, returns a jQuery object. */
+    function renderAsset(node) {
+        let li = $('<li>').addClass('tagged-asset');
+        let link = $('<a>')
+            .attr('href', '/nodes/' + node._id + '/redir')
+            .appendTo(li);
+
+        function warnNoPicture() {
+            li.addClass('warning');
+            link.text('no picture for node ' + node._id);
+        }
+
+        if (!node.picture) {
+            warnNoPicture();
+            return li;
+        }
+
+        // TODO: show 'loading' thingy
+        $.get('/api/files/' + node.picture)
+            .fail(function(error) {
+                let msg = xhrErrorResponseMessage(error);
+                li.addClass('error').text(msg);
+            })
+            .done(function(resp) {
+                // Render the picture if it has the proper size.
+                var show_variation = null;
+                if (typeof resp.variations != 'undefined') {
+                    for (variation of resp.variations) {
+                        if (variation.size != 'm') continue;
+                        show_variation = variation;
+                        break;
+                    }
+                }
+
+                if (show_variation == null) {
+                    warnNoPicture();
+                    return;
+                }
+
+                let img = $('<img>')
+                    .attr('alt', node.name)
+                    .attr('src', variation.link)
+                    .attr('width', variation.width)
+                    .attr('height', variation.height);
+                link.append(img);
+            });
+
+        return li;
+    }
+
+    function loadNext(ul_element) {
+        let $ul = $(ul_element);
+        let tagged_assets = ul_element.tagged_assets;  // Stored here by loadTaggedAssets().
+        let already_loaded = $ul.find('li.tagged-asset').length;
+
+        let load_next = $ul.find('li.load-next');
+
+        let nodes_to_load = tagged_assets.slice(already_loaded, already_loaded + LOAD_NEXT_COUNT);
+        for (node of nodes_to_load) {
+            let li = renderAsset(node);
+            load_next.before(li);
+        }
+
+        if (already_loaded + LOAD_NEXT_COUNT >= tagged_assets.length)
+            load_next.remove();
+    }
+
+    $.fn.loadTaggedAssets = function(api_base_url) {
+        this.each(function(index, ul_element) {
+            // TODO(Sybren): show a 'loading' animation.
+            $.get('/api/nodes/tagged/' + ul_element.dataset.assetTag)
+                .fail(function(error) {
+                    let msg = xhrErrorResponseMessage(error);
+                    $('<li>').addClass('error').text(msg).appendTo(ul_element);
+                })
+                .done(function(resp) {
+                    // 'resp' is a list of node documents.
+                    // Store the response on the DOM <ul>-element so that we can later render more.
+                    ul_element.tagged_assets = resp;
+
+                    // Here render the first N.
+                    for (node of resp.slice(0, LOAD_INITIAL_COUNT)) {
+                        let li = renderAsset(node);
+                        li.appendTo(ul_element);
+                    }
+
+                    // Don't bother with a 'load next' link if there is no more.
+                    if (resp.length <= LOAD_INITIAL_COUNT) return;
+
+                    // Construct the 'load next' link.
+                    let load_next = $('<li>').addClass('load-next');
+                    let link = $('<a>')
+                        .attr('href', 'javascript:void(0);')
+                        .click(function() { loadNext(ul_element); return false; })
+                        .text('Load next')
+                        .appendTo(load_next);
+                    load_next.appendTo(ul_element);
+                });
+        });
+    };
+}(jQuery));

src/styles/_about.sass

@@ -0,0 +1,9 @@
+section.team
+	h2, .people-container
+		text-align: center
+	h3
+		margin-bottom: 0
+	h3 small
+		display: block
+	.people-intro, .row
+		margin-bottom: 20px

src/styles/_homepage.sass

@@ -0,0 +1,502 @@
+
+
+.title-underline
+	padding-bottom: 5px
+	position: relative
+	margin-bottom: 20px
+
+	&:before
+		background-color: $primary
+		content: ' '
+		display: block
+		height: 2px
+		top: 125%
+		position: absolute
+		width: 50px
+
+
+nav#nav-tabs,
+nav#sub-nav-tabs
+	ul#nav-tabs__list,
+	ul#sub-nav-tabs__list
+		margin: 0
+		padding: 0
+		list-style: none
+		border-bottom: thin solid $color-background
+		+clearfix
+
+		li.nav-tabs__list-tab
+			float: left
+			border: none
+			border-bottom: 3px solid transparent
+			color: $color-text-dark-primary
+			user-select: none
+
+			&:hover
+				border-color: rgba($color-secondary, .3)
+				cursor: pointer
+				color: $color-text-dark
+				a
+					color: $color-text-dark
+
+			a
+				display: block
+				text-decoration: none
+				padding: 10px 15px 5px
+				color: $color-text-dark-primary
+
+			i
+				margin-right: 5px
+				color: $color-text-dark-secondary
+				font-size: .9em
+
+				&.pi-blender
+					margin-right: 10px
+
+			span
+				color: $color-text-dark-hint
+				margin-left: 5px
+
+			&.active
+				border-color: $color-secondary
+				color: $color-secondary-dark
+				a, i
+					color: $color-secondary-dark
+
+			&.disabled
+				border-color: $color-background-light
+				color: $color-text-dark-hint
+				cursor: default
+				a, i
+					color: $color-text-dark-hint
+
+				&:hover
+					border-color: $color-background-light
+					pointer-events: none
+
+.dashboard-container
+	word-break: break-word
+
+section.stream
+	ul.activity-stream__list
+		$activity-stream-thumbnail-size: 110px
+
+		> li
+			position: relative
+			display: flex
+			padding: 10px 0
+			overflow: hidden
+			border-top: thin solid $color-background-dark
+
+			&:first-child
+				border: none
+
+			&.active .activity-stream__list-details .title
+				color: $color-primary
+
+			&:hover
+				.title
+					text-decoration: underline
+				&.video
+					a.image
+						&:hover
+							i
+								font-size: 3.5em
+							img
+								opacity: .9
+						img
+							opacity: .7
+							z-index: 0
+							transition: opacity 150ms ease-in-out
+
+						i
+							+position-center-translate
+							z-index: 1
+							color: rgba(white, .6)
+							font-size: 3em
+							transition: font-size 100ms ease-in-out
+
+			&.comment
+				.activity-stream__list-details
+					padding: 0
+					.title
+						color: $color-text-dark
+						padding: 7px 10px 2px 10px
+						font-size: 1em
+						margin: 0
+
+					ul.meta
+						+list-meta
+						font-size: .9em
+						padding: 0 10px 7px 10px
+
+						li
+							&.where-parent:before
+								content: '\e83a'
+								font-family: 'pillar-font'
+
+							&.what:before
+								display: none
+
+			&.post
+				.activity-stream__list-thumbnail
+					border-color: $node-type-post
+					background-color: $node-type-post
+				.activity-stream__list-details .title
+					color: darken($node-type-post, 15%)
+					font:
+						size: 1.3em
+						weight: 500
+
+			&.asset, &.comment, &.post
+				&:hover
+					cursor: pointer
+			&.empty
+				display: none
+				color: $color-text-dark-primary
+				padding: 20px
+				text-align: center
+				span
+					color: $color-primary
+					&:hover
+						text-decoration: underline
+						cursor: pointer
+
+			&.with-picture
+				min-height: $activity-stream-thumbnail-size
+
+				.activity-stream__list-thumbnail
+					background-color: black
+					width: $activity-stream-thumbnail-size * 1.69
+					min-width: $activity-stream-thumbnail-size * 1.69
+
+					.activity-stream__list-thumbnail-icon
+						position: absolute
+						top: 0
+						left: 0
+						right: 0
+						bottom: 0
+						font-size: 1.3em
+						text-shadow: 1px 1px 0 rgba(black, .2)
+						background-image: linear-gradient(10deg, rgba(black, .5) 0%, transparent 40%)
+
+						i
+							position: absolute
+							bottom: -8px
+							left: 20px
+							top: initial
+							right: initial
+							color: white
+
+			.activity-stream__list-thumbnail
+				position: relative
+				display: flex
+				justify-content: center
+				align-items: center
+				overflow: hidden
+				width: 35px
+				height: auto
+				min-width: 35px
+				min-height: auto
+
+				+media-xs
+					display: none
+
+
+				&.image i
+					color: $node-type-asset_image
+				&.file i
+					color: $node-type-asset_file
+				&.video i
+					color: $node-type-asset_video
+
+				i
+					+position-center-translate
+					left: 23px
+					top: 21px
+					font-size: 1.1em
+
+				img
+					max-height: $activity-stream-thumbnail-size
+					+position-center-translate
+
+			.activity-stream__list-details
+				display: flex
+				flex-direction: column
+				justify-content: space-around
+				flex: 1
+				overflow: hidden
+				position: relative
+				max-width: 100%
+				padding: 10px 0
+
+				+media-xs
+					margin-left: 0
+
+				.ribbon
+					+ribbon
+					right: -47px
+					top: 5px
+					font-size: 12px
+
+					span
+						padding: 1px 50px
+
+				.title
+					padding: 0 10px
+					color: $color-text-dark
+
+					span
+						@include badge(hsl(hue($color-success), 60%, 45%), 3px)
+						font-size: .7em
+						padding: 1px 5px
+						margin-right: 5px
+
+section.comments
+	padding: 0 15px 5px
+
+	ul
+		padding: 0
+
+	> ul
+		list-style-type: none
+		margin: 10px 0 0
+
+		> li
+			+text-overflow-ellipsis
+			border-top: thin solid $color-background-dark
+			padding: 10px 0
+
+			&:first-child
+				border: none
+
+			> a
+				+text-overflow-ellipsis
+				color: $color-text
+				display: block
+				padding-bottom: 5px
+
+section.random-asset
+	border-bottom: thin solid $color-background-dark
+
+	ul.random-asset__list
+		list-style: none
+		padding: 0
+
+		> li
+			align-items: center
+			border-top: thin solid $color-background
+			display: flex
+			padding: 7px 0
+			position: relative
+			overflow: hidden
+
+			&:first-child
+				border-top: none
+
+			.ribbon
+				+ribbon
+				right: -47px
+				top: 5px
+				font:
+					size: 12px
+					weight: 500
+
+				z-index: 1
+
+				span
+					padding: 1px 50px
+
+			.random-asset__list-thumbnail
+				background-color: $color-background
+				display: block
+				height: 50px
+				margin-right: 15px
+				min-height: 50px
+				min-width: 50px
+				overflow: hidden
+				position: relative
+				width: 50px
+
+				img
+					width: 100%
+
+				i
+					+position-center-translate
+					font-size: 1.6em
+					color: $color-text-light
+
+				&.image
+					background-color: $node-type-asset_image
+				&.file
+					background-color: $node-type-asset_file
+					font-size: .8em
+				&.video
+					background-color: $node-type-asset_video
+					font-size: .8em
+				&.None
+					background-color: $node-type-group
+
+			.random-asset__list-details
+				.title
+					display: block
+					font-size: 1em
+					color: $color-text-dark-primary
+
+					&:hover
+						color: $color-primary
+
+				ul.meta
+					+list-meta
+					padding-top: 5px
+					font-size: .9em
+
+					li
+						&:before
+							left: -5px
+						&.what
+							text-transform: capitalize
+
+			&.featured
+				align-items: flex-start
+				flex-direction: column
+				padding: 0
+
+				a.title
+					font-size: 1.1em
+					padding: 10px 0 5px
+					display: block
+					color: $color-text
+
+					&:hover
+						color: $color-primary
+
+				a.random-asset__thumbnail
+					display: block
+					position: relative
+
+					&.video
+						background-color: black
+						img
+							opacity: .7
+
+					img
+						transition: opacity 150ms ease-in-out
+						width: 100%
+						max-width: 100%
+
+					i
+						+position-center-translate
+						color: white
+						font-size: 3em
+						text-shadow: 0 0 25px black
+						transition: font-size 150ms ease-in-out
+
+					&:hover
+						i
+							font-size: 3.5em
+						img
+							opacity: .85
+
+				ul.meta
+					+list-meta
+					padding-bottom: 10px
+
+
+section.announcement
+	+container-box
+	margin-left: 15px
+	margin-right: 15px
+
+	.header-icons
+		display: flex
+		align-items: center
+		justify-content: center
+		padding: 20px 0 5px 0
+
+		i
+			font-size: 2.5em
+			color: $color-info
+
+			&.pi-heart-filled
+				color: $color-danger
+				margin-left: 5px
+
+	img.header
+		width: 100%
+		margin: 0 auto
+		border-top-left-radius: 3px
+		border-top-right-radius: 3px
+
+	iframe
+		width: 100%
+		position: relative
+		left: 15px
+		margin: 25px auto
+
+		+media-sm
+			height: 500px
+		+media-md
+			height: 520px
+		+media-lg
+			height: 580px
+
+	.text
+		padding: 15px
+
+		.title
+			padding-bottom: 10px
+
+			+media-xs
+				font-size: 1.4em
+
+			strong
+				color: $color-primary-dark
+
+			a
+				color: $color-text-dark-primary
+
+		.lead
+			font-size: 1em
+			+list-bullets
+
+			ul
+				margin-top: 10px
+				padding-left: 10px
+
+			hr
+				border: none
+				height: 1px
+				width: 100%
+				margin: 10px 0
+				background-color: $color-background
+				clear: both
+
+			+media-xs
+				padding-left: 10px
+
+		.buttons
+			margin: 15px auto 0 auto
+			display: flex
+			align-items: center
+			justify-content: space-around
+			flex-wrap: wrap
+
+
+body.homepage
+	.blog
+		// Custom tweak to Bootstrap grid for the only case when
+		// the post is inside a column (it's usually centered in the page).
+		.col-md-9
+			flex: 1
+			max-width: 100%
+
+		.jumbotron
+			padding-top: 6em
+			padding-bottom: 6em
+
+			*
+				font-size: $h1-font-size
+
+			.lead
+				font-size: $font-size-base

src/styles/_services.sass

@@ -0,0 +1,39 @@
+.services
+	#page-header
+		text-shadow: 1px 1px 0 rgba(black, .2)
+		border-bottom: none
+		align-items: initial
+
+		.page-title
+			text-align: left
+			font-size: 3em
+			margin: 0
+
+		.page-title-summary
+			max-width: 900px
+			text-align: left
+			font-size: 1.4em
+
+	.page-card-side
+		img
+			max-width: 100%
+			border-radius: 3px
+
+	.tip
+		margin-top: 15px
+		color: $color-text-dark-secondary
+		font-size: .8em
+
+		a
+			color: $color-primary
+			text-decoration: underline
+
+	span.text-background
+		+text-background(white, #358, 2px, 5px 0)
+
+	.navbar-backdrop-overlay
+		background-image: linear-gradient(rgba(black, .3), rgba(black, .8))
+
+	#blender-sync
+		small strong
+			color: $color-success

src/styles/_welcome.sass

@@ -0,0 +1,380 @@
+.join
+	position: relative
+
+	nav.navbar
+		background-color: white
+
+		.navbar-brand
+			color: $color-text
+
+		li a.navbar-item
+			color: $color-text
+
+		.navbar-toggle
+			border: 2px solid $color-text-dark-primary
+			color: $color-text
+
+		.navbar-nav
+			+media-xs
+				padding: 10px
+
+	#page-header
+		+media-lg
+			min-height: 600px
+		+media-xs
+			background-size: cover
+			background-position: right
+		min-height: 500px
+
+		.page-title,
+		.page-title-summary
+			line-height: 1.2em
+			text-align: left
+			text-shadow: 1px 1px 0 rgba(black, .5), 0 0 25px rgba(black, .5)
+
+		.page-title-summary
+			font-size: 1.4em
+
+	li.special
+		color: $color-success
+		font-weight: bold
+
+	.page-card
+		&:nth-child(even)
+			background-color: white
+
+		&-header
+			margin-bottom: 0
+
+		&-side
+			+media-xs
+				width: 100%
+				max-width: 100%
+
+			&.text
+				+media-xs
+					padding-left: 50px
+				padding:
+					bottom: 150px
+					top: 150px
+
+		&.right
+			background-color: $color-background-light
+
+			.page-card-title,
+			.page-card-summary
+				padding-left: 25px
+
+			.page-card-title:after
+				left: 25px
+
+			.text
+				+media-xs
+					padding-left: 25px
+
+		&.intro
+			+media-xs
+				margin-left: 50px
+
+			.page-card-side
+				+media-sm
+					max-width: 500px
+				max-width: 600px
+
+			padding:
+				bottom: 0
+				top: 50px
+
+			&+.page-card-header
+				padding-top: 0
+
+			.page-card-summary
+				font-size: 1.2em
+
+			.page-card-image
+				align-items: center
+				display: flex
+				height: 100%
+				justify-content: center
+
+		&.light
+			overflow: hidden
+			position: relative
+
+			.learn
+				color: white
+				text-decoration: underline
+
+			.page-card
+				&-title
+					color: white
+					font-weight: bold
+
+					&:after
+						border-color: white
+
+				&-summary
+					color: white
+					position: relative
+					text-shadow: 1px 1px 0 rgba(black, .5), 0 0 25px rgba(black, .2)
+					z-index: 1
+
+					p
+						font-size: .9em
+
+						&.summary-action
+							font-size: .8em
+
+							a.page-card-cta
+								font-size: .9em
+
+					a
+						color: white
+						text-decoration: underline
+
+
+			a.page-card-cta
+				background: #ff4970
+				border-radius: 3px
+				border: none
+				box-shadow: 1px 1px 0 rgba(black, .2)
+				font-weight: bold
+				margin-right: 15px
+				margin-top: 25px
+				padding: 7px 20px
+				text-decoration: none
+				text-shadow: none
+
+				&:hover
+					background: lighten(#ff4970, 5%)
+
+			.page-card-image
+				img
+					+media-xs
+						display: none
+					max-width: initial
+					position: absolute
+					width: initial
+					z-index: 0
+
+		&.training
+			margin-top: 45px
+
+			a.page-card-cta
+				background: #0082ff
+
+			.page-card-image
+				img
+					right: 50px
+					top: 50%
+					transform: translateY(-50%)
+
+		&.open-movies
+			.page-card-image
+				img
+					top: 50px
+					left: 50px
+
+		&.services
+			.page-card-image
+				img
+					left: 50px
+
+		&.subscribe
+			padding: 40px 0
+			text-align: center
+
+			span strong
+				color: $color-danger
+
+			.page-card-side
+				width: 100%
+				max-width: 100%
+
+			.page-card-title
+				line-height: 1.5em
+				color: white
+				text-align: center
+				padding-bottom: 15px
+
+				&:after
+					border: none
+
+			.page-card-summary
+				color: $color-text-light-primary
+
+			.page-card-cta
+				text-align: center
+				font-size: 1.3em
+				padding: 7px 35px
+				margin-right: initial
+				border: thin solid rgba(white, .8)
+				border-radius: 3px
+
+
+	.training-other
+		color: $color-text-dark-secondary
+		font-size: .9em
+		padding: 30px
+		text-align: center
+
+		a
+			color: $color-text-dark-secondary
+			text-decoration: underline
+
+			&:hover
+				color: $color-text-dark-primary
+
+	.pricing
+		+media-xs
+			padding-top: 30px
+		margin-bottom: 0
+		padding-bottom: 100px
+
+
+	.supported-by
+		padding: 50px 0
+		text-align: center
+		background-color: $color-background
+		border-top: 5px solid $color-background-dark
+
+		img.logos
+			padding: 60px 0 100px 0
+			max-height: 500px
+			max-width: 100%
+
+			+media-xs
+				max-width: 100%
+
+	.assets
+		padding-bottom: 80px
+
+	.flex
+		+media-xs
+			flex-direction: column
+
+		display: flex
+
+		.box
+			flex: 1
+			margin: 20px
+
+	.page-triplet-container
+		.triplet-card
+			+media-xs
+				margin-top: 20px
+
+
+.container.wide-on-sm
+	+media-sm
+		width: 100%
+
+
+section.pricing
+	padding: 50px 0
+	margin-bottom: 25px
+	background-color: $color-background-light
+	+clearfix
+	+media-xs
+		padding: 0
+
+	h2
+		text-align: center
+		font-size: 2.3em
+		text-shadow: 1px 1px 0 white
+		margin:
+			top: 10px
+			bottom: 50px
+		padding: 0
+
+		+media-xs
+			font-size: 1.6em
+
+	.box
+		margin-top: 40px
+		padding: 20px 20px 60px 20px
+		border: 1px solid $color-text-dark-hint
+		background-color: white
+		position: relative
+		text-align: center
+		border-top: 3px solid rgba($color-text-dark-secondary, .5)
+		border-bottom-left-radius: 3px
+		border-bottom-right-radius: 3px
+
+		+media-xs
+			margin-bottom: 15px
+
+		&.yearly
+			border-top: 3px solid $color-info
+			transform: scale(1.1)
+
+			+media-xs
+				transform: scale(1)
+
+			a.sign-up-now
+				+button($color-primary, 3px, true)
+
+		h3
+			font-size: 1.8em
+			padding-bottom: 0
+			margin: 25px 0 0 10px
+
+		.pricing-display
+			position: relative
+			color: $color-info
+			padding: 10px 0
+
+			.currency-sign,
+			.digit-dec
+				font-size: 1.7em
+				position: relative
+				top: -25px
+				font-weight: 100
+
+			.digit-int
+				font-size: 4.5em
+				font-weight: 100
+
+		.pricing-caption
+			color: $color-text-dark-hint
+			text-align: center
+			padding-bottom: 25px
+
+			ul
+				text-align: left
+
+		+list-bullets
+
+		ul
+			color: $color-text-dark-primary
+			padding-left: 10px
+			margin-top: 15px
+
+			li
+				padding-bottom: 8px
+
+		.sign-up-now
+			position: absolute
+			bottom: 25px
+			width: 65%
+			left: 50%
+			transform: translateX(-50%)
+			font-size: 1.2em
+
+			+button($color-primary, 3px)
+			padding: 5px 25px
+			white-space: nowrap
+			text-align: center
+
+	.education
+		color: $color-text-dark-primary
+		font-size: 1.1em
+		padding: 25px 15px 0
+		text-align: center
+
+		h3
+			color: $color-primary-dark
+
+		.btn
+			margin-top: 15px
+			min-width: 200px

src/styles/main.sass

@@ -0,0 +1,99 @@
+// Bootstrap variables and utilities.
+@import "../../node_modules/bootstrap/scss/functions"
+@import "../../node_modules/bootstrap/scss/variables"
+@import "../../node_modules/bootstrap/scss/mixins"
+
+// Pillar variables and utilities.
+@import "../../../pillar/src/styles/config"
+@import "../../../pillar/src/styles/utils"
+
+
+// Bootstrap components.
+@import "../../node_modules/bootstrap/scss/root"
+@import "../../node_modules/bootstrap/scss/reboot"
+
+@import "../../node_modules/bootstrap/scss/type"
+@import "../../node_modules/bootstrap/scss/images"
+
+@import "../../node_modules/bootstrap/scss/code"
+@import "../../node_modules/bootstrap/scss/grid"
+@import "../../node_modules/bootstrap/scss/tables"
+@import "../../node_modules/bootstrap/scss/forms"
+@import "../../node_modules/bootstrap/scss/buttons"
+@import "../../node_modules/bootstrap/scss/transitions"
+@import "../../node_modules/bootstrap/scss/dropdown"
+@import "../../node_modules/bootstrap/scss/button-group"
+@import "../../node_modules/bootstrap/scss/input-group"
+@import "../../node_modules/bootstrap/scss/custom-forms"
+
+@import "../../node_modules/bootstrap/scss/nav"
+@import "../../node_modules/bootstrap/scss/navbar"
+
+@import "../../node_modules/bootstrap/scss/card"
+@import "../../node_modules/bootstrap/scss/breadcrumb"
+@import "../../node_modules/bootstrap/scss/pagination"
+@import "../../node_modules/bootstrap/scss/badge"
+@import "../../node_modules/bootstrap/scss/jumbotron"
+@import "../../node_modules/bootstrap/scss/alert"
+@import "../../node_modules/bootstrap/scss/progress"
+@import "../../node_modules/bootstrap/scss/media"
+@import "../../node_modules/bootstrap/scss/list-group"
+@import "../../node_modules/bootstrap/scss/close"
+@import "../../node_modules/bootstrap/scss/modal"
+@import "../../node_modules/bootstrap/scss/tooltip"
+@import "../../node_modules/bootstrap/scss/popover"
+@import "../../node_modules/bootstrap/scss/carousel"
+
+@import "../../node_modules/bootstrap/scss/utilities"
+@import "../../node_modules/bootstrap/scss/print"
+
+
+// Pillar components.
+@import "../../../pillar/src/styles/apps_base"
+@import "../../../pillar/src/styles/error"
+
+@import "../../../pillar/src/styles/components/base"
+
+@import "../../../pillar/src/styles/components/jumbotron"
+@import "../../../pillar/src/styles/components/alerts"
+@import "../../../pillar/src/styles/components/navbar"
+@import "../../../pillar/src/styles/components/dropdown"
+@import "../../../pillar/src/styles/components/footer"
+@import "../../../pillar/src/styles/components/shortcode"
+@import "../../../pillar/src/styles/components/statusbar"
+@import "../../../pillar/src/styles/components/search"
+
+@import "../../../pillar/src/styles/components/flyout"
+@import "../../../pillar/src/styles/components/forms"
+@import "../../../pillar/src/styles/components/inputs"
+@import "../../../pillar/src/styles/components/buttons"
+@import "../../../pillar/src/styles/components/popover"
+@import "../../../pillar/src/styles/components/tooltip"
+@import "../../../pillar/src/styles/components/checkbox"
+@import "../../../pillar/src/styles/components/overlay"
+@import "../../../pillar/src/styles/components/card"
+
+@import "../../../pillar/src/styles/comments"
+@import "../../../pillar/src/styles/notifications"
+
+/* Blender Cloud specific styles */
+@import "../../../pillar/src/styles/_project"
+@import "../../../pillar/src/styles/_project-sharing"
+@import "../../../pillar/src/styles/_project-dashboard"
+@import "../../../pillar/src/styles/_user"
+
+@import _welcome
+@import _homepage
+@import _services
+@import _about
+@import "../../../pillar/src/styles/_search"
+@import "../../../pillar/src/styles/_organizations"
+
+/* services, about, etc */
+@import "../../../pillar/src/styles/_pages"
+
+/* plugins are included here, don't include in base unless needed by other pillar apps */
+@import "../../../pillar/src/styles/plugins/_jstree"
+@import "../../../pillar/src/styles/plugins/_js_select2"
+
+/* CSS for pillar-font comes from fontello.com using static/assets/font/config.json */