blender-studio/docs/development.md

Development

Requirements

• git
• Python 3.10
• virtualenv
• PostgreSQL (tested with PostgreSQL 14)

Recommended for local development:

• virtualenvwrapper

virtualenvwrapper adds commands for naming, creating, listing, activating and deactivating Python virtualenvs, which makes it much easier to work on various Python projects.

If you prefer not to use it, any other way of creating a virtualenv or venv will do.

Set up instructions

1. Clone the repo: git clone git@projects.blender.org:studio/blender-studio.git
2. Install project dependencies
   • mkvirtualenv -a pwd blender-studio -p /usr/bin/python3.10
   • pip install 'poetry==1.4.2'
   • poetry install If the installation of psycopg2 fails, make sure that you have the required apt packages installed (more details).
3. Create a PostgreSQL user named studio: sudo -u postgres createuser -d -l -P studio
4. Create a database named studio: sudo -u postgres createdb -O studio studio
5. Add studio.local to /etc/hosts as an alias of 127.0.0.1:

   127.0.0.1    localhost studio.local  # studio.local can be added on the same line as localhost
    ...
   

6. Create a .env file (cp .env.example .env). This file is gitignored, and it must not be committed.
7. Fill in AWS S3 and CloudFront credentials in your .env:
   • Set values of AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY values to your access keys;
   • Download the CloudFront key file and save it to the project directory (it should be named pk-APK***.pem);
   • Set AWS_CLOUDFRONT_KEY_ID='APK***' where APK*** is from the name of the key file above.
8. In the command line, activate the virtual environment created by poetry: poetry shell
   • Configure your IDE to use the venv by default.
9. In the project folder, run migrations: ./manage.py migrate
10. Create a superuser: echo "from django.contrib.auth import get_user_model; User = get_user_model(); User.objects.create_superuser('admin', 'admin@example.com', 'password')" | python manage.py shell
11. Run the server: ./manage.py runserver 8001. The project will be available at studio.local:8001.
12. (Optional) Install pre-commit hooks (see pre-commit details): pre-commit install
13. In the admin panel (http://studio.local:8001/admin), edit the Site object's domain. The default domain is example.com; change it to studio.local:8001. This will make it possible to immediately view objects created/edited via admin on site.
14. Set up the Blender ID server for authentication and MeiliSerach server for the search functionality.
15. Setup for video processing jobs. Download ngrok (https://ngrok.com/).
    • Run ./ngrok http 8010
    • Update .env:
      • Set COCONUT_API_KEY to a valid value
      • Set COCONUT_DECLARED_HOSTNAME to https://<random-value>.ngrok.io

Data import

You can add objects to the database manually via the Django's Admin panel. There are also commands that import data from production, but running them requires some additional arrangements - ask Francesco about it.

Troubleshooting

If the loaddata command hangs, disable the post-save signals: in the search/app.py file comment out the ready() function definition (specifically, the import statement in its body).

The staging database has different content type ids than the databases created recently (after the 'static_assets' app was renamed). This may cause problems with loading fixtures. The most probable case is that some users have permissions assigned, and these permissions relate to nonexistent content types.

To fix this, open the fixture_users.json file, find the occurrences of the string "user_permissions": [ (including the inverted commas). If any user has any numbers listed there, delete all those numbers, and leave an empty array, like this: user_permissions": [].

You can then manually reassign user permissions in the admin panel.

Blender ID authentication and webhook

Login to Blender Studio is possible using Blender ID. For development, you can set up a local instance of Blender ID.

In your local Blender ID virtualenv (workon blender-id, if using virtualenvwrapper), load the following fixture:

./manage.py loaddata blender_studio_devserver

This creates an OAuth2 application with BID_OAUTH_CLIENT, BID_OAUTH_SECRET and BID_WEBHOOK_USER_MODIFIED_SECRET values already hardcoded in .env.example.

N.B.: the webhook view delegates the actual updating of the user profile to a background task, so in order to see the updates locally, start the processing of tasks using the following:

./manage.py process_tasks

Search setup

For a complete description of the search feature, see the search documentation. It also includes production setup and troubleshooting instructions.

1. Download meilisearch binary of version 0.25.2 from its release page, save it in the project's directory, make it executable and launch it:

   chmod +x meilisearch
   ./meilisearch
   

2. With the project's venv activated, create indexes and fill them with data:

   ./manage.py create_search_indexes
   ./manage.py index_documents
   

The server will be available on port 7700 by default. If you change it, adjust the MEILISEARCH_API_ADDRESS in .env as necessary.

Workflow

Before committing

The following assumes that the virtual environment is activated: poetry shell.

Pre-commit hooks are responsible for automatically running black, mypy, etc. on the staged files, before each commit. If there are issues, committing is aborted.

The pre-commit configuration is in the .pre-commit-config.yaml file.

In case of emergency, it is possible to disable one or more hooks. To completely disable all the hooks, run pre-commit uninstall. To enable them again, run pre-commit install.

You can also execute the test.sh script: it runs mypy, black, tests, eslint, and stylelint on the entire project (so it's slower and more likely to error out).

Git workflow

1. Rebase, don't merge.
2. main is the working branch. In order to work on a task, create a new branch off main. It is technically possible to git push --force to main, however please consider at least warning other developers if you plan to do it.

Deployments

Studio doesn't use Docker, only a clone of its own repository and a few systemd units. To deploy latest production, use the following script:

./deploy.sh studio.blender.org

This will

• pull latest main and production;
• fast-forward production to main;
• SSH into studio.blender.org and there:
  • pull latest production;
  • do poetry install;
  • run database migrations;
  • do collectstatic;
  • restart studio-background.service and studio-background.service.

Setting up timers for periodic tasks

Production Studio uses systemd timers intead of crontab. The following periodic services exist at the moment:

• studio-clearsessions.timer: calls clearsessions;
• studio-process-deletion-requests.timer: processes outstanding deletion requests;
• studio-background-restart.timer: takes care of a heisenbug that causes background process to hang on rare occasion.

In order to set them up in production, the following commands were used:

ssh root@studio.blender.org
cd /var/www/blender-studio/
cp systemd/system/*.{service,timer} /lib/systemd/system/
systemctl enable studio-process-deletion-requests.timer
systemctl enable studio-clearsessions.timer
systemctl enable studio-background-restart.timer
systemctl start studio-process-deletion-requests.timer
systemctl start studio-clearsessions.timer
systemctl start studio-background-restart.timer

To view existing timers and details about when they were called last and other usefull info:

systemctl list-timers --all

All the units and timers can be found in systemd/system/ of this repository.