flamenco/pkg/shaman/README.md

Shaman

Shaman is a file storage server. It accepts uploaded files via HTTP, and stores them based on their SHA256-sum and their file length. It can recreate directory structures by symlinking those files. Shaman is intended to complement Blender Asset Tracer (BAT) and Flamenco, but can be used as a standalone component.

The overall use looks like this:

• User creates a set of files (generally via BAT-packing).
• User creates a Checkout Definition File (CDF), consisting of the SHA256-sums, file sizes, and file paths.
• User sends the CDF to Shaman for inspection.
• Shaman replies which files still need uploading.
• User sends those files.
• User sends the CDF to Shaman and requests a checkout with a certain ID.
• Shaman creates the checkout by symlinking the files listed in the CDF.
• Shaman responds with the directory the checkout was created in.

After this process, the checkout directory contains symlinks to all the files in the Checkout Definition File. The user only had to upload new and changed files.

File Store Structure

The Shaman file store is structured as follows:

shaman-store/
    .. uploading/
        .. /{checksum[0:2]}/{checksum[2:]}/{filesize}-{unique-suffix}.tmp
    .. stored/
        .. /{checksum[0:2]}/{checksum[2:]}/{filesize}.blob

When a file is uploaded, it goes through several stages:

• Uploading: the file is being streamed over HTTP and in the process of being stored to disk. The {checksum} and {filesize} fields are as given by the user. While the file is being streamed to disk the SHA256 hash is calculated. After upload is complete the user-provided checksum and file size are compared to the SHA256 hash and actual size. If these differ, the file is rejected.
• Stored: after uploading is complete, the file is stored in the stored directory. Here the {checksum} and {filesize} fields can be assumed to be correct.

Garbage Collection

To prevent infinite growth of the File Store, the Shaman will periodically perform a garbage collection sweep. Garbage Collection can be configured by setting the following settings in shaman.yaml:

• garbageCollect.period: this is the sleep time between garbage collector sweeps. Default is 8h. Set to 0 to disable garbage collection.
• garbageCollect.maxAge: files that are newer than this age are not considered for garbage collection. Default is 744h or 31 days.
• garbageCollect.extraCheckoutPaths: list of directories to include when searching for symlinks. Shaman will never create a checkout here. Default is empty.

Every time a file is symlinked into a checkout directory, it is 'touched' (that is, its modification time is set to 'now').

Files that are not referenced in any checkout, and that have a modification time that is older than garbageCollectMaxAge will be deleted.

To perform a dry run of the garbage collector, use shaman -gc.

Key file generation

SHAman uses JWT with ES256 signatures. The public keys of the JWT-signing authority need to be known, and stored in jwtkeys/*-public*.pem. For more info, see jwtkeys/README.md

Source code structure

• Makefile: Used for building Shaman, testing, etc.
• main.go: The main entry point of the Shaman server. Handles CLI arguments, setting up logging, starting & stopping the server.
• auth: JWT token handling, authentication wrappers for HTTP handlers.
• checkout: Creates (and deletes) checkouts of files by creating directories and symlinking to the file storage.
• config: Configuration file handling.
• fileserver: Stores uploaded files in the file store, and serves files from it.
• filestore: Stores files by SHA256-sum and file size. Has separate storage bins for currently-uploading files and fully-stored files.
• hasher: Computes SHA256 sums.
• httpserver: The HTTP server itself (other packages just contain request handlers, and not the actual server).
• libshaman: Combines the other modules into one Shaman server struct. This allows main.go to start the Shaman server, and makes it possible in the future to embed a Shaman server into another Go project. _py_client: An example client in Python. Just hacked together as a proof of concept and by no means of any official status.

Non-source directories

• jwtkeys: Public keys + a private key for JWT sigining. For now Shaman can create its own dummy JWT keys, but in the future this will become optional or be removed altogether.
• static: For serving static files for the web interface.
• views: Contains HTML files for the web interface. This probably will be merged with static at some point.