Symbolic links on Windows require some special user privilege, and Shaman can now check for this at startup. Hopefully this helps in guiding people towards a working Shaman system. |
||
---|---|---|
.. | ||
_test_file_store | ||
checkout | ||
config | ||
fileserver | ||
filestore | ||
hasher | ||
jwtauth | ||
testsupport | ||
touch | ||
cleanup_test.go | ||
cleanup.go | ||
humanize.go | ||
README.md | ||
server.go | ||
TODO.md |
Shaman
Shaman is a file storage server. It accepts uploaded files via HTTP, and stores them based on their SHA256-sum and their file length. It can recreate directory structures by symlinking those files. Shaman is intended to complement Blender Asset Tracer (BAT) and Flamenco, but can be used as a standalone component.
The overall use looks like this:
- User creates a set of files (generally via BAT-packing).
- User creates a Checkout Definition File (CDF), consisting of the SHA256-sums, file sizes, and file paths.
- User sends the CDF to Shaman for inspection.
- Shaman replies which files still need uploading.
- User sends those files.
- User sends the CDF to Shaman and requests a checkout with a certain ID.
- Shaman creates the checkout by symlinking the files listed in the CDF.
- Shaman responds with the directory the checkout was created in.
After this process, the checkout directory contains symlinks to all the files in the Checkout Definition File. The user only had to upload new and changed files.
File Store Structure
The Shaman file store is structured as follows:
shaman-store/
.. uploading/
.. /{checksum[0:2]}/{checksum[2:]}/{filesize}-{unique-suffix}.tmp
.. stored/
.. /{checksum[0:2]}/{checksum[2:]}/{filesize}.blob
When a file is uploaded, it goes through several stages:
- Uploading: the file is being streamed over HTTP and in the process of
being stored to disk. The
{checksum}
and{filesize}
fields are as given by the user. While the file is being streamed to disk the SHA256 hash is calculated. After upload is complete the user-provided checksum and file size are compared to the SHA256 hash and actual size. If these differ, the file is rejected. - Stored: after uploading is complete, the file is stored in the
stored
directory. Here the{checksum}
and{filesize}
fields can be assumed to be correct.
Garbage Collection
To prevent infinite growth of the File Store, the Shaman will periodically
perform a garbage collection sweep. Garbage Collection can be configured by
setting the following settings in shaman.yaml
:
garbageCollect.period
: this is the sleep time between garbage collector sweeps. Default is8h
. Set to0
to disable garbage collection.garbageCollect.maxAge
: files that are newer than this age are not considered for garbage collection. Default is744h
or 31 days.garbageCollect.extraCheckoutPaths
: list of directories to include when searching for symlinks. Shaman will never create a checkout here. Default is empty.
Every time a file is symlinked into a checkout directory, it is 'touched' (that is, its modification time is set to 'now').
Files that are not referenced in any checkout, and that have a modification
time that is older than garbageCollectMaxAge
will be deleted.
To perform a dry run of the garbage collector, use shaman -gc
.
Key file generation
SHAman uses JWT with ES256
signatures. The public keys of the JWT-signing
authority need to be known, and stored in jwtkeys/*-public*.pem
.
For more info, see jwtkeys/README.md
Source code structure
Makefile
: Used for building Shaman, testing, etc.main.go
: The main entry point of the Shaman server. Handles CLI arguments, setting up logging, starting & stopping the server.auth
: JWT token handling, authentication wrappers for HTTP handlers.checkout
: Creates (and deletes) checkouts of files by creating directories and symlinking to the file storage.config
: Configuration file handling.fileserver
: Stores uploaded files in the file store, and serves files from it.filestore
: Stores files by SHA256-sum and file size. Has separate storage bins for currently-uploading files and fully-stored files.hasher
: Computes SHA256 sums.httpserver
: The HTTP server itself (other packages just contain request handlers, and not the actual server).libshaman
: Combines the other modules into one Shaman server struct. This allowsmain.go
to start the Shaman server, and makes it possible in the future to embed a Shaman server into another Go project._py_client
: An example client in Python. Just hacked together as a proof of concept and by no means of any official status.
Non-source directories
jwtkeys
: Public keys + a private key for JWT sigining. For now Shaman can create its own dummy JWT keys, but in the future this will become optional or be removed altogether.static
: For serving static files for the web interface.views
: Contains HTML files for the web interface. This probably will be merged withstatic
at some point.