e2296a0ff7
Summary:
Fixes T5843. File storage engines use a very old "selector" mechanism which makes them difficult to extend.
This mechanism predates widespread use of `PhutilSymbolLoader` to discover available implementations at runtime. Runtime discovery has generally proven more flexible and easier to use than explicit selection (although it sometimes needs more UI to support it in cases where order or enabled/disabled flags can not be directly determined).
Use a modern runtime discovery mechanism instead of an explicit selector. This might break any installs which subclassed the `Selector`, but I believe almost no such installs exist, and they'll receive a meaningful exception upon upgrading (any custom engines will no longer implement all of the required methods).
Looking forward, this modernizes infrastructure to prepare for new "virtual" chunked-storage engines, with the eventual goal of supporting very large file uploads and data import into the Phacility cluster.
This uses D12051 to add UI to make it easier to understand the state of storage engines.
Test Plan:
Used new UI panel to assess storage engines:
{F336270}
- Uploaded a small file, saw it go to MySQL engine.
- Uploaded a larger file, saw it go to S3 engine.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5843
Differential Revision: https://secure.phabricator.com/D12053
105 lines
3.3 KiB
Plaintext
105 lines
3.3 KiB
Plaintext
@title Configuring File Storage
|
|
@group config
|
|
|
|
Setup how Phabricator will store files.
|
|
|
|
Overview
|
|
========
|
|
|
|
Phabricator allows users to upload files, and several applications use file
|
|
storage (for instance, Maniphest allows you to attach files to tasks). You can
|
|
configure several different storage systems.
|
|
|
|
| System | Setup | Cost | Notes |
|
|
|========|=======|======|=======|
|
|
| MySQL | Automatic | Free | May not scale well. |
|
|
| Local Disk | Easy | Free | Does not scale well. |
|
|
| Amazon S3 | Easy | Cheap | Scales well. |
|
|
| Custom | Hard | Varies | Implement a custom storage engine. |
|
|
|
|
By default, Phabricator is configured to store files up to 1MB in MySQL, and
|
|
reject files larger than 1MB. To store larger files, you can either:
|
|
|
|
- configure local disk storage; or
|
|
- configure Amazon S3 storage; or
|
|
- raise the limits on MySQL.
|
|
|
|
See the rest of this document for some additional discussion of engines.
|
|
|
|
You don't have to fully configure this immediately, the defaults are okay until
|
|
you need to upload larger files and it's relatively easy to port files between
|
|
storage engines later.
|
|
|
|
Storage Engines
|
|
===============
|
|
|
|
Builtin storage engines and information on how to configure them.
|
|
|
|
== MySQL ==
|
|
|
|
- **Pros**: Fast, no setup required.
|
|
- **Cons**: Storing files in a database is a classic bad idea. Does not scale
|
|
well. Maximum file size is limited.
|
|
|
|
MySQL storage is configured by default, for files up to (just under) 1MB. You
|
|
can configure it with these keys:
|
|
|
|
- `storage.mysql-engine.max-size`: Change the filesize limit. Set to 0
|
|
to disable.
|
|
|
|
For most installs, it is reasonable to leave this engine as-is and let small
|
|
files (like thumbnails and profile images) be stored in MySQL, which is usually
|
|
the lowest-latency filestore.
|
|
|
|
To support larger files, configure another engine or increase this limit.
|
|
|
|
== Local Disk ==
|
|
|
|
- **Pros**: Very simple. Almost no setup required.
|
|
- **Cons**: Doesn't scale to multiple web frontends without NFS.
|
|
|
|
To upload larger files:
|
|
|
|
- `storage.local-disk.path`: Set to some writable directory on local disk.
|
|
Make that directory.
|
|
|
|
== Amazon S3 ==
|
|
|
|
- **Pros**: Scales well.
|
|
- **Cons**: More complicated and expensive than other approaches.
|
|
|
|
To enable file storage in S3, set these key:
|
|
|
|
- ##amazon-s3.access-key## Your AWS access key.
|
|
- ##amazon-s3.secret-key## Your AWS secret key.
|
|
- ##storage.s3.bucket## S3 bucket name where files should be stored.
|
|
|
|
= Testing Storage Engines =
|
|
|
|
You can test that things are correctly configured by going to the Files
|
|
application (##/file/##) and uploading files.
|
|
|
|
= Migrating Files Between Engines =
|
|
|
|
If you want to move files between storage engines, you can use the `bin/files`
|
|
script to perform migrations. For example, suppose you previously used MySQL but
|
|
recently set up S3 and want to migrate all your files there. First, migrate one
|
|
file to make sure things work:
|
|
|
|
phabricator/ $ ./bin/files migrate --engine amazon-s3 F12345
|
|
|
|
If that works properly, you can then migrate everything:
|
|
|
|
phabricator/ $ ./bin/files migrate --engine amazon-s3 --all
|
|
|
|
You can use `--dry-run` to show which migrations would be performed without
|
|
taking any action. Run `bin/files help` for more options and information.
|
|
|
|
= Next Steps =
|
|
|
|
Continue by:
|
|
|
|
- configuring file size upload limits with
|
|
@{article:Configuring File Upload Limits}; or
|
|
- returning to the @{article:Configuration Guide}.
|