Allow Phabricator storage engines to be extended and configured

Summary:
See T344. Currently, there's a hard-coded 12MB filesize limit and some awkward
interactions with MySQL's max_allowed_packet. Make this system generally more
robust:

  - Move the upload limit to configuration.
  - Add setup steps which reconcile max_allowed_packet vs MySQL file storage
limits.
  - Add a layer of indirection between uploading files and storage engines.
  - Allow the definition of new storage engines.
  - Define a local disk storage engine.
  - Add a "storage engine selector" class which manages choosing which storage
engines to put files in.
  - Document storage engines.
  - Document file storage classes.

Test Plan:
Setup mode:

  - Disabled MySQL storage engine, misconfigured it, configured it correctly.
  - Disabled file storage engine, set it to something invalid, set it to
something valid.
  - Verified max_allowed_packet is read correctly.

Application mode:

  - Configured local file storage.
  - Uploaded large and small files.
  - Verified larger files were written to local storage.
  - Verified smaller files were written to MySQL blob storage.

Documentation:

  - Read documentation.

Reviewed By: jungejason
Reviewers: jungejason, tuomaspelkonen, aran
CC: aran, epriestley, jungejason
Differential Revision: 695

src/applications/files/engine/localdisk/PhabricatorLocalDiskFileStorageEngine.php

@@ -0,0 +1,138 @@
+<?php
+
+/*
+ * Copyright 2011 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Local disk storage engine. Keeps files on local disk. This engine is easy
+ * to set up, but it doesn't work if you have multiple web frontends!
+ *
+ * @task impl     Implementation
+ * @task internal Internals
+ * @group filestorage
+ */
+final class PhabricatorLocalDiskFileStorageEngine
+  extends PhabricatorFileStorageEngine {
+
+
+/* -(  Implementation  )----------------------------------------------------- */
+
+
+  /**
+   * This engine identifies as "local-disk".
+   * @task impl
+   */
+  public function getEngineIdentifier() {
+    return 'local-disk';
+  }
+
+
+  /**
+   * Write the file data to local disk. Returns the relative path as the
+   * file data handle.
+   * @task impl
+   */
+  public function writeFile($data, array $params) {
+
+    $root = $this->getLocalDiskFileStorageRoot();
+
+    // Generate a random, unique file path like "ab/29/1f918a9ac39201ff". We
+    // put a couple of subdirectories up front to avoid a situation where we
+    // have one directory with a zillion files in it, since this is generally
+    // bad news.
+    do {
+      $name = md5(mt_rand());
+      $name = preg_replace('/^(..)(..)(.*)$/', '\\1/\\2/\\3', $name);
+      if (!Filesystem::pathExists($root.'/'.$name)) {
+        break;
+      }
+    } while (true);
+
+    $parent = $root.'/'.dirname($name);
+    if (!Filesystem::pathExists($parent)) {
+      execx('mkdir -p %s', $parent);
+    }
+
+    Filesystem::writeFile($root.'/'.$name, $data);
+
+    return $name;
+  }
+
+
+  /**
+   * Read the file data off local disk.
+   * @task impl
+   */
+  public function readFile($handle) {
+    $path = $this->getLocalDiskFileStorageFullPath($handle);
+    return Filesystem::readFile($path);
+  }
+
+
+  /**
+   * Deletes the file from local disk, if it exists.
+   * @task impl
+   */
+  public function deleteFile($handle) {
+    $path = $this->getLocalDiskFileStorageFullPath($handle);
+    if (Filesystem::pathExists($path)) {
+      Filesystem::remove($path);
+    }
+  }
+
+
+/* -(  Internals  )---------------------------------------------------------- */
+
+
+  /**
+   * Get the configured local disk path for file storage.
+   *
+   * @return string Absolute path to somewhere that files can be stored.
+   * @task internal
+   */
+  private function getLocalDiskFileStorageRoot() {
+    $root = PhabricatorEnv::getEnvConfig('storage.local-disk.path');
+
+    if (!$root || $root == '/' || $root[0] != '/') {
+      throw new Exception(
+        "Malformed local disk storage root. You must provide an absolute ".
+        "path, and can not use '/' as the root.");
+    }
+
+    return rtrim($root, '/');
+  }
+
+
+  /**
+   * Convert a handle into an absolute local disk path.
+   *
+   * @param string File data handle.
+   * @return string Absolute path to the corresponding file.
+   * @task internal
+   */
+  private function getLocalDiskFileStorageFullPath($handle) {
+    // Make sure there's no funny business going on here. Users normally have
+    // no ability to affect the content of handles, but double-check that
+    // we're only accessing local storage just in case.
+    if (!preg_match('@^[a-f0-9]{2}/[a-f0-9]{2}/[a-f0-9]{28}$@', $handle)) {
+      throw new Exception(
+        "Local disk filesystem handle '{$handle}' is malformed!");
+    }
+    $root = $this->getLocalDiskFileStorageRoot();
+    return $root.'/'.$handle;
+  }
+
+}