archive/phabricator
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

Make SQL patch management DAG-based and provide namespace support

Browse Source 
Summary:
This addresses three issues with the current patch management system:

  # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
  # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
  # There's no way to build mock databases for unit tests that need to do reads.

To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.

Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).

The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.

A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.

Test Plan:
  - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
  - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
  - Upgraded a legacy database to the new storage format.
  - Destroyed / dumped storage.

Reviewers: edward, vrana, btrahan, jungejason

Reviewed By: btrahan

CC: aran, nh

Maniphest Tasks: T140, T345

Differential Revision: https://secure.phabricator.com/D2323
This commit is contained in:
epriestley
2012-04-30 07:54:00 -07:00
parent 68b597ff75
commit 087cc0808a
37 changed files with 2168 additions and 354 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
scripts/sql/manage_storage.php Executable file
View File

@@ -0,0 +1,123 @@
#!/usr/bin/env php
<?php
/*
* Copyright 2012 Facebook, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
$root = dirname(dirname(dirname(__FILE__)));
require_once $root.'/scripts/__init_script__.php';
$args = new PhutilArgumentParser($argv);
$args->setTagline('manage Phabricator storage and schemata');
$args->setSynopsis(<<<EOHELP
**storage** __workflow__ [__options__]
Manage Phabricator database storage and schema versioning.
**storage** upgrade
Initialize or upgrade Phabricator storage.
**storage** upgrade --user __root__ --password __hunter2__
Use administrative credentials for schema changes.
EOHELP
);
$args->parseStandardArguments();
$conf = PhabricatorEnv::newObjectFromConfig('mysql.configuration-provider');
$default_user = $conf->getUser();
$default_password = $conf->getPassword();
$default_host = $conf->getHost();
$default_namespace = 'phabricator';
try {
$args->parsePartial(
array(
array(
'name' => 'force',
'short' => 'f',
'help' => 'Do not prompt before performing dangerous operations.',
),
array(
'name' => 'user',
'short' => 'u',
'param' => 'username',
'default' => $default_user,
'help' => "Connect with __username__ instead of the configured ".
"default ('{$default_user}').",
),
array(
'name' => 'password',
'short' => 'p',
'param' => 'password',
'default' => $default_password,
'help' => 'Use __password__ instead of the configured default.',
),
array(
'name' => 'namespace',
'param' => 'name',
'default' => $default_namespace,
'help' => "Use namespace __namespace__ instead of the configured ".
"default ('{$default_namespace}'). This is an advanced ".
"feature used by unit tests; you should not normally ".
"use this flag.",
),
array(
'name' => 'dryrun',
'help' => 'Do not actually change anything, just show what would be '.
'changed.',
),
));
} catch (PhutilArgumentUsageException $ex) {
$args->printUsageException($ex);
exit(77);
}
$api = new PhabricatorStorageManagementAPI();
$api->setUser($args->getArg('user'));
$api->setHost($default_host);
$api->setPassword($args->getArg('password'));
$api->setNamespace($args->getArg('namespace'));
try {
queryfx(
$api->getConn('meta_data', $select_database = false),
'SELECT 1');
} catch (AphrontQueryException $ex) {
echo phutil_console_format(
"**%s**: %s\n",
'Unable To Connect',
$ex->getMessage());
exit(1);
}
$workflows = array(
new PhabricatorStorageManagementDatabasesWorkflow(),
new PhabricatorStorageManagementDestroyWorkflow(),
new PhabricatorStorageManagementDumpWorkflow(),
new PhabricatorStorageManagementStatusWorkflow(),
new PhabricatorStorageManagementUpgradeWorkflow(),
);
$patches = PhabricatorSQLPatchList::buildAllPatches();
foreach ($workflows as $workflow) {
$workflow->setAPI($api);
$workflow->setPatches($patches);
}
$workflows[] = new PhutilHelpArgumentWorkflow();
$args->parseWorkflows($workflows);

208
scripts/sql/upgrade_schema.php
View File

@@ -17,204 +17,10 @@
* limitations under the License.
*/
$root = dirname(dirname(dirname(__FILE__)));
require_once $root.'/scripts/__init_script__.php';
phutil_require_module('phutil', 'console');
phutil_require_module('phabricator', 'infrastructure/setup/sql');
define('SCHEMA_VERSION_TABLE_NAME', 'schema_version');
// TODO: getopt() is super terrible, move to something less terrible.
$options = getopt('fhdv:u:p:m:') + array(
'v' => null, // Upgrade from specific version
'u' => null, // Override MySQL User
'p' => null, // Override MySQL Pass
'm' => null, // Specify max version to upgrade to
);
foreach (array('h', 'f', 'd') as $key) {
// By default, these keys are set to 'false' to indicate that the flag was
// passed.
if (array_key_exists($key, $options)) {
$options[$key] = true;
}
}
if (!empty($options['h']) || ($options['v'] && !is_numeric($options['v']))
|| ($options['m'] && !is_numeric($options['m']))) {
usage();
}
if (empty($options['f']) && empty($options['d'])) {
echo phutil_console_wrap(
"Before running this script, you should take down the Phabricator web ".
"interface and stop any running Phabricator daemons.");
if (!phutil_console_confirm('Are you ready to continue?')) {
echo "Cancelled.\n";
exit(1);
}
}
// Use always the version from the commandline if it is defined
$next_version = isset($options['v']) ? (int)$options['v'] : null;
$max_version = isset($options['m']) ? (int)$options['m'] : null;
$conf = PhabricatorEnv::newObjectFromConfig('mysql.configuration-provider');
if ($options['u']) {
$conn_user = $options['u'];
$conn_pass = $options['p'];
} else {
$conn_user = $conf->getUser();
$conn_pass = $conf->getPassword();
}
$conn_host = $conf->getHost();
// Split out port information, since the command-line client requires a
// separate flag for the port.
$uri = new PhutilURI('mysql://'.$conn_host);
if ($uri->getPort()) {
$conn_port = $uri->getPort();
$conn_bare_hostname = $uri->getDomain();
} else {
$conn_port = null;
$conn_bare_hostname = $conn_host;
}
$conn = PhabricatorEnv::newObjectFromConfig(
'mysql.implementation',
array(
array(
'user' => $conn_user,
'pass' => $conn_pass,
'host' => $conn_host,
'database' => null,
),
));
try {
$create_sql = <<<END
CREATE DATABASE IF NOT EXISTS `phabricator_meta_data`;
END;
queryfx($conn, $create_sql);
$create_sql = <<<END
CREATE TABLE IF NOT EXISTS phabricator_meta_data.`schema_version` (
`version` INTEGER not null
);
END;
queryfx($conn, $create_sql);
// Get the version only if commandline argument wasn't given
if ($next_version === null) {
$version = queryfx_one(
$conn,
'SELECT * FROM phabricator_meta_data.%T',
SCHEMA_VERSION_TABLE_NAME);
if (!$version) {
print "*** No version information in the database ***\n";
print "*** Give the first patch version which to ***\n";
print "*** apply as the command line argument ***\n";
exit(-1);
}
$next_version = $version['version'] + 1;
}
$patches = PhabricatorSQLPatchList::getPatchList();
$patch_applied = false;
foreach ($patches as $patch) {
if ($patch['version'] < $next_version) {
continue;
}
if ($max_version && $patch['version'] > $max_version) {
continue;
}
$short_name = basename($patch['path']);
print "Applying patch {$short_name}...\n";
if (!empty($options['d'])) {
$patch_applied = true;
continue;
}
if ($conn_port) {
$port = '--port='.(int)$conn_port;
} else {
$port = null;
}
if (preg_match('/\.php$/', $patch['path'])) {
$schema_conn = $conn;
require_once $patch['path'];
} else {
list($stdout, $stderr) = execx(
"mysql --user=%s --password=%s --host=%s {$port} ".
"--default-character-set=utf8 < %s",
$conn_user,
$conn_pass,
$conn_bare_hostname,
$patch['path']);
if ($stderr) {
print $stderr;
exit(-1);
}
}
// Patch was successful, update the db with the latest applied patch version
// 'DELETE' and 'INSERT' instead of update, because the table might be empty
queryfx(
$conn,
'DELETE FROM phabricator_meta_data.%T',
SCHEMA_VERSION_TABLE_NAME);
queryfx(
$conn,
'INSERT INTO phabricator_meta_data.%T VALUES (%d)',
SCHEMA_VERSION_TABLE_NAME,
$patch['version']);
$patch_applied = true;
}
if (!$patch_applied) {
print "Your database is already up-to-date.\n";
}
} catch (AphrontQueryAccessDeniedException $ex) {
echo
"ACCESS DENIED\n".
"The user '{$conn_user}' does not have sufficient MySQL privileges to\n".
"execute the schema upgrade. Use the -u and -p flags to run as a user\n".
"with more privileges (e.g., root).".
"\n\n".
"EXCEPTION:\n".
$ex->getMessage().
"\n\n";
exit(1);
}
function usage() {
echo
"usage: upgrade_schema.php [-v version] [-u user -p pass] [-f] [-h]".
"\n\n".
"Run 'upgrade_schema.php -u root -p hunter2' to override the configured ".
"default user.\n".
"Run 'upgrade_schema.php -v 12' to apply all patches starting from ".
"version 12. It is very unlikely you need to do this.\n".
"Run 'upgrade_schema.php -m 110' to apply all patches up to and ".
"including version 110 (but nothing past).\n".
"Use the -f flag to upgrade noninteractively, without prompting.\n".
"Use the -d flag to do a dry run - patches that would be applied ".
"will be listed, but not applied.\n".
"Use the -h flag to show this help.\n";
exit(1);
}
echo "This script has been replaced by 'phabricator/bin/storage'.\n\n".
"Run:\n\n".
" phabricator/bin $ ./storage help\n\n".
"...for help, or:\n\n".
" phabricator/bin $ ./storage upgrade\n\n".
"...to upgrade storage.\n\n";
exit(1);