Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
<?php
|
|
|
|
|
|
|
|
|
|
final class PhabricatorStorageManagementAPI {
|
|
|
|
|
|
|
|
|
|
private $host;
|
|
|
|
|
private $user;
|
2013-08-26 17:14:57 -07:00
|
|
|
private $port;
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
private $password;
|
|
|
|
|
private $namespace;
|
2013-01-16 16:39:29 -08:00
|
|
|
private $conns = array();
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
|
|
|
|
|
public function setNamespace($namespace) {
|
|
|
|
|
$this->namespace = $namespace;
|
2012-04-30 11:56:58 -07:00
|
|
|
PhabricatorLiskDAO::pushStorageNamespace($namespace);
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
return $this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function getNamespace() {
|
|
|
|
|
return $this->namespace;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function setUser($user) {
|
|
|
|
|
$this->user = $user;
|
|
|
|
|
return $this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function getUser() {
|
|
|
|
|
return $this->user;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function setPassword($password) {
|
|
|
|
|
$this->password = $password;
|
|
|
|
|
return $this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function getPassword() {
|
|
|
|
|
return $this->password;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function setHost($host) {
|
|
|
|
|
$this->host = $host;
|
|
|
|
|
return $this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function getHost() {
|
|
|
|
|
return $this->host;
|
|
|
|
|
}
|
|
|
|
|
|
2013-07-14 16:02:12 -07:00
|
|
|
public function setPort($port) {
|
|
|
|
|
$this->port = $port;
|
|
|
|
|
return $this;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function getPort() {
|
|
|
|
|
return $this->port;
|
|
|
|
|
}
|
|
|
|
|
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
public function getDatabaseName($fragment) {
|
|
|
|
|
return $this->namespace.'_'.$fragment;
|
|
|
|
|
}
|
|
|
|
|
|
2013-07-19 11:07:29 -07:00
|
|
|
public function getDatabaseList(array $patches, $only_living = false) {
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
assert_instances_of($patches, 'PhabricatorStoragePatch');
|
|
|
|
|
|
|
|
|
|
$list = array();
|
|
|
|
|
|
|
|
|
|
foreach ($patches as $patch) {
|
|
|
|
|
if ($patch->getType() == 'db') {
|
2013-07-19 11:07:29 -07:00
|
|
|
if ($only_living && $patch->isDead()) {
|
|
|
|
|
continue;
|
|
|
|
|
}
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
$list[] = $this->getDatabaseName($patch->getName());
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return $list;
|
|
|
|
|
}
|
|
|
|
|
|
2013-01-16 16:39:29 -08:00
|
|
|
public function getConn($fragment) {
|
|
|
|
|
$database = $this->getDatabaseName($fragment);
|
|
|
|
|
$return = &$this->conns[$this->host][$this->user][$database];
|
|
|
|
|
if (!$return) {
|
|
|
|
|
$return = PhabricatorEnv::newObjectFromConfig(
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
'mysql.implementation',
|
|
|
|
|
array(
|
|
|
|
|
array(
|
|
|
|
|
'user' => $this->user,
|
|
|
|
|
'pass' => $this->password,
|
|
|
|
|
'host' => $this->host,
|
2013-07-14 16:02:12 -07:00
|
|
|
'port' => $this->port,
|
2013-01-16 16:39:29 -08:00
|
|
|
'database' => $fragment
|
|
|
|
|
? $database
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
: null,
|
|
|
|
|
),
|
|
|
|
|
));
|
2013-01-16 16:39:29 -08:00
|
|
|
}
|
|
|
|
|
return $return;
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function getAppliedPatches() {
|
|
|
|
|
try {
|
|
|
|
|
$applied = queryfx_all(
|
|
|
|
|
$this->getConn('meta_data'),
|
|
|
|
|
'SELECT patch FROM patch_status');
|
|
|
|
|
return ipull($applied, 'patch');
|
|
|
|
|
} catch (AphrontQueryException $ex) {
|
|
|
|
|
return null;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function createDatabase($fragment) {
|
2014-10-03 06:01:21 -07:00
|
|
|
$info = $this->getCharsetInfo();
|
|
|
|
|
list($charset, $collate_text, $collate_sort) = $info;
|
|
|
|
|
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
queryfx(
|
2013-01-16 16:39:29 -08:00
|
|
|
$this->getConn(null),
|
2014-10-06 13:03:23 -07:00
|
|
|
'CREATE DATABASE IF NOT EXISTS %T COLLATE %T',
|
2014-10-03 06:01:21 -07:00
|
|
|
$this->getDatabaseName($fragment),
|
|
|
|
|
$collate_text);
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function createTable($fragment, $table, array $cols) {
|
|
|
|
|
queryfx(
|
|
|
|
|
$this->getConn($fragment),
|
|
|
|
|
'CREATE TABLE IF NOT EXISTS %T.%T (%Q) '.
|
|
|
|
|
'ENGINE=InnoDB, COLLATE utf8_general_ci',
|
|
|
|
|
$this->getDatabaseName($fragment),
|
|
|
|
|
$table,
|
|
|
|
|
implode(', ', $cols));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function getLegacyPatches(array $patches) {
|
|
|
|
|
assert_instances_of($patches, 'PhabricatorStoragePatch');
|
|
|
|
|
|
|
|
|
|
try {
|
|
|
|
|
$row = queryfx_one(
|
|
|
|
|
$this->getConn('meta_data'),
|
|
|
|
|
'SELECT version FROM %T',
|
|
|
|
|
'schema_version');
|
|
|
|
|
$version = $row['version'];
|
|
|
|
|
} catch (AphrontQueryException $ex) {
|
|
|
|
|
return array();
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
$legacy = array();
|
|
|
|
|
foreach ($patches as $key => $patch) {
|
|
|
|
|
if ($patch->getLegacy() !== false && $patch->getLegacy() <= $version) {
|
|
|
|
|
$legacy[] = $key;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return $legacy;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function markPatchApplied($patch) {
|
|
|
|
|
queryfx(
|
|
|
|
|
$this->getConn('meta_data'),
|
|
|
|
|
'INSERT INTO %T (patch, applied) VALUES (%s, %d)',
|
|
|
|
|
'patch_status',
|
|
|
|
|
$patch,
|
|
|
|
|
time());
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function applyPatch(PhabricatorStoragePatch $patch) {
|
|
|
|
|
$type = $patch->getType();
|
|
|
|
|
$name = $patch->getName();
|
|
|
|
|
switch ($type) {
|
|
|
|
|
case 'db':
|
|
|
|
|
$this->createDatabase($name);
|
|
|
|
|
break;
|
|
|
|
|
case 'sql':
|
|
|
|
|
$this->applyPatchSQL($name);
|
|
|
|
|
break;
|
|
|
|
|
case 'php':
|
|
|
|
|
$this->applyPatchPHP($name);
|
|
|
|
|
break;
|
|
|
|
|
default:
|
|
|
|
|
throw new Exception("Unable to apply patch of type '{$type}'.");
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function applyPatchSQL($sql) {
|
|
|
|
|
$sql = Filesystem::readFile($sql);
|
|
|
|
|
$queries = preg_split('/;\s+/', $sql);
|
|
|
|
|
$queries = array_filter($queries);
|
|
|
|
|
|
2013-01-16 16:39:29 -08:00
|
|
|
$conn = $this->getConn(null);
|
2012-04-30 12:10:33 -07:00
|
|
|
|
2014-10-01 08:22:37 -07:00
|
|
|
$charset_info = $this->getCharsetInfo();
|
|
|
|
|
list($charset, $collate_text, $collate_sort) = $charset_info;
|
|
|
|
|
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
foreach ($queries as $query) {
|
|
|
|
|
$query = str_replace('{$NAMESPACE}', $this->namespace, $query);
|
2014-10-01 08:22:37 -07:00
|
|
|
$query = str_replace('{$CHARSET}', $charset, $query);
|
2014-10-06 15:51:42 -07:00
|
|
|
$escaped_text = qsprintf($conn, '%T', $collate_text);
|
|
|
|
|
$query = str_replace('{$COLLATE_TEXT}', $escaped_text, $query);
|
|
|
|
|
$escaped_text = qsprintf($conn, '%T', $collate_sort);
|
|
|
|
|
$query = str_replace('{$COLLATE_SORT}', $escaped_text, $query);
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
queryfx(
|
2012-04-30 12:10:33 -07:00
|
|
|
$conn,
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
'%Q',
|
|
|
|
|
$query);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
public function applyPatchPHP($script) {
|
2013-01-16 16:39:29 -08:00
|
|
|
$schema_conn = $this->getConn(null);
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
require_once $script;
|
|
|
|
|
}
|
|
|
|
|
|
2014-09-18 08:22:18 -07:00
|
|
|
public function isCharacterSetAvailable($character_set) {
|
|
|
|
|
$conn = $this->getConn(null);
|
|
|
|
|
|
|
|
|
|
$result = queryfx_one(
|
|
|
|
|
$conn,
|
|
|
|
|
'SELECT CHARACTER_SET_NAME FROM INFORMATION_SCHEMA.CHARACTER_SETS
|
|
|
|
|
WHERE CHARACTER_SET_NAME = %s',
|
|
|
|
|
$character_set);
|
|
|
|
|
|
|
|
|
|
return (bool)$result;
|
|
|
|
|
}
|
|
|
|
|
|
2014-10-01 08:22:37 -07:00
|
|
|
public function getCharsetInfo() {
|
|
|
|
|
if ($this->isCharacterSetAvailable('utf8mb4')) {
|
|
|
|
|
// If utf8mb4 is available, we use it with the utf8mb4_unicode_ci
|
|
|
|
|
// collation. This is most correct, and will sort properly.
|
|
|
|
|
|
|
|
|
|
$charset = 'utf8mb4';
|
|
|
|
|
$collate_text = 'utf8mb4_bin';
|
|
|
|
|
$collate_sort = 'utf8mb4_unicode_ci';
|
|
|
|
|
} else {
|
|
|
|
|
// If utf8mb4 is not available, we use binary. This allows us to store
|
|
|
|
|
// 4-byte unicode characters. This has some tradeoffs:
|
|
|
|
|
//
|
|
|
|
|
// Unicode characters won't sort correctly. There's nothing we can do
|
|
|
|
|
// about this while still supporting 4-byte characters.
|
|
|
|
|
//
|
|
|
|
|
// It's possible that strings will be truncated in the middle of a
|
|
|
|
|
// character on insert. We encourage users to set STRICT_ALL_TABLES
|
|
|
|
|
// to prevent this.
|
|
|
|
|
|
|
|
|
|
$charset = 'binary';
|
|
|
|
|
$collate_text = 'binary';
|
|
|
|
|
$collate_sort = 'binary';
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return array($charset, $collate_text, $collate_sort);
|
|
|
|
|
}
|
|
|
|
|
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
|
|
|
}
|