1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-30 01:10:58 +01:00
phorge-phorge/scripts/sql/manage_storage.php

172 lines
5.3 KiB
PHP
Raw Normal View History

Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
#!/usr/bin/env php
<?php
$root = dirname(dirname(dirname(__FILE__)));
require_once $root.'/scripts/__init_script__.php';
$args = new PhutilArgumentParser($argv);
$args->setTagline('manage Phabricator storage and schemata');
$args->setSynopsis(<<<EOHELP
**storage** __workflow__ [__options__]
Manage Phabricator database storage and schema versioning.
**storage** upgrade
Initialize or upgrade Phabricator storage.
**storage** upgrade --user __root__ --password __hunter2__
Use administrative credentials for schema changes.
EOHELP
);
$args->parseStandardArguments();
$conf = PhabricatorEnv::newObjectFromConfig(
'mysql.configuration-provider',
array($dao = null, 'w'));
Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
$default_user = $conf->getUser();
$default_host = $conf->getHost();
$default_port = $conf->getPort();
$default_namespace = PhabricatorLiskDAO::getDefaultStorageNamespace();
Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
try {
$args->parsePartial(
array(
array(
'name' => 'force',
'short' => 'f',
'help' => 'Do not prompt before performing dangerous operations.',
),
array(
'name' => 'user',
'short' => 'u',
'param' => 'username',
'default' => $default_user,
'help' => "Connect with __username__ instead of the configured ".
"default ('{$default_user}').",
),
array(
'name' => 'password',
'short' => 'p',
'param' => 'password',
'help' => 'Use __password__ instead of the configured default.',
),
array(
'name' => 'namespace',
'param' => 'name',
'default' => $default_namespace,
'help' => "Use namespace __namespace__ instead of the configured ".
"default ('{$default_namespace}'). This is an advanced ".
"feature used by unit tests; you should not normally ".
"use this flag.",
),
array(
'name' => 'dryrun',
'help' => 'Do not actually change anything, just show what would be '.
'changed.',
),
array(
'name' => 'disable-utf8mb4',
'help' => pht(
'Disable utf8mb4, even if the database supports it. This is an '.
'advanced feature used for testing changes to Phabricator; you '.
'should not normally use this flag.'),
),
Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
));
} catch (PhutilArgumentUsageException $ex) {
$args->printUsageException($ex);
exit(77);
}
// First, test that the Phabricator configuration is set up correctly. After
// we know this works we'll test any administrative credentials specifically.
$test_api = new PhabricatorStorageManagementAPI();
$test_api->setUser($default_user);
$test_api->setHost($default_host);
$test_api->setPort($default_port);
$test_api->setPassword($conf->getPassword());
$test_api->setNamespace($args->getArg('namespace'));
try {
queryfx(
$test_api->getConn(null),
'SELECT 1');
} catch (AphrontQueryException $ex) {
$message = phutil_console_format(
pht(
"**MySQL Credentials Not Configured**\n\n".
"Unable to connect to MySQL using the configured credentials. ".
"You must configure standard credentials before you can upgrade ".
"storage. Run these commands to set up credentials:\n".
"\n".
" phabricator/ $ ./bin/config set mysql.host __host__\n".
" phabricator/ $ ./bin/config set mysql.user __username__\n".
" phabricator/ $ ./bin/config set mysql.pass __password__\n".
"\n".
"These standard credentials are separate from any administrative ".
"credentials provided to this command with __--user__ or ".
"__--password__, and must be configured correctly before you can ".
"proceed.\n".
"\n".
"**Raw MySQL Error**: %s\n",
$ex->getMessage()));
echo phutil_console_wrap($message);
exit(1);
}
if ($args->getArg('password') === null) {
// This is already a PhutilOpaqueEnvelope.
$password = $conf->getPassword();
} else {
// Put this in a PhutilOpaqueEnvelope.
$password = new PhutilOpaqueEnvelope($args->getArg('password'));
PhabricatorEnv::overrideConfig('mysql.pass', $args->getArg('password'));
}
Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
$api = new PhabricatorStorageManagementAPI();
$api->setUser($args->getArg('user'));
PhabricatorEnv::overrideConfig('mysql.user', $args->getArg('user'));
Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
$api->setHost($default_host);
$api->setPort($default_port);
$api->setPassword($password);
Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
$api->setNamespace($args->getArg('namespace'));
$api->setDisableUTF8MB4($args->getArg('disable-utf8mb4'));
Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
try {
queryfx(
$api->getConn(null),
Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
'SELECT 1');
} catch (AphrontQueryException $ex) {
$message = phutil_console_format(
pht(
"**Bad Administrative Credentials**\n\n".
"Unable to connnect to MySQL using the administrative credentials ".
"provided with the __--user__ and __--password__ flags. Check that ".
"you have entered them correctly.\n".
"\n".
"**Raw MySQL Error**: %s\n",
$ex->getMessage()));
echo phutil_console_wrap($message);
Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
exit(1);
}
$workflows = id(new PhutilSymbolLoader())
->setAncestorClass('PhabricatorStorageManagementWorkflow')
->loadObjects();
Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
$patches = PhabricatorSQLPatchList::buildAllPatches();
foreach ($workflows as $workflow) {
$workflow->setAPI($api);
$workflow->setPatches($patches);
}
$workflows[] = new PhutilHelpArgumentWorkflow();
$args->parseWorkflows($workflows);