Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
#!/usr/bin/env php
|
|
|
|
<?php
|
|
|
|
|
|
|
|
$root = dirname(dirname(dirname(__FILE__)));
|
2016-09-06 22:20:31 +02:00
|
|
|
require_once $root.'/scripts/init/init-setup.php';
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
|
|
|
|
$args = new PhutilArgumentParser($argv);
|
2022-04-25 19:14:43 +02:00
|
|
|
$args->setTagline(pht('manage storage and schemata'));
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
$args->setSynopsis(<<<EOHELP
|
|
|
|
**storage** __workflow__ [__options__]
|
2022-04-25 19:14:43 +02:00
|
|
|
Manage database storage and schema versioning.
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
|
|
|
|
**storage** upgrade
|
2022-04-25 19:14:43 +02:00
|
|
|
Initialize or upgrade storage.
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
|
|
|
|
**storage** upgrade --user __root__ --password __hunter2__
|
|
|
|
Use administrative credentials for schema changes.
|
|
|
|
EOHELP
|
|
|
|
);
|
|
|
|
$args->parseStandardArguments();
|
|
|
|
|
2012-04-30 20:56:58 +02:00
|
|
|
$default_namespace = PhabricatorLiskDAO::getDefaultStorageNamespace();
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
|
|
|
|
try {
|
|
|
|
$args->parsePartial(
|
|
|
|
array(
|
|
|
|
array(
|
|
|
|
'name' => 'force',
|
|
|
|
'short' => 'f',
|
2015-05-22 09:27:56 +02:00
|
|
|
'help' => pht(
|
|
|
|
'Do not prompt before performing dangerous operations.'),
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
),
|
2016-04-14 20:34:16 +02:00
|
|
|
array(
|
|
|
|
'name' => 'host',
|
|
|
|
'param' => 'hostname',
|
|
|
|
'help' => pht(
|
2016-11-12 20:39:45 +01:00
|
|
|
'Operate on the database server identified by __hostname__.'),
|
|
|
|
),
|
|
|
|
array(
|
|
|
|
'name' => 'ref',
|
|
|
|
'param' => 'ref',
|
|
|
|
'help' => pht(
|
|
|
|
'Operate on the database identified by __ref__.'),
|
2016-04-14 20:34:16 +02:00
|
|
|
),
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
array(
|
|
|
|
'name' => 'user',
|
|
|
|
'short' => 'u',
|
|
|
|
'param' => 'username',
|
2015-05-22 09:27:56 +02:00
|
|
|
'help' => pht(
|
When `cluster.databases` is configured, read the master connection from it
Summary:
Ref T4571. Ref T10759. Ref T10758. This isn't complete, but gets most of the job done:
- When `cluster.databases` is set up, most things ignore `mysql.host` now.
- You can `bin/storage upgrade` and stuff works.
- You can browse around in the web UI and stuff works.
There's still a lot of weird tricky stuff to navigate, and this has real no advantages over configuring a single server yet (no automatic failover, etc).
Test Plan:
- Configured `cluster.databases` to point at my `t1.micro` hosts in EC2 (master + replica).
- Ran `bin/storage upgrade`, got a new install setup on them properly.
- Survived setup warnings, browsed around.
- Switched back to local config, ran `bin/storage upgrade`, browsed around, went through setup checks.
- Intentionally broke config (bad hosts, no masters) and things seemed to react reasonably well.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4571, T10758, T10759
Differential Revision: https://secure.phabricator.com/D15668
2016-04-10 04:46:42 +02:00
|
|
|
'Connect with __username__ instead of the configured default.'),
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
),
|
|
|
|
array(
|
|
|
|
'name' => 'password',
|
|
|
|
'short' => 'p',
|
|
|
|
'param' => 'password',
|
2015-05-22 09:27:56 +02:00
|
|
|
'help' => pht('Use __password__ instead of the configured default.'),
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
),
|
|
|
|
array(
|
|
|
|
'name' => 'namespace',
|
|
|
|
'param' => 'name',
|
|
|
|
'default' => $default_namespace,
|
2015-05-22 09:27:56 +02:00
|
|
|
'help' => pht(
|
|
|
|
"Use namespace __namespace__ instead of the configured ".
|
|
|
|
"default ('%s'). This is an advanced feature used by unit tests; ".
|
|
|
|
"you should not normally use this flag.",
|
|
|
|
$default_namespace),
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
),
|
|
|
|
array(
|
2015-11-30 21:58:14 +01:00
|
|
|
'name' => 'dryrun',
|
|
|
|
'help' => pht(
|
2015-05-22 09:27:56 +02:00
|
|
|
'Do not actually change anything, just show what would be changed.'),
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
),
|
2014-10-29 23:49:29 +01:00
|
|
|
array(
|
2015-11-30 21:58:14 +01:00
|
|
|
'name' => 'disable-utf8mb4',
|
|
|
|
'help' => pht(
|
|
|
|
'Disable %s, even if the database supports it. This is an '.
|
2022-04-25 19:14:43 +02:00
|
|
|
'advanced feature used for testing internal changes; you '.
|
2015-11-30 21:58:14 +01:00
|
|
|
'should not normally use this flag.',
|
|
|
|
'utf8mb4'),
|
2014-12-08 20:08:05 +01:00
|
|
|
),
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
));
|
|
|
|
} catch (PhutilArgumentUsageException $ex) {
|
|
|
|
$args->printUsageException($ex);
|
|
|
|
exit(77);
|
|
|
|
}
|
|
|
|
|
2014-08-09 00:44:40 +02:00
|
|
|
// First, test that the Phabricator configuration is set up correctly. After
|
|
|
|
// we know this works we'll test any administrative credentials specifically.
|
|
|
|
|
2016-11-12 20:39:45 +01:00
|
|
|
$refs = PhabricatorDatabaseRef::getActiveDatabaseRefs();
|
|
|
|
if (!$refs) {
|
|
|
|
throw new PhutilArgumentUsageException(
|
|
|
|
pht('No databases are configured.'));
|
|
|
|
}
|
2016-04-14 20:34:16 +02:00
|
|
|
|
2016-11-12 20:39:45 +01:00
|
|
|
$host = $args->getArg('host');
|
|
|
|
$ref_key = $args->getArg('ref');
|
2021-12-16 23:07:59 +01:00
|
|
|
if (($host !== null) || ($ref_key !== null)) {
|
2016-11-12 20:39:45 +01:00
|
|
|
if ($host && $ref_key) {
|
|
|
|
throw new PhutilArgumentUsageException(
|
|
|
|
pht(
|
|
|
|
'Use "--host" or "--ref" to select a database, but not both.'));
|
2016-06-14 15:56:12 +02:00
|
|
|
}
|
2016-04-14 20:34:16 +02:00
|
|
|
|
2016-11-12 20:39:45 +01:00
|
|
|
$refs = PhabricatorDatabaseRef::getActiveDatabaseRefs();
|
|
|
|
|
|
|
|
$possible_refs = array();
|
2016-04-14 20:34:16 +02:00
|
|
|
foreach ($refs as $possible_ref) {
|
2016-11-12 20:39:45 +01:00
|
|
|
if ($host && ($possible_ref->getHost() == $host)) {
|
|
|
|
$possible_refs[] = $possible_ref;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if ($ref_key && ($possible_ref->getRefKey() == $ref_key)) {
|
|
|
|
$possible_refs[] = $possible_ref;
|
2016-04-14 20:34:16 +02:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-11-12 20:39:45 +01:00
|
|
|
if (!$possible_refs) {
|
|
|
|
if ($host) {
|
|
|
|
throw new PhutilArgumentUsageException(
|
|
|
|
pht(
|
|
|
|
'There is no configured database on host "%s". This command can '.
|
|
|
|
'only interact with configured databases.',
|
|
|
|
$host));
|
|
|
|
} else {
|
|
|
|
throw new PhutilArgumentUsageException(
|
|
|
|
pht(
|
|
|
|
'There is no configured database with ref "%s". This command can '.
|
|
|
|
'only interact with configured databases.',
|
|
|
|
$ref_key));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (count($possible_refs) > 1) {
|
2016-04-14 20:34:16 +02:00
|
|
|
throw new PhutilArgumentUsageException(
|
|
|
|
pht(
|
2016-11-12 20:39:45 +01:00
|
|
|
'Host "%s" identifies more than one database. Use "--ref" to select '.
|
|
|
|
'a specific database.',
|
2016-04-14 20:34:16 +02:00
|
|
|
$host));
|
|
|
|
}
|
When `cluster.databases` is configured, read the master connection from it
Summary:
Ref T4571. Ref T10759. Ref T10758. This isn't complete, but gets most of the job done:
- When `cluster.databases` is set up, most things ignore `mysql.host` now.
- You can `bin/storage upgrade` and stuff works.
- You can browse around in the web UI and stuff works.
There's still a lot of weird tricky stuff to navigate, and this has real no advantages over configuring a single server yet (no automatic failover, etc).
Test Plan:
- Configured `cluster.databases` to point at my `t1.micro` hosts in EC2 (master + replica).
- Ran `bin/storage upgrade`, got a new install setup on them properly.
- Survived setup warnings, browsed around.
- Switched back to local config, ran `bin/storage upgrade`, browsed around, went through setup checks.
- Intentionally broke config (bad hosts, no masters) and things seemed to react reasonably well.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4571, T10758, T10759
Differential Revision: https://secure.phabricator.com/D15668
2016-04-10 04:46:42 +02:00
|
|
|
|
2016-11-12 20:39:45 +01:00
|
|
|
$refs = $possible_refs;
|
2014-08-09 00:44:40 +02:00
|
|
|
}
|
|
|
|
|
2016-11-12 20:39:45 +01:00
|
|
|
$apis = array();
|
|
|
|
foreach ($refs as $ref) {
|
|
|
|
$default_user = $ref->getUser();
|
|
|
|
$default_host = $ref->getHost();
|
|
|
|
$default_port = $ref->getPort();
|
|
|
|
|
|
|
|
$test_api = id(new PhabricatorStorageManagementAPI())
|
|
|
|
->setUser($default_user)
|
|
|
|
->setHost($default_host)
|
|
|
|
->setPort($default_port)
|
|
|
|
->setPassword($ref->getPass())
|
|
|
|
->setNamespace($args->getArg('namespace'));
|
|
|
|
|
|
|
|
try {
|
|
|
|
queryfx(
|
|
|
|
$test_api->getConn(null),
|
|
|
|
'SELECT 1');
|
|
|
|
} catch (AphrontQueryException $ex) {
|
|
|
|
$message = phutil_console_format(
|
|
|
|
"**%s**\n\n%s\n\n%s\n\n%s\n\n**%s**: %s\n",
|
|
|
|
pht('MySQL Credentials Not Configured'),
|
|
|
|
pht(
|
|
|
|
'Unable to connect to MySQL using the configured credentials. '.
|
|
|
|
'You must configure standard credentials before you can upgrade '.
|
|
|
|
'storage. Run these commands to set up credentials:'),
|
2022-04-25 19:14:43 +02:00
|
|
|
" $ ./bin/config set mysql.host __host__\n".
|
|
|
|
" $ ./bin/config set mysql.user __username__\n".
|
|
|
|
" $ ./bin/config set mysql.pass __password__",
|
2016-11-12 20:39:45 +01:00
|
|
|
pht(
|
|
|
|
'These standard credentials are separate from any administrative '.
|
|
|
|
'credentials provided to this command with __%s__ or '.
|
|
|
|
'__%s__, and must be configured correctly before you can proceed.',
|
|
|
|
'--user',
|
|
|
|
'--password'),
|
|
|
|
pht('Raw MySQL Error'),
|
|
|
|
$ex->getMessage());
|
|
|
|
echo phutil_console_wrap($message);
|
|
|
|
exit(1);
|
|
|
|
}
|
2012-07-24 20:13:53 +02:00
|
|
|
|
2016-11-12 20:39:45 +01:00
|
|
|
if ($args->getArg('password') === null) {
|
|
|
|
// This is already a PhutilOpaqueEnvelope.
|
|
|
|
$password = $ref->getPass();
|
|
|
|
} else {
|
|
|
|
// Put this in a PhutilOpaqueEnvelope.
|
|
|
|
$password = new PhutilOpaqueEnvelope($args->getArg('password'));
|
|
|
|
PhabricatorEnv::overrideConfig('mysql.pass', $args->getArg('password'));
|
|
|
|
}
|
When `cluster.databases` is configured, read the master connection from it
Summary:
Ref T4571. Ref T10759. Ref T10758. This isn't complete, but gets most of the job done:
- When `cluster.databases` is set up, most things ignore `mysql.host` now.
- You can `bin/storage upgrade` and stuff works.
- You can browse around in the web UI and stuff works.
There's still a lot of weird tricky stuff to navigate, and this has real no advantages over configuring a single server yet (no automatic failover, etc).
Test Plan:
- Configured `cluster.databases` to point at my `t1.micro` hosts in EC2 (master + replica).
- Ran `bin/storage upgrade`, got a new install setup on them properly.
- Survived setup warnings, browsed around.
- Switched back to local config, ran `bin/storage upgrade`, browsed around, went through setup checks.
- Intentionally broke config (bad hosts, no masters) and things seemed to react reasonably well.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4571, T10758, T10759
Differential Revision: https://secure.phabricator.com/D15668
2016-04-10 04:46:42 +02:00
|
|
|
|
2016-11-12 20:39:45 +01:00
|
|
|
$selected_user = $args->getArg('user');
|
|
|
|
if ($selected_user === null) {
|
|
|
|
$selected_user = $default_user;
|
|
|
|
}
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
|
2016-11-12 20:39:45 +01:00
|
|
|
$api = id(new PhabricatorStorageManagementAPI())
|
|
|
|
->setUser($selected_user)
|
|
|
|
->setHost($default_host)
|
|
|
|
->setPort($default_port)
|
|
|
|
->setPassword($password)
|
|
|
|
->setNamespace($args->getArg('namespace'))
|
|
|
|
->setDisableUTF8MB4($args->getArg('disable-utf8mb4'));
|
|
|
|
PhabricatorEnv::overrideConfig('mysql.user', $api->getUser());
|
|
|
|
|
2016-11-19 17:35:29 +01:00
|
|
|
$ref->setUser($selected_user);
|
|
|
|
$ref->setPass($password);
|
|
|
|
|
2016-11-12 20:39:45 +01:00
|
|
|
try {
|
|
|
|
queryfx(
|
|
|
|
$api->getConn(null),
|
|
|
|
'SELECT 1');
|
|
|
|
} catch (AphrontQueryException $ex) {
|
|
|
|
$message = phutil_console_format(
|
|
|
|
"**%s**\n\n%s\n\n**%s**: %s\n",
|
|
|
|
pht('Bad Administrative Credentials'),
|
|
|
|
pht(
|
|
|
|
'Unable to connect to MySQL using the administrative credentials '.
|
|
|
|
'provided with the __%s__ and __%s__ flags. Check that '.
|
|
|
|
'you have entered them correctly.',
|
|
|
|
'--user',
|
|
|
|
'--password'),
|
|
|
|
pht('Raw MySQL Error'),
|
|
|
|
$ex->getMessage());
|
|
|
|
echo phutil_console_wrap($message);
|
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
|
|
|
|
$api->setRef($ref);
|
|
|
|
$apis[] = $api;
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
}
|
|
|
|
|
2015-08-13 23:49:00 +02:00
|
|
|
$workflows = id(new PhutilClassMapQuery())
|
2013-12-27 22:15:48 +01:00
|
|
|
->setAncestorClass('PhabricatorStorageManagementWorkflow')
|
2015-08-13 23:49:00 +02:00
|
|
|
->execute();
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
|
|
|
|
$patches = PhabricatorSQLPatchList::buildAllPatches();
|
|
|
|
|
|
|
|
foreach ($workflows as $workflow) {
|
2016-11-12 20:39:45 +01:00
|
|
|
$workflow->setAPIs($apis);
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
$workflow->setPatches($patches);
|
|
|
|
}
|
|
|
|
|
|
|
|
$workflows[] = new PhutilHelpArgumentWorkflow();
|
|
|
|
|
|
|
|
$args->parseWorkflows($workflows);
|