2011-01-16 22:51:39 +01:00
|
|
|
<?php
|
|
|
|
|
|
|
|
/*
|
Add an assocations-like "Edges" framework
Summary:
We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems:
- We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant.
- Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks.
- I want to add more of these and don't want to continue building custom stuff.
- UIs like the "attach stuff to other stuff" UI need custom branches for each object type.
- Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage.
Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences:
- I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook.
- I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type.
This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904.
Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, 20after4
Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
|
|
|
* Copyright 2012 Facebook, Inc.
|
2011-01-16 22:51:39 +01:00
|
|
|
*
|
|
|
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
* you may not use this file except in compliance with the License.
|
|
|
|
* You may obtain a copy of the License at
|
|
|
|
*
|
|
|
|
* http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
*
|
|
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
* See the License for the specific language governing permissions and
|
|
|
|
* limitations under the License.
|
|
|
|
*/
|
|
|
|
|
Add an assocations-like "Edges" framework
Summary:
We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems:
- We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant.
- Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks.
- I want to add more of these and don't want to continue building custom stuff.
- UIs like the "attach stuff to other stuff" UI need custom branches for each object type.
- Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage.
Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences:
- I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook.
- I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type.
This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904.
Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, 20after4
Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
|
|
|
/**
|
|
|
|
* @task edges Managing Edges
|
|
|
|
* @task config Configuring Storage
|
|
|
|
*/
|
2011-01-23 02:48:55 +01:00
|
|
|
abstract class PhabricatorLiskDAO extends LiskDAO {
|
2011-01-16 22:51:39 +01:00
|
|
|
|
Add an assocations-like "Edges" framework
Summary:
We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems:
- We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant.
- Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks.
- I want to add more of these and don't want to continue building custom stuff.
- UIs like the "attach stuff to other stuff" UI need custom branches for each object type.
- Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage.
Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences:
- I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook.
- I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type.
This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904.
Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, 20after4
Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
|
|
|
private $edges = array();
|
2012-04-30 20:56:58 +02:00
|
|
|
private static $namespaceStack = array();
|
Add an assocations-like "Edges" framework
Summary:
We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems:
- We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant.
- Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks.
- I want to add more of these and don't want to continue building custom stuff.
- UIs like the "attach stuff to other stuff" UI need custom branches for each object type.
- Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage.
Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences:
- I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook.
- I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type.
This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904.
Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, 20after4
Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
|
|
|
|
|
|
|
|
|
|
|
/* -( Managing Edges )----------------------------------------------------- */
|
|
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @task edges
|
|
|
|
*/
|
|
|
|
public function attachEdges(array $edges) {
|
|
|
|
foreach ($edges as $type => $type_edges) {
|
|
|
|
$this->edges[$type] = $type_edges;
|
|
|
|
}
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @task edges
|
|
|
|
*/
|
|
|
|
public function getEdges($type) {
|
|
|
|
$edges = idx($this->edges, $type);
|
|
|
|
if ($edges === null) {
|
|
|
|
throw new Exception("Call attachEdges() before getEdges()!");
|
|
|
|
}
|
|
|
|
return $edges;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @task edges
|
|
|
|
*/
|
|
|
|
public function getEdgePHIDs($type) {
|
|
|
|
return ipull($this->getEdges($type), 'dst');
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/* -( Configuring Storage )------------------------------------------------ */
|
|
|
|
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
/**
|
|
|
|
* @task config
|
|
|
|
*/
|
2012-04-30 20:56:58 +02:00
|
|
|
public static function pushStorageNamespace($namespace) {
|
|
|
|
self::$namespaceStack[] = $namespace;
|
Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:
# Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
# Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
# There's no way to build mock databases for unit tests that need to do reads.
To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.
Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).
The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.
A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.
Test Plan:
- Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
- Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
- Upgraded a legacy database to the new storage format.
- Destroyed / dumped storage.
Reviewers: edward, vrana, btrahan, jungejason
Reviewed By: btrahan
CC: aran, nh
Maniphest Tasks: T140, T345
Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
|
|
|
}
|
|
|
|
|
2012-04-30 20:56:58 +02:00
|
|
|
/**
|
|
|
|
* @task config
|
|
|
|
*/
|
2012-05-02 21:42:23 +02:00
|
|
|
public static function popStorageNamespace() {
|
2012-04-30 20:56:58 +02:00
|
|
|
array_pop(self::$namespaceStack);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @task config
|
|
|
|
*/
|
|
|
|
public static function getDefaultStorageNamespace() {
|
|
|
|
return PhabricatorEnv::getEnvConfig('storage.default-namespace');
|
|
|
|
}
|
Add an assocations-like "Edges" framework
Summary:
We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems:
- We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant.
- Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks.
- I want to add more of these and don't want to continue building custom stuff.
- UIs like the "attach stuff to other stuff" UI need custom branches for each object type.
- Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage.
Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences:
- I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook.
- I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type.
This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904.
Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, 20after4
Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
|
|
|
|
|
|
|
/**
|
|
|
|
* @task config
|
|
|
|
*/
|
2012-04-30 21:14:37 +02:00
|
|
|
public static function getStorageNamespace() {
|
2012-04-30 20:56:58 +02:00
|
|
|
$namespace = end(self::$namespaceStack);
|
|
|
|
if (!strlen($namespace)) {
|
|
|
|
$namespace = self::getDefaultStorageNamespace();
|
|
|
|
}
|
|
|
|
if (!strlen($namespace)) {
|
|
|
|
throw new Exception("No storage namespace configured!");
|
|
|
|
}
|
2012-04-30 21:14:37 +02:00
|
|
|
return $namespace;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @task config
|
|
|
|
*/
|
|
|
|
public function establishLiveConnection($mode) {
|
|
|
|
$namespace = self::getStorageNamespace();
|
2012-04-30 20:56:58 +02:00
|
|
|
|
2012-04-06 23:42:02 +02:00
|
|
|
$conf = PhabricatorEnv::newObjectFromConfig(
|
|
|
|
'mysql.configuration-provider',
|
2012-04-30 20:56:58 +02:00
|
|
|
array($this, $mode, $namespace));
|
2011-04-30 02:23:25 +02:00
|
|
|
|
2012-04-07 06:29:19 +02:00
|
|
|
return PhabricatorEnv::newObjectFromConfig(
|
|
|
|
'mysql.implementation',
|
2011-01-16 22:51:39 +01:00
|
|
|
array(
|
2012-04-07 06:29:19 +02:00
|
|
|
array(
|
|
|
|
'user' => $conf->getUser(),
|
|
|
|
'pass' => $conf->getPassword(),
|
|
|
|
'host' => $conf->getHost(),
|
|
|
|
'database' => $conf->getDatabase(),
|
|
|
|
),
|
2011-01-16 22:51:39 +01:00
|
|
|
));
|
|
|
|
}
|
|
|
|
|
Add an assocations-like "Edges" framework
Summary:
We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems:
- We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant.
- Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks.
- I want to add more of these and don't want to continue building custom stuff.
- UIs like the "attach stuff to other stuff" UI need custom branches for each object type.
- Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage.
Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences:
- I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook.
- I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type.
This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904.
Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, 20after4
Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
|
|
|
/**
|
|
|
|
* @task config
|
|
|
|
*/
|
2011-01-16 22:51:39 +01:00
|
|
|
public function getTableName() {
|
2011-01-23 02:48:55 +01:00
|
|
|
$str = 'phabricator';
|
|
|
|
$len = strlen($str);
|
|
|
|
|
2011-01-16 22:51:39 +01:00
|
|
|
$class = strtolower(get_class($this));
|
2011-01-23 02:48:55 +01:00
|
|
|
if (!strncmp($class, $str, $len)) {
|
|
|
|
$class = substr($class, $len);
|
2011-01-16 22:51:39 +01:00
|
|
|
}
|
|
|
|
$app = $this->getApplicationName();
|
|
|
|
if (!strncmp($class, $app, strlen($app))) {
|
|
|
|
$class = substr($class, strlen($app));
|
|
|
|
}
|
2011-01-23 06:09:13 +01:00
|
|
|
|
|
|
|
if (strlen($class)) {
|
|
|
|
return $app.'_'.$class;
|
|
|
|
} else {
|
|
|
|
return $app;
|
|
|
|
}
|
2011-01-16 22:51:39 +01:00
|
|
|
}
|
|
|
|
|
Add an assocations-like "Edges" framework
Summary:
We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems:
- We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant.
- Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks.
- I want to add more of these and don't want to continue building custom stuff.
- UIs like the "attach stuff to other stuff" UI need custom branches for each object type.
- Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage.
Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences:
- I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook.
- I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type.
This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904.
Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran, 20after4
Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
|
|
|
/**
|
|
|
|
* @task config
|
|
|
|
*/
|
2011-01-16 22:51:39 +01:00
|
|
|
abstract public function getApplicationName();
|
2012-04-30 20:57:10 +02:00
|
|
|
|
|
|
|
protected function getConnectionNamespace() {
|
2012-04-30 21:14:37 +02:00
|
|
|
return self::getStorageNamespace().'_'.$this->getApplicationName();
|
2012-04-30 20:57:10 +02:00
|
|
|
}
|
2011-01-16 22:51:39 +01:00
|
|
|
}
|