1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2025-01-09 14:21:02 +01:00
phorge-phorge/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php

210 lines
4.6 KiB
PHP
Raw Normal View History

<?php
Add an assocations-like "Edges" framework Summary: We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems: - We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant. - Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks. - I want to add more of these and don't want to continue building custom stuff. - UIs like the "attach stuff to other stuff" UI need custom branches for each object type. - Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage. Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences: - I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook. - I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type. This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904. Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation. Reviewers: btrahan Reviewed By: btrahan CC: aran, 20after4 Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
/**
* @task edges Managing Edges
* @task config Configuring Storage
*/
2011-01-23 02:48:55 +01:00
abstract class PhabricatorLiskDAO extends LiskDAO {
Add an assocations-like "Edges" framework Summary: We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems: - We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant. - Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks. - I want to add more of these and don't want to continue building custom stuff. - UIs like the "attach stuff to other stuff" UI need custom branches for each object type. - Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage. Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences: - I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook. - I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type. This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904. Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation. Reviewers: btrahan Reviewed By: btrahan CC: aran, 20after4 Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
private $edges = array();
private static $namespaceStack = array();
Add an assocations-like "Edges" framework Summary: We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems: - We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant. - Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks. - I want to add more of these and don't want to continue building custom stuff. - UIs like the "attach stuff to other stuff" UI need custom branches for each object type. - Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage. Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences: - I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook. - I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type. This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904. Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation. Reviewers: btrahan Reviewed By: btrahan CC: aran, 20after4 Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
/* -( Managing Edges )----------------------------------------------------- */
/**
* @task edges
*/
public function attachEdges(array $edges) {
foreach ($edges as $type => $type_edges) {
$this->edges[$type] = $type_edges;
}
return $this;
}
/**
* @task edges
*/
public function getEdges($type) {
$edges = idx($this->edges, $type);
if ($edges === null) {
throw new Exception("Call attachEdges() before getEdges()!");
}
return $edges;
}
/**
* @task edges
*/
public function loadRelativeEdges($type) {
if (!$this->getInSet()) {
id(new LiskDAOSet())->addToSet($this);
}
$this->getInSet()->loadRelativeEdges($type);
return $this->getEdges($type);
}
Add an assocations-like "Edges" framework Summary: We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems: - We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant. - Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks. - I want to add more of these and don't want to continue building custom stuff. - UIs like the "attach stuff to other stuff" UI need custom branches for each object type. - Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage. Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences: - I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook. - I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type. This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904. Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation. Reviewers: btrahan Reviewed By: btrahan CC: aran, 20after4 Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
/**
* @task edges
*/
public function getEdgePHIDs($type) {
return ipull($this->getEdges($type), 'dst');
}
/* -( Configuring Storage )------------------------------------------------ */
Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
/**
* @task config
*/
public static function pushStorageNamespace($namespace) {
self::$namespaceStack[] = $namespace;
Make SQL patch management DAG-based and provide namespace support Summary: This addresses three issues with the current patch management system: # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly. # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add. # There's no way to build mock databases for unit tests that need to do reads. To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts. Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably). The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage. A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms. Test Plan: - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good. - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things. - Upgraded a legacy database to the new storage format. - Destroyed / dumped storage. Reviewers: edward, vrana, btrahan, jungejason Reviewed By: btrahan CC: aran, nh Maniphest Tasks: T140, T345 Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 16:54:00 +02:00
}
/**
* @task config
*/
public static function popStorageNamespace() {
array_pop(self::$namespaceStack);
}
/**
* @task config
*/
public static function getDefaultStorageNamespace() {
return PhabricatorEnv::getEnvConfig('storage.default-namespace');
}
Add an assocations-like "Edges" framework Summary: We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems: - We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant. - Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks. - I want to add more of these and don't want to continue building custom stuff. - UIs like the "attach stuff to other stuff" UI need custom branches for each object type. - Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage. Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences: - I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook. - I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type. This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904. Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation. Reviewers: btrahan Reviewed By: btrahan CC: aran, 20after4 Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
/**
* @task config
*/
public static function getStorageNamespace() {
$namespace = end(self::$namespaceStack);
if (!strlen($namespace)) {
$namespace = self::getDefaultStorageNamespace();
}
if (!strlen($namespace)) {
throw new Exception("No storage namespace configured!");
}
return $namespace;
}
/**
* @task config
*/
public function establishLiveConnection($mode) {
$namespace = self::getStorageNamespace();
$conf = PhabricatorEnv::newObjectFromConfig(
'mysql.configuration-provider',
array($this, $mode, $namespace));
return PhabricatorEnv::newObjectFromConfig(
'mysql.implementation',
array(
array(
'user' => $conf->getUser(),
'pass' => $conf->getPassword(),
'host' => $conf->getHost(),
'database' => $conf->getDatabase(),
'retries' => 3,
),
));
}
Add an assocations-like "Edges" framework Summary: We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems: - We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant. - Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks. - I want to add more of these and don't want to continue building custom stuff. - UIs like the "attach stuff to other stuff" UI need custom branches for each object type. - Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage. Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences: - I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook. - I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type. This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904. Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation. Reviewers: btrahan Reviewed By: btrahan CC: aran, 20after4 Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
/**
* @task config
*/
public function getTableName() {
2011-01-23 02:48:55 +01:00
$str = 'phabricator';
$len = strlen($str);
$class = strtolower(get_class($this));
2011-01-23 02:48:55 +01:00
if (!strncmp($class, $str, $len)) {
$class = substr($class, $len);
}
$app = $this->getApplicationName();
if (!strncmp($class, $app, strlen($app))) {
$class = substr($class, strlen($app));
}
2011-01-23 06:09:13 +01:00
if (strlen($class)) {
return $app.'_'.$class;
} else {
return $app;
}
}
Add an assocations-like "Edges" framework Summary: We have a lot of cases where we store object relationships, but it's all kind of messy and custom. Some particular problems: - We go to great lengths to enforce order stability in Differential revisions, but the implementation is complex and inelegant. - Some relationships are stored on-object, so we can't pull the inverses easily. For example, Maniphest shows child tasks but not parent tasks. - I want to add more of these and don't want to continue building custom stuff. - UIs like the "attach stuff to other stuff" UI need custom branches for each object type. - Stuff like "allow commits to close tasks" is notrivial because of nonstandard metadata storage. Provide an association-like "edge" framework to fix these problems. This is nearly identical to associations, with a few differences: - I put edge metadata in a separate table and don't load it by default, to keep edge rows small and allow large metadata if necessary. The on-edge metadata seemed to get abused a lot at Facebook. - I put a 'seq' column on the edges to ensure they have an explicit, stable ordering within a source and type. This isn't actually used anywhere yet, but my first target is attaching commits to tasks for T904. Test Plan: Made a mock page that used Editor and Query. Verified adding and removing edges, overwriting edges, writing and loading edge data, sequence number generation. Reviewers: btrahan Reviewed By: btrahan CC: aran, 20after4 Differential Revision: https://secure.phabricator.com/D2088
2012-04-05 00:30:21 +02:00
/**
* @task config
*/
abstract public function getApplicationName();
protected function getConnectionNamespace() {
return self::getStorageNamespace().'_'.$this->getApplicationName();
}
/**
* Break a list of escaped SQL statement fragments (e.g., VALUES lists for
* INSERT, previously built with @{function:qsprintf}) into chunks which will
* fit under the MySQL 'max_allowed_packet' limit.
*
* Chunks are glued together with `$glue`, by default ", ".
*
* If a statement is too large to fit within the limit, it is broken into
* its own chunk (but might fail when the query executes).
*/
public static function chunkSQL(
array $fragments,
$glue = ', ',
$limit = null) {
if ($limit === null) {
// NOTE: Hard-code this at 1MB for now, minus a 10% safety buffer.
// Eventually we could query MySQL or let the user configure it.
$limit = (int)((1024 * 1024) * 0.90);
}
$result = array();
$chunk = array();
$len = 0;
$glue_len = strlen($glue);
foreach ($fragments as $fragment) {
$this_len = strlen($fragment);
if ($chunk) {
// Chunks after the first also imply glue.
$this_len += $glue_len;
}
if ($len + $this_len <= $limit) {
$len += $this_len;
$chunk[] = $fragment;
} else {
if ($chunk) {
$result[] = $chunk;
}
$len = strlen($fragment);
$chunk = array($fragment);
}
}
if ($chunk) {
$result[] = $chunk;
}
foreach ($result as $key => $fragment_list) {
$result[$key] = implode($glue, $fragment_list);
}
return $result;
}
}