Migrate Differential comments to ApplicationTransactions
Summary:
Ref T2222. This is the big one.
This migrates each `DifferentialComment` to one or more ApplicationTransactions (action, cc, reviewers, update, comment, inlines), and makes `DifferentialComment` a double-reader for ApplicationTransactions.
The migration is pretty straightforward:
- If a comment took an action not otherwise covered, it gets an "action" transaction. This is something like "epriestley abandoned this revision.".
- If a comment updated the diff, it gets an "updated diff" transaction. Very old transactions of this type may not have a diff ID (probably only at Facebook).
- If a comment added or removed reviewers, it gets a "changed reviewers" transaction.
- If a comment added CCs, it gets a "subscribers" transaction.
- If a comment added comment text, it gets a "comment" transaction.
- For each inline attached to a comment, we generate an "inline" transaction.
Most comments generate a small number of transactions, but a few generate a significant number.
At HEAD, the code is basically already doing this, so comments in the last day or two already obey these rules, roughly, and will all generate only one transaction (except inlines).
Because we've already preallocated PHIDs in the comment text table, we only need to write to the transaction table.
NOTE: This significantly degrades Differential, making inline comments pretty much useless (they each get their own transaction, and don't show line numbers or files). The data is all fine, but the UI is garbage now. This needs to be fixed before we can deploy this to users, but it's easily separable since it's all just display code.
Specifically, they look like this:
{F112270}
Test Plan:
I've migrated locally and put things through their paces, but it's hard to catch sketchy stuff locally because most of my test data is nonsense and bad migrations wouldn't necessarily look out of place.
IMPORTANT: I'm planning to push this to a branch and then shift production over to the branch, and run it for a day or two before bringing it to master.
I generally feel good about this change: it's not that big since we were able to separate a lot of pieces out of it, and it's pretty straightforward. That said, it's still one of the most scary/dangerous changes we've ever made.
Reviewers: btrahan
CC: chad, aran
Maniphest Tasks: T2222
Differential Revision: https://secure.phabricator.com/D8210
2014-02-12 00:36:58 +01:00
|
|
|
<?php
|
|
|
|
|
|
|
|
$conn_w = id(new DifferentialRevision())->establishConnection('w');
|
|
|
|
$rows = new LiskRawMigrationIterator($conn_w, 'differential_comment');
|
|
|
|
|
|
|
|
$content_source = PhabricatorContentSource::newForSource(
|
|
|
|
PhabricatorContentSource::SOURCE_LEGACY,
|
|
|
|
array())->serialize();
|
|
|
|
|
|
|
|
echo "Migrating Differential comments to modern storage...\n";
|
|
|
|
foreach ($rows as $row) {
|
|
|
|
$id = $row['id'];
|
|
|
|
echo "Migrating comment {$id}...\n";
|
|
|
|
|
|
|
|
$revision = id(new DifferentialRevision())->load($row['revisionID']);
|
|
|
|
if (!$revision) {
|
|
|
|
echo "No revision, continuing.\n";
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
$revision_phid = $revision->getPHID();
|
|
|
|
|
|
|
|
$comments = queryfx_all(
|
|
|
|
$conn_w,
|
|
|
|
'SELECT * FROM %T WHERE legacyCommentID = %d',
|
|
|
|
'differential_transaction_comment',
|
|
|
|
$id);
|
|
|
|
|
|
|
|
$main_comments = array();
|
|
|
|
$inline_comments = array();
|
|
|
|
|
|
|
|
foreach ($comments as $comment) {
|
|
|
|
if ($comment['changesetID']) {
|
|
|
|
$inline_comments[] = $comment;
|
|
|
|
} else {
|
|
|
|
$main_comments[] = $comment;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
$metadata = json_decode($row['metadata'], true);
|
|
|
|
if (!is_array($metadata)) {
|
|
|
|
$metadata = array();
|
|
|
|
}
|
|
|
|
|
2014-03-09 22:53:05 +01:00
|
|
|
$key_cc = 'added-ccs';
|
|
|
|
$key_add_rev = 'added-reviewers';
|
|
|
|
$key_rem_rev = 'removed-reviewers';
|
|
|
|
$key_diff_id = 'diff-id';
|
Migrate Differential comments to ApplicationTransactions
Summary:
Ref T2222. This is the big one.
This migrates each `DifferentialComment` to one or more ApplicationTransactions (action, cc, reviewers, update, comment, inlines), and makes `DifferentialComment` a double-reader for ApplicationTransactions.
The migration is pretty straightforward:
- If a comment took an action not otherwise covered, it gets an "action" transaction. This is something like "epriestley abandoned this revision.".
- If a comment updated the diff, it gets an "updated diff" transaction. Very old transactions of this type may not have a diff ID (probably only at Facebook).
- If a comment added or removed reviewers, it gets a "changed reviewers" transaction.
- If a comment added CCs, it gets a "subscribers" transaction.
- If a comment added comment text, it gets a "comment" transaction.
- For each inline attached to a comment, we generate an "inline" transaction.
Most comments generate a small number of transactions, but a few generate a significant number.
At HEAD, the code is basically already doing this, so comments in the last day or two already obey these rules, roughly, and will all generate only one transaction (except inlines).
Because we've already preallocated PHIDs in the comment text table, we only need to write to the transaction table.
NOTE: This significantly degrades Differential, making inline comments pretty much useless (they each get their own transaction, and don't show line numbers or files). The data is all fine, but the UI is garbage now. This needs to be fixed before we can deploy this to users, but it's easily separable since it's all just display code.
Specifically, they look like this:
{F112270}
Test Plan:
I've migrated locally and put things through their paces, but it's hard to catch sketchy stuff locally because most of my test data is nonsense and bad migrations wouldn't necessarily look out of place.
IMPORTANT: I'm planning to push this to a branch and then shift production over to the branch, and run it for a day or two before bringing it to master.
I generally feel good about this change: it's not that big since we were able to separate a lot of pieces out of it, and it's pretty straightforward. That said, it's still one of the most scary/dangerous changes we've ever made.
Reviewers: btrahan
CC: chad, aran
Maniphest Tasks: T2222
Differential Revision: https://secure.phabricator.com/D8210
2014-02-12 00:36:58 +01:00
|
|
|
|
|
|
|
$xactions = array();
|
|
|
|
|
|
|
|
// Build the main action transaction.
|
|
|
|
switch ($row['action']) {
|
|
|
|
case DifferentialAction::ACTION_COMMENT:
|
|
|
|
case DifferentialAction::ACTION_ADDREVIEWERS:
|
|
|
|
case DifferentialAction::ACTION_ADDCCS:
|
|
|
|
case DifferentialAction::ACTION_UPDATE:
|
|
|
|
case DifferentialTransaction::TYPE_INLINE:
|
|
|
|
// These actions will have their transactions created by other rules.
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
// Otherwise, this is a normal action (like an accept or reject).
|
|
|
|
$xactions[] = array(
|
|
|
|
'type' => DifferentialTransaction::TYPE_ACTION,
|
|
|
|
'old' => null,
|
|
|
|
'new' => $row['action'],
|
|
|
|
);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Build the diff update transaction, if one exists.
|
|
|
|
$diff_id = idx($metadata, $key_diff_id);
|
|
|
|
if (!is_scalar($diff_id)) {
|
|
|
|
$diff_id = null;
|
|
|
|
}
|
|
|
|
|
|
|
|
if ($diff_id || $row['action'] == DifferentialAction::ACTION_UPDATE) {
|
|
|
|
$xactions[] = array(
|
|
|
|
'type' => DifferentialTransaction::TYPE_UPDATE,
|
|
|
|
'old' => null,
|
|
|
|
'new' => $diff_id,
|
|
|
|
);
|
|
|
|
}
|
|
|
|
|
|
|
|
// Build the add/remove reviewers transaction, if one exists.
|
|
|
|
$add_rev = idx($metadata, $key_add_rev, array());
|
|
|
|
if (!is_array($add_rev)) {
|
|
|
|
$add_rev = array();
|
|
|
|
}
|
|
|
|
$rem_rev = idx($metadata, $key_rem_rev, array());
|
|
|
|
if (!is_array($rem_rev)) {
|
|
|
|
$rem_rev = array();
|
|
|
|
}
|
|
|
|
|
|
|
|
if ($add_rev || $rem_rev) {
|
|
|
|
$old = array();
|
|
|
|
foreach ($rem_rev as $phid) {
|
|
|
|
if (!is_scalar($phid)) {
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
$old[$phid] = array(
|
|
|
|
'src' => $revision_phid,
|
|
|
|
'type' => PhabricatorEdgeConfig::TYPE_DREV_HAS_REVIEWER,
|
|
|
|
'dst' => $phid,
|
|
|
|
);
|
|
|
|
}
|
|
|
|
|
|
|
|
$new = array();
|
|
|
|
foreach ($add_rev as $phid) {
|
|
|
|
if (!is_scalar($phid)) {
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
$new[$phid] = array(
|
|
|
|
'src' => $revision_phid,
|
|
|
|
'type' => PhabricatorEdgeConfig::TYPE_DREV_HAS_REVIEWER,
|
|
|
|
'dst' => $phid,
|
|
|
|
);
|
|
|
|
}
|
|
|
|
|
|
|
|
$xactions[] = array(
|
|
|
|
'type' => PhabricatorTransactions::TYPE_EDGE,
|
|
|
|
'old' => $old,
|
|
|
|
'new' => $new,
|
|
|
|
'meta' => array(
|
|
|
|
'edge:type' => PhabricatorEdgeConfig::TYPE_DREV_HAS_REVIEWER,
|
|
|
|
),
|
|
|
|
);
|
|
|
|
}
|
|
|
|
|
|
|
|
// Build the CC transaction, if one exists.
|
|
|
|
$add_cc = idx($metadata, $key_cc, array());
|
|
|
|
if (!is_array($add_cc)) {
|
|
|
|
$add_cc = array();
|
|
|
|
}
|
|
|
|
|
|
|
|
if ($add_cc) {
|
|
|
|
$xactions[] = array(
|
|
|
|
'type' => PhabricatorTransactions::TYPE_SUBSCRIBERS,
|
|
|
|
'old' => array(),
|
|
|
|
'new' => array_fuse($add_cc),
|
|
|
|
);
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// Build the main comment transaction.
|
|
|
|
foreach ($main_comments as $main) {
|
|
|
|
$xactions[] = array(
|
|
|
|
'type' => PhabricatorTransactions::TYPE_COMMENT,
|
|
|
|
'old' => null,
|
|
|
|
'new' => null,
|
|
|
|
'phid' => $main['transactionPHID'],
|
|
|
|
'comment' => $main,
|
|
|
|
);
|
|
|
|
}
|
|
|
|
|
|
|
|
// Build inline comment transactions.
|
|
|
|
foreach ($inline_comments as $inline) {
|
|
|
|
$xactions[] = array(
|
|
|
|
'type' => DifferentialTransaction::TYPE_INLINE,
|
|
|
|
'old' => null,
|
|
|
|
'new' => null,
|
|
|
|
'phid' => $inline['transactionPHID'],
|
|
|
|
'comment' => $inline,
|
|
|
|
);
|
|
|
|
}
|
|
|
|
|
|
|
|
foreach ($xactions as $xaction) {
|
|
|
|
// Generate a new PHID, if we don't already have one from the comment
|
|
|
|
// table. We pregenerated into the comment table to make this a little
|
|
|
|
// easier, so we only need to write to one table.
|
|
|
|
$xaction_phid = idx($xaction, 'phid');
|
|
|
|
if (!$xaction_phid) {
|
|
|
|
$xaction_phid = PhabricatorPHID::generateNewPHID(
|
2014-07-24 00:05:46 +02:00
|
|
|
PhabricatorApplicationTransactionTransactionPHIDType::TYPECONST,
|
|
|
|
DifferentialRevisionPHIDType::TYPECONST);
|
Migrate Differential comments to ApplicationTransactions
Summary:
Ref T2222. This is the big one.
This migrates each `DifferentialComment` to one or more ApplicationTransactions (action, cc, reviewers, update, comment, inlines), and makes `DifferentialComment` a double-reader for ApplicationTransactions.
The migration is pretty straightforward:
- If a comment took an action not otherwise covered, it gets an "action" transaction. This is something like "epriestley abandoned this revision.".
- If a comment updated the diff, it gets an "updated diff" transaction. Very old transactions of this type may not have a diff ID (probably only at Facebook).
- If a comment added or removed reviewers, it gets a "changed reviewers" transaction.
- If a comment added CCs, it gets a "subscribers" transaction.
- If a comment added comment text, it gets a "comment" transaction.
- For each inline attached to a comment, we generate an "inline" transaction.
Most comments generate a small number of transactions, but a few generate a significant number.
At HEAD, the code is basically already doing this, so comments in the last day or two already obey these rules, roughly, and will all generate only one transaction (except inlines).
Because we've already preallocated PHIDs in the comment text table, we only need to write to the transaction table.
NOTE: This significantly degrades Differential, making inline comments pretty much useless (they each get their own transaction, and don't show line numbers or files). The data is all fine, but the UI is garbage now. This needs to be fixed before we can deploy this to users, but it's easily separable since it's all just display code.
Specifically, they look like this:
{F112270}
Test Plan:
I've migrated locally and put things through their paces, but it's hard to catch sketchy stuff locally because most of my test data is nonsense and bad migrations wouldn't necessarily look out of place.
IMPORTANT: I'm planning to push this to a branch and then shift production over to the branch, and run it for a day or two before bringing it to master.
I generally feel good about this change: it's not that big since we were able to separate a lot of pieces out of it, and it's pretty straightforward. That said, it's still one of the most scary/dangerous changes we've ever made.
Reviewers: btrahan
CC: chad, aran
Maniphest Tasks: T2222
Differential Revision: https://secure.phabricator.com/D8210
2014-02-12 00:36:58 +01:00
|
|
|
}
|
|
|
|
unset($xaction['phid']);
|
|
|
|
|
|
|
|
$comment_phid = null;
|
|
|
|
$comment_version = 0;
|
|
|
|
if (idx($xaction, 'comment')) {
|
|
|
|
$comment_phid = $xaction['comment']['phid'];
|
|
|
|
$comment_version = 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
$old = idx($xaction, 'old');
|
|
|
|
$new = idx($xaction, 'new');
|
|
|
|
$meta = idx($xaction, 'meta', array());
|
|
|
|
|
|
|
|
queryfx(
|
|
|
|
$conn_w,
|
|
|
|
'INSERT INTO %T (phid, authorPHID, objectPHID, viewPolicy, editPolicy,
|
|
|
|
commentPHID, commentVersion, transactionType, oldValue, newValue,
|
|
|
|
contentSource, metadata, dateCreated, dateModified)
|
|
|
|
VALUES (%s, %s, %s, %s, %s, %ns, %d, %s, %ns, %ns, %s, %s, %d, %d)',
|
|
|
|
'differential_transaction',
|
|
|
|
|
|
|
|
// PHID, authorPHID, objectPHID
|
|
|
|
$xaction_phid,
|
|
|
|
(string)$row['authorPHID'],
|
|
|
|
$revision_phid,
|
|
|
|
|
|
|
|
// viewPolicy, editPolicy, commentPHID, commentVersion
|
|
|
|
'public',
|
|
|
|
(string)$row['authorPHID'],
|
|
|
|
$comment_phid,
|
|
|
|
$comment_version,
|
|
|
|
|
|
|
|
// transactionType, oldValue, newValue, contentSource, metadata
|
|
|
|
$xaction['type'],
|
|
|
|
json_encode($old),
|
|
|
|
json_encode($new),
|
|
|
|
$content_source,
|
|
|
|
json_encode($meta),
|
|
|
|
|
|
|
|
// dates
|
|
|
|
$row['dateCreated'],
|
|
|
|
$row['dateModified']);
|
|
|
|
}
|
|
|
|
|
|
|
|
}
|
|
|
|
echo "Done.\n";
|