revi-archive/phorge-phorge
1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-22 21:40:55 +01:00
Code Issues Releases Wiki Activity

Improve performance of "phabricator:20210215.changeset.02.phid-populate.php"

Browse source 
Summary: Ref T13613. Improve the performance of this migration by using a temporary table and an "UPDATE x JOIN y ..." pattern.

Test Plan:
  - Ran on `secure`, got exit after a few seconds since the migration is idempotent and changesets already had PHIDs.
  - Ran on `secure` with the `continue;` commented out, got valid new PHIDs in 53s (from 153s).
  - Tried a larger page size (16K), didn't see any improvement.
  - From "--trace", client PHID generation seems to be the limiting factor.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13613

Differential Revision: https://secure.phabricator.com/D21570
This commit is contained in:
epriestley 2021-02-19 07:44:28 -08:00
parent 7c44657ca5
commit b3976acc40
1 changed files with 53 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

61
resources/sql/autopatches/20210215.changeset.02.phid-populate.php
View file

@ -7,20 +7,65 @@ $changeset_table = new DifferentialChangeset();
$conn = $changeset_table->establishConnection('w');
$table_name = $changeset_table->getTableName();
$iterator = new LiskRawMigrationIterator($conn, $table_name);
foreach ($iterator as $changeset_row) {
$phid = $changeset_row['phid'];
$chunk_size = 4096;
if (strlen($phid)) {
$temporary_table = 'tmp_20210215_changeset_id_map';
queryfx(
$conn,
'CREATE TEMPORARY TABLE %T (
changeset_id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
changeset_phid VARBINARY(64) NOT NULL)',
$temporary_table);
$table_iterator = id(new LiskRawMigrationIterator($conn, $table_name))
->setPageSize($chunk_size);
$chunk_iterator = new PhutilChunkedIterator($table_iterator, $chunk_size);
foreach ($chunk_iterator as $chunk) {
$map = array();
foreach ($chunk as $changeset_row) {
$phid = $changeset_row['phid'];
if (strlen($phid)) {
continue;
}
$phid = PhabricatorPHID::generateNewPHID($phid_type);
$id = $changeset_row['id'];
$map[(int)$id] = $phid;
}
if (!$map) {
continue;
}
$phid = PhabricatorPHID::generateNewPHID($phid_type);
$sql = array();
foreach ($map as $changeset_id => $changeset_phid) {
$sql[] = qsprintf(
$conn,
'(%d, %s)',
$changeset_id,
$changeset_phid);
}
queryfx(
$conn,
'UPDATE %T SET phid = %s WHERE id = %d',
'TRUNCATE TABLE %T',
$temporary_table);
queryfx(
$conn,
'INSERT INTO %T (changeset_id, changeset_phid) VALUES %LQ',
$temporary_table,
$sql);
queryfx(
$conn,
'UPDATE %T c JOIN %T x ON c.id = x.changeset_id
SET c.phid = x.changeset_phid',
$table_name,
$phid,
$changeset_row['id']);
$temporary_table);
}