revi-archive/phorge-phorge
1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-19 12:00:55 +01:00
Code Issues Releases Wiki Activity

Implement smart waits for rarely updated repositories

Browse source 
Summary:
Ref T4605. When figuring out how long to wait to update a repository, factor in when it was last pushed. For rarely updated repositories, wait longer between updates.

(A slightly funky thing about this is that empty repos update every 15 seconds, but that seems OK for the moment.)

Test Plan:
Ran `bin/phd debug pulllocal` and saw sensible calculations and output:

```
...
<VERB> PhabricatorRepositoryPullLocalDaemon Last commit to repository "rPOEMS" was 1,239,608 seconds ago; considering a wait of 6,198 seconds before update.
>>> [79] <query> SELECT * FROM `repository` r   ORDER BY r.id DESC
<<< [79] <query> 514 us
>>> [80] <query> SELECT * FROM `repository_statusmessage` WHERE statusType = 'needs-update'
<<< [80] <query> 406 us
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rINIH" is not due for an update for 8,754 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rDUCK" is not due for an update for 14 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rMTESTX" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rQWER" is not due for an update for 14 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rBT" is not due for an update for 13 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rSVNX" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rINIG" is not due for an update for 13 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rHGTEST" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rBTX" is not due for an update for 14 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rGX" is not due for an update for 13 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rMTX" is currently updating.
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPOEMS" is not due for an update for 6,198 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPHU" is currently updating.
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rSVN" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPHY" is currently updating.
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rGTEST" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rINIS" is not due for an update for 6,894 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rARCLINT" is not due for an update for 21,599 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rLPHX" is not due for an update for 1,979 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rARC" is not due for an update for 1,824 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rINIHG" is not due for an update for 21,599 second(s).
...
```

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4605

Differential Revision: https://secure.phabricator.com/D8782
This commit is contained in:
epriestley 2014-04-16 13:01:04 -07:00
parent 5671c4b276
commit c402d7d307
1 changed files with 43 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

43
src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php
View file

@ -346,6 +346,49 @@ final class PhabricatorRepositoryPullLocalDaemon
} }
$sleep_for = (int)$repository->getDetail('pull-frequency', $min_sleep); $sleep_for = (int)$repository->getDetail('pull-frequency', $min_sleep);
// Smart wait: pull rarely used repositories less frequently. Find the
// most recent commit which is older than the current time (this keeps us
// from spinning on repositories with a silly commit post-dated to some time
// in 2037), and adjust how frequently we pull based on how frequently this
// repository updates.
$table = id(new PhabricatorRepositoryCommit());
$last_commit = queryfx_one(
$table->establishConnection('w'),
'SELECT epoch FROM %T
WHERE repositoryID = %d AND epoch <= %d
ORDER BY epoch DESC LIMIT 1',
$table->getTableName(),
$repository->getID(),
time() + $min_sleep);
if ($last_commit) {
$time_since_commit = (time() + $min_sleep) - $last_commit['epoch'];
// Wait 0.5% of the time since the last commit before we pull. This gives
// us these wait times:
//
// 50 minutes or less: 15 seconds
// about 3 hours: 1 minute
// about 16 hours: 5 minutes
// about 2 days: 15 minutes
// 50 days or more: 6 hours
$smart_wait = ($time_since_commit / 200);
$smart_wait = min($smart_wait, phutil_units('6 hours in seconds'));
$this->log(
pht(
'Last commit to repository "%s" was %s seconds ago; considering '.
'a wait of %s seconds before update.',
$repository->getMonogram(),
new PhutilNumber($time_since_commit),
new PhutilNumber($smart_wait)));
$smart_wait = max(15, $smart_wait);
$sleep_for = max($smart_wait, $sleep_for);
}
if ($sleep_for < $min_sleep) { if ($sleep_for < $min_sleep) {
$sleep_for = $min_sleep; $sleep_for = $min_sleep;
} }