revi-archive/phorge-phorge
1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-26 16:52:41 +01:00
Code Issues Releases Wiki Activity

Increase the size of the Diffusion commit cache

Browse source 
Summary:
Ref T12296. This cache is used to cache Git ref heads (branches, tags, etc). Reasonable repositories may have more than 2048 of these.

When we miss the cache, we need to single-get refs to check them, which is relatively expensive.

Increasing the size of the cache to 65535 should only require about 7.5MB of RAM.

Additionally, fill only as much of the cache as actually fits. The FIFO nature of the cache can get us into trouble otherwise.

If we insert "A, B, C, D" and then lookup A, B, C, D, but the cache has maximum size 3, we get this:

  - Insert A, B, C, D: cache is now "B, C, D".
  - Lookup A: miss, single get, insert, purge, cache is now "C, D, A".
  - Lookup B: miss, singel get, insert, purge, cache is now "D, A, B".

Test Plan:
  - Reduced cache size to 5, observed reasonable behavior on the `array_slice()` locally with `bin/repository update` + `var_dump()`.
  - Used this script to estimate the size of 65535 cache entries as 7.5MB:

```
epriestley@orbital ~ $ cat size.php
<?php

$cache = array();

$mem_start = memory_get_usage();
for ($ii = 0; $ii < 65535; $ii++) {
  $cache[sha1($ii)] = true;
}

echo number_format(memory_get_usage() - $mem_start)." bytes\n";
epriestley@orbital ~ $ php -f size.php
7,602,176 bytes
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12296

Differential Revision: https://secure.phabricator.com/D17409
This commit is contained in:
epriestley 2017-02-24 10:01:11 -08:00
parent 40cc403d23
commit 4270649abe
1 changed files with 11 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
src/applications/repository/engine/PhabricatorRepositoryDiscoveryEngine.php
View file

@ -14,7 +14,7 @@ final class PhabricatorRepositoryDiscoveryEngine
private $commitCache = array();
private $workingSet = array();
const MAX_COMMIT_CACHE_SIZE = 2048;
const MAX_COMMIT_CACHE_SIZE = 65535;
/* -( Discovering Repositories )------------------------------------------- */
@ -476,6 +476,15 @@ final class PhabricatorRepositoryDiscoveryEngine
return;
}
$max_size = self::MAX_COMMIT_CACHE_SIZE;
// If we're filling more identifiers than would fit in the cache, ignore
// the ones that don't fit. Because the cache is FIFO, overfilling it can
// cause the entire cache to miss. See T12296.
if (count($identifiers) > $max_size) {
$identifiers = array_slice($identifiers, 0, $max_size);
}
// When filling the cache we ignore commits which have been marked as
// unreachable, treating them as though they do not exist. When recording
// commits later we'll revive commits that exist but are unreachable.
@ -492,7 +501,7 @@ final class PhabricatorRepositoryDiscoveryEngine
$this->commitCache[$commit->getCommitIdentifier()] = true;
}
while (count($this->commitCache) > self::MAX_COMMIT_CACHE_SIZE) {
while (count($this->commitCache) > $max_size) {
array_shift($this->commitCache);
}
}