2012-12-25 15:09:51 +01:00
|
|
|
<?php
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @task setup Setup Cache
|
|
|
|
*/
|
|
|
|
final class PhabricatorCaches {
|
|
|
|
|
2013-03-23 00:28:06 +01:00
|
|
|
public static function getNamespace() {
|
|
|
|
return PhabricatorEnv::getEnvConfig('phabricator.cache-namespace');
|
|
|
|
}
|
2012-12-25 15:09:51 +01:00
|
|
|
|
Implement a chunked, APC-backed graph cache
Summary:
Ref T2683. This is a refinement and simplification of D5257. In particular:
- D5257 only cached the commit chain, not path changes. This meant that we had to go issue an awkward query (which was slow on Facebook's install) periodically while reading the cache. This was reasonable locally but killed performance at FB scale. Instead, we can include path information in the cache. It is very rare that this is large except in Subversion, and we do not need to use this cache in Subversion. In other VCSes, the scale of this data is quite small (a handful of bytes per commit on average).
- D5257 required a large, slow offline computation step. This relies on D9044 to populate parent data so we can build the cache online at will, and let it expire with normal LRU/LFU/whatever semantics. We need this parent data for other reasons anyway.
- D5257 separated graph chunks per-repository. This change assumes we'll be able to pull stuff from APC most of the time and that the cost of switching chunks is not very large, so we can just build one chunk cache across all repositories. This allows the cache to be simpler.
- D5257 needed an offline cache, and used a unique cache structure. Since this one can be built online it can mostly use normal cache code.
- This also supports online appends to the cache.
- Finally, this has a timeout to guarantee a ceiling on the worst case: the worst case is something like a query for a file that has never existed, in a repository which receives exactly 1 commit every time other repositories receive 4095 commits, on a cold cache. If we hit cases like this we can bail after warming the cache up a bit and fall back to asking the VCS for an answer.
This cache isn't perfect, but I believe it will give us substantial gains in the average case. It can often satisfy "average-looking" queries in 4-8ms, and pathological-ish queries in 20ms on my machine; `hg` usually can't even start up in less than 100ms. The major thing that's attractive about this approach is that it does not require anything external or complicated, and will "just work", even producing reasonble improvements for users without APC.
In followups, I'll modify queries to use this cache and see if it holds up in more realistic workloads.
Test Plan:
- Used `bin/repository cache` to examine the behavior of this cache.
- Did some profiling/testing from the web UI using `debug.php`.
- This //appears// to provide a reasonable fast way to issue this query very quickly in the average case, without the various issues that plagued D5257.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley, jhurwitz
Maniphest Tasks: T2683
Differential Revision: https://secure.phabricator.com/D9045
2014-05-10 19:10:13 +02:00
|
|
|
private static function newStackFromCaches(array $caches) {
|
|
|
|
$caches = self::addNamespaceToCaches($caches);
|
|
|
|
$caches = self::addProfilerToCaches($caches);
|
|
|
|
return id(new PhutilKeyValueCacheStack())
|
|
|
|
->setCaches($caches);
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/* -( Repository Graph Cache )--------------------------------------------- */
|
|
|
|
|
|
|
|
|
|
|
|
public static function getRepositoryGraphL1Cache() {
|
|
|
|
static $cache;
|
|
|
|
if (!$cache) {
|
|
|
|
$caches = self::buildRepositoryGraphL1Caches();
|
|
|
|
$cache = self::newStackFromCaches($caches);
|
|
|
|
}
|
|
|
|
return $cache;
|
|
|
|
}
|
|
|
|
|
|
|
|
private static function buildRepositoryGraphL1Caches() {
|
|
|
|
$caches = array();
|
|
|
|
|
2014-05-12 21:34:04 +02:00
|
|
|
$request = new PhutilKeyValueCacheInRequest();
|
|
|
|
$request->setLimit(32);
|
|
|
|
$caches[] = $request;
|
|
|
|
|
Implement a chunked, APC-backed graph cache
Summary:
Ref T2683. This is a refinement and simplification of D5257. In particular:
- D5257 only cached the commit chain, not path changes. This meant that we had to go issue an awkward query (which was slow on Facebook's install) periodically while reading the cache. This was reasonable locally but killed performance at FB scale. Instead, we can include path information in the cache. It is very rare that this is large except in Subversion, and we do not need to use this cache in Subversion. In other VCSes, the scale of this data is quite small (a handful of bytes per commit on average).
- D5257 required a large, slow offline computation step. This relies on D9044 to populate parent data so we can build the cache online at will, and let it expire with normal LRU/LFU/whatever semantics. We need this parent data for other reasons anyway.
- D5257 separated graph chunks per-repository. This change assumes we'll be able to pull stuff from APC most of the time and that the cost of switching chunks is not very large, so we can just build one chunk cache across all repositories. This allows the cache to be simpler.
- D5257 needed an offline cache, and used a unique cache structure. Since this one can be built online it can mostly use normal cache code.
- This also supports online appends to the cache.
- Finally, this has a timeout to guarantee a ceiling on the worst case: the worst case is something like a query for a file that has never existed, in a repository which receives exactly 1 commit every time other repositories receive 4095 commits, on a cold cache. If we hit cases like this we can bail after warming the cache up a bit and fall back to asking the VCS for an answer.
This cache isn't perfect, but I believe it will give us substantial gains in the average case. It can often satisfy "average-looking" queries in 4-8ms, and pathological-ish queries in 20ms on my machine; `hg` usually can't even start up in less than 100ms. The major thing that's attractive about this approach is that it does not require anything external or complicated, and will "just work", even producing reasonble improvements for users without APC.
In followups, I'll modify queries to use this cache and see if it holds up in more realistic workloads.
Test Plan:
- Used `bin/repository cache` to examine the behavior of this cache.
- Did some profiling/testing from the web UI using `debug.php`.
- This //appears// to provide a reasonable fast way to issue this query very quickly in the average case, without the various issues that plagued D5257.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley, jhurwitz
Maniphest Tasks: T2683
Differential Revision: https://secure.phabricator.com/D9045
2014-05-10 19:10:13 +02:00
|
|
|
$apc = new PhutilKeyValueCacheAPC();
|
|
|
|
if ($apc->isAvailable()) {
|
|
|
|
$caches[] = $apc;
|
|
|
|
}
|
|
|
|
|
|
|
|
return $caches;
|
|
|
|
}
|
|
|
|
|
|
|
|
public static function getRepositoryGraphL2Cache() {
|
|
|
|
static $cache;
|
|
|
|
if (!$cache) {
|
|
|
|
$caches = self::buildRepositoryGraphL2Caches();
|
|
|
|
$cache = self::newStackFromCaches($caches);
|
|
|
|
}
|
|
|
|
return $cache;
|
|
|
|
}
|
|
|
|
|
|
|
|
private static function buildRepositoryGraphL2Caches() {
|
|
|
|
$caches = array();
|
|
|
|
$caches[] = new PhabricatorKeyValueDatabaseCache();
|
|
|
|
return $caches;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2012-12-25 15:09:51 +01:00
|
|
|
/* -( Setup Cache )-------------------------------------------------------- */
|
|
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Highly specialized cache for performing setup checks. We use this cache
|
2013-01-22 19:32:26 +01:00
|
|
|
* to determine if we need to run expensive setup checks when the page
|
2012-12-25 15:09:51 +01:00
|
|
|
* loads. Without it, we would need to run these checks every time.
|
|
|
|
*
|
|
|
|
* Normally, this cache is just APC. In the absence of APC, this cache
|
|
|
|
* degrades into a slow, quirky on-disk cache.
|
|
|
|
*
|
|
|
|
* NOTE: Do not use this cache for anything else! It is not a general-purpose
|
|
|
|
* cache!
|
|
|
|
*
|
|
|
|
* @return PhutilKeyValueCacheStack Most qualified available cache stack.
|
|
|
|
* @task setup
|
|
|
|
*/
|
|
|
|
public static function getSetupCache() {
|
|
|
|
static $cache;
|
|
|
|
if (!$cache) {
|
|
|
|
$caches = self::buildSetupCaches();
|
Implement a chunked, APC-backed graph cache
Summary:
Ref T2683. This is a refinement and simplification of D5257. In particular:
- D5257 only cached the commit chain, not path changes. This meant that we had to go issue an awkward query (which was slow on Facebook's install) periodically while reading the cache. This was reasonable locally but killed performance at FB scale. Instead, we can include path information in the cache. It is very rare that this is large except in Subversion, and we do not need to use this cache in Subversion. In other VCSes, the scale of this data is quite small (a handful of bytes per commit on average).
- D5257 required a large, slow offline computation step. This relies on D9044 to populate parent data so we can build the cache online at will, and let it expire with normal LRU/LFU/whatever semantics. We need this parent data for other reasons anyway.
- D5257 separated graph chunks per-repository. This change assumes we'll be able to pull stuff from APC most of the time and that the cost of switching chunks is not very large, so we can just build one chunk cache across all repositories. This allows the cache to be simpler.
- D5257 needed an offline cache, and used a unique cache structure. Since this one can be built online it can mostly use normal cache code.
- This also supports online appends to the cache.
- Finally, this has a timeout to guarantee a ceiling on the worst case: the worst case is something like a query for a file that has never existed, in a repository which receives exactly 1 commit every time other repositories receive 4095 commits, on a cold cache. If we hit cases like this we can bail after warming the cache up a bit and fall back to asking the VCS for an answer.
This cache isn't perfect, but I believe it will give us substantial gains in the average case. It can often satisfy "average-looking" queries in 4-8ms, and pathological-ish queries in 20ms on my machine; `hg` usually can't even start up in less than 100ms. The major thing that's attractive about this approach is that it does not require anything external or complicated, and will "just work", even producing reasonble improvements for users without APC.
In followups, I'll modify queries to use this cache and see if it holds up in more realistic workloads.
Test Plan:
- Used `bin/repository cache` to examine the behavior of this cache.
- Did some profiling/testing from the web UI using `debug.php`.
- This //appears// to provide a reasonable fast way to issue this query very quickly in the average case, without the various issues that plagued D5257.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley, jhurwitz
Maniphest Tasks: T2683
Differential Revision: https://secure.phabricator.com/D9045
2014-05-10 19:10:13 +02:00
|
|
|
$cache = self::newStackFromCaches($caches);
|
2012-12-25 15:09:51 +01:00
|
|
|
}
|
|
|
|
return $cache;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @task setup
|
|
|
|
*/
|
|
|
|
private static function buildSetupCaches() {
|
|
|
|
// In most cases, we should have APC. This is an ideal cache for our
|
|
|
|
// purposes -- it's fast and empties on server restart.
|
|
|
|
$apc = new PhutilKeyValueCacheAPC();
|
|
|
|
if ($apc->isAvailable()) {
|
|
|
|
return array($apc);
|
|
|
|
}
|
|
|
|
|
|
|
|
// If we don't have APC, build a poor approximation on disk. This is still
|
|
|
|
// much better than nothing; some setup steps are quite slow.
|
|
|
|
$disk_path = self::getSetupCacheDiskCachePath();
|
|
|
|
if ($disk_path) {
|
|
|
|
$disk = new PhutilKeyValueCacheOnDisk();
|
|
|
|
$disk->setCacheFile($disk_path);
|
2013-04-13 16:09:32 +02:00
|
|
|
$disk->setWait(0.1);
|
2012-12-25 15:09:51 +01:00
|
|
|
if ($disk->isAvailable()) {
|
|
|
|
return array($disk);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return array();
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @task setup
|
|
|
|
*/
|
|
|
|
private static function getSetupCacheDiskCachePath() {
|
|
|
|
// The difficulty here is in choosing a path which will change on server
|
|
|
|
// restart (we MUST have this property), but as rarely as possible
|
|
|
|
// otherwise (we desire this property to give the cache the best hit rate
|
|
|
|
// we can).
|
|
|
|
|
|
|
|
// In some setups, the parent PID is more stable and longer-lived that the
|
|
|
|
// PID (e.g., under apache, our PID will be a worker while the ppid will
|
|
|
|
// be the main httpd process). If we're confident we're running under such
|
|
|
|
// a setup, we can try to use the PPID as the basis for our cache instead
|
|
|
|
// of our own PID.
|
|
|
|
$use_ppid = false;
|
|
|
|
|
|
|
|
switch (php_sapi_name()) {
|
|
|
|
case 'cli-server':
|
|
|
|
// This is the PHP5.4+ built-in webserver. We should use the pid
|
|
|
|
// (the server), not the ppid (probably a shell or something).
|
|
|
|
$use_ppid = false;
|
|
|
|
break;
|
|
|
|
case 'fpm-fcgi':
|
|
|
|
// We should be safe to use PPID here.
|
|
|
|
$use_ppid = true;
|
|
|
|
break;
|
|
|
|
case 'apache2handler':
|
|
|
|
// We're definitely safe to use the PPID.
|
|
|
|
$use_ppid = true;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
$pid_basis = getmypid();
|
|
|
|
if ($use_ppid) {
|
|
|
|
if (function_exists('posix_getppid')) {
|
|
|
|
$parent_pid = posix_getppid();
|
|
|
|
// On most systems, normal processes can never have PIDs lower than 100,
|
|
|
|
// so something likely went wrong if we we get one of these.
|
|
|
|
if ($parent_pid > 100) {
|
|
|
|
$pid_basis = $parent_pid;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// If possible, we also want to know when the process launched, so we can
|
|
|
|
// drop the cache if a process restarts but gets the same PID an earlier
|
|
|
|
// process had. "/proc" is not available everywhere (e.g., not on OSX), but
|
|
|
|
// check if we have it.
|
|
|
|
$epoch_basis = null;
|
|
|
|
$stat = @stat("/proc/{$pid_basis}");
|
|
|
|
if ($stat !== false) {
|
|
|
|
$epoch_basis = $stat['ctime'];
|
|
|
|
}
|
|
|
|
|
|
|
|
$tmp_dir = sys_get_temp_dir();
|
|
|
|
|
|
|
|
$tmp_path = $tmp_dir.DIRECTORY_SEPARATOR.'phabricator-setup';
|
|
|
|
if (!file_exists($tmp_path)) {
|
|
|
|
@mkdir($tmp_path);
|
|
|
|
}
|
|
|
|
|
|
|
|
$is_ok = self::testTemporaryDirectory($tmp_path);
|
|
|
|
if (!$is_ok) {
|
|
|
|
$tmp_path = $tmp_dir;
|
|
|
|
$is_ok = self::testTemporaryDirectory($tmp_path);
|
|
|
|
if (!$is_ok) {
|
|
|
|
// We can't find anywhere to write the cache, so just bail.
|
|
|
|
return null;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
$tmp_name = 'setup-'.$pid_basis;
|
|
|
|
if ($epoch_basis) {
|
|
|
|
$tmp_name .= '.'.$epoch_basis;
|
|
|
|
}
|
|
|
|
$tmp_name .= '.cache';
|
|
|
|
|
|
|
|
return $tmp_path.DIRECTORY_SEPARATOR.$tmp_name;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @task setup
|
|
|
|
*/
|
|
|
|
private static function testTemporaryDirectory($dir) {
|
|
|
|
if (!@file_exists($dir)) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
if (!@is_dir($dir)) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
if (!@is_writable($dir)) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2012-12-31 02:04:38 +01:00
|
|
|
private static function addProfilerToCaches(array $caches) {
|
|
|
|
foreach ($caches as $key => $cache) {
|
|
|
|
$pcache = new PhutilKeyValueCacheProfiler($cache);
|
|
|
|
$pcache->setProfiler(PhutilServiceProfiler::getInstance());
|
|
|
|
$caches[$key] = $pcache;
|
|
|
|
}
|
|
|
|
return $caches;
|
|
|
|
}
|
|
|
|
|
2013-04-13 16:09:32 +02:00
|
|
|
private static function addNamespaceToCaches(array $caches) {
|
|
|
|
$namespace = PhabricatorCaches::getNamespace();
|
|
|
|
if (!$namespace) {
|
|
|
|
return $caches;
|
|
|
|
}
|
|
|
|
|
|
|
|
foreach ($caches as $key => $cache) {
|
|
|
|
$ncache = new PhutilKeyValueCacheNamespace($cache);
|
|
|
|
$ncache->setNamespace($namespace);
|
|
|
|
$caches[$key] = $ncache;
|
|
|
|
}
|
|
|
|
|
|
|
|
return $caches;
|
|
|
|
}
|
|
|
|
|
2012-12-25 15:09:51 +01:00
|
|
|
}
|