1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-21 22:32:41 +01:00

Add an option to make it easier to debug page hangs

Summary:
Fixes T6044. We've had two cases (both the same install, coincidentally) where pages got hung doing too much data fetching.

When pages hang, we don't get a useful stack trace out of them, since nginx, php-fpm, or PHP eventually terminates things in a non-useful way without any diagnostic information.

The second time (the recent Macros issue) I was able to walk the install through removing limits on nginx, php-fpm, php, and eventually getting a profile by letting the page run for several minutes until the request completed. However, this install is exceptionally technically proficient and this was still a big pain for everyone, and this approach would not have worked if the page actually looped rather than just taking a long time.

Provide `debug.time-limit`, which should give us a better tool for reacting to this situation: by setting it to a small value (like 10), we'll kill the page after 10 seconds with a trace, before nginx/php-fpm/php/etc can kill it uselessly. Hopefully that will be enough information to find the issue (generally, getting a trace has been 95% of the problem in the two cases we've encountered).

Test Plan: Set this option to `3` and added a sleep loop, saw a termination after 3 seconds with a useful trace.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: csilvers, joshuaspence, epriestley

Maniphest Tasks: T6044

Differential Revision: https://secure.phabricator.com/D10465
This commit is contained in:
epriestley 2014-09-11 06:28:21 -07:00
parent d3cd9115f9
commit cae59d8345
3 changed files with 96 additions and 0 deletions

View file

@ -46,6 +46,31 @@ final class PhabricatorDeveloperConfigOptions
"enable this option in production.\n\n".
"You must enable DarkConsole by setting {{darkconsole.enabled}} ".
"before this option will have any effect.")),
$this->newOption('debug.time-limit', 'int', null)
->setSummary(
pht(
'Limit page execution time to debug hangs.'))
->setDescription(
pht(
"This option can help debug pages which are taking a very ".
"long time (more than 30 seconds) to render.\n\n".
"If a page is slow to render (but taking less than 30 seconds), ".
"the best tools to use to figure out why it is slow are usually ".
"the DarkConsole service call profiler and XHProf.\n\n".
"However, if a request takes a very long time to return, some ".
"components (like Apache, nginx, or PHP itself) may abort the ".
"request before it finishes. This can prevent you from using ".
"profiling tools to understand page performance in detail.\n\n".
"In these cases, you can use this option to force the page to ".
"abort after a smaller number of seconds (for example, 10), and ".
"dump a useful stack trace. This can provide useful information ".
"about why a page is hanging.\n\n".
"To use this option, set it to a small number (like 10), and ".
"reload a hanging page. The page should exit after 10 seconds ".
"and give you a stack trace.\n\n".
"You should turn this option off (set it to 0) when you are ".
"done with it. Leaving it on creates a small amount of overhead ".
"for all requests, even if they do not hit the time limit.")),
$this->newOption('debug.stop-on-redirect', 'bool', false)
->setBoolOptions(
array(

View file

@ -38,6 +38,7 @@
final class PhabricatorStartup {
private static $startTime;
private static $debugTimeLimit;
private static $globals = array();
private static $capturingOutput;
private static $rawInput;
@ -226,6 +227,70 @@ final class PhabricatorStartup {
}
/* -( Debug Time Limit )--------------------------------------------------- */
/**
* Set a time limit (in seconds) for the current script. After time expires,
* the script fatals.
*
* This works like `max_execution_time`, but prints out a useful stack trace
* when the time limit expires. This is primarily intended to make it easier
* to debug pages which hang by allowing extraction of a stack trace: set a
* short debug limit, then use the trace to figure out what's happening.
*
* The limit is implemented with a tick function, so enabling it implies
* some accounting overhead.
*
* @param int Time limit in seconds.
* @return void
*/
public static function setDebugTimeLimit($limit) {
self::$debugTimeLimit = $limit;
static $initialized;
if (!$initialized) {
declare(ticks=1);
register_tick_function(array('PhabricatorStartup', 'onDebugTick'));
}
}
/**
* Callback tick function used by @{method:setDebugTimeLimit}.
*
* Fatals with a useful stack trace after the time limit expires.
*
* @return void
*/
public static function onDebugTick() {
$limit = self::$debugTimeLimit;
if (!$limit) {
return;
}
$elapsed = (microtime(true) - self::getStartTime());
if ($elapsed > $limit) {
$frames = array();
foreach (debug_backtrace() as $frame) {
$file = isset($frame['file']) ? $frame['file'] : '-';
$file = basename($file);
$line = isset($frame['line']) ? $frame['line'] : '-';
$class = isset($frame['class']) ? $frame['class'].'->' : null;
$func = isset($frame['function']) ? $frame['function'].'()' : '?';
$frames[] = "{$file}:{$line} {$class}{$func}";
}
self::didFatal(
"Request aborted by debug time limit after {$limit} seconds.\n\n".
"STACK TRACE\n".
implode("\n", $frames));
}
}
/* -( In Case of Apocalypse )---------------------------------------------- */

View file

@ -16,6 +16,12 @@ try {
PhabricatorStartup::loadCoreLibraries();
PhabricatorEnv::initializeWebEnvironment();
$debug_time_limit = PhabricatorEnv::getEnvConfig('debug.time-limit');
if ($debug_time_limit) {
PhabricatorStartup::setDebugTimeLimit($debug_time_limit);
}
$show_unexpected_traces = PhabricatorEnv::getEnvConfig(
'phabricator.developer-mode');