2013-02-15 01:16:08 +01:00
|
|
|
<?php
|
|
|
|
|
|
|
|
abstract class ArcanistFutureLinter extends ArcanistLinter {
|
|
|
|
|
|
|
|
private $futures;
|
|
|
|
|
2013-02-19 22:09:34 +01:00
|
|
|
abstract protected function buildFutures(array $paths);
|
|
|
|
abstract protected function resolveFuture($path, Future $future);
|
2013-02-15 01:16:08 +01:00
|
|
|
|
2014-05-12 18:48:52 +02:00
|
|
|
final protected function getFuturesLimit() {
|
2013-02-21 10:22:25 +01:00
|
|
|
return 8;
|
|
|
|
}
|
|
|
|
|
2014-05-12 18:48:52 +02:00
|
|
|
final public function willLintPaths(array $paths) {
|
2013-02-21 10:22:25 +01:00
|
|
|
$limit = $this->getFuturesLimit();
|
2014-12-30 13:14:32 +01:00
|
|
|
$this->futures = id(new FutureIterator(array()))->limit($limit);
|
2013-02-21 10:22:25 +01:00
|
|
|
foreach ($this->buildFutures($paths) as $path => $future) {
|
|
|
|
$this->futures->addFuture($future, $path);
|
2013-02-15 01:16:08 +01:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Split large path lists into blocks when linting
Summary:
Fixes T5097. When linting a large list of paths (e.g., with `--everything`), do this internally:
$chunks = array_chunk($paths, 32);
foreach ($chunks as $chunk) {
$this->lintChunk($chunk);
}
This keeps the advantages of parallelism and artifact sharing for future-based linters, without having memory usage grow in an unbounded way.
These callbacks changed:
- `willLintPath()`: Useless, no meaningful implementations. Internalized the required side effect and broke the hook.
- `didRunLinters()`: Now useless, with no meaningful implementations. Broke the hook.
- `didLintPaths()`: New hook which executes opposite `willLintPaths()`.
- `lintPath()`: Linters no longer need to implement this method.
XHPAST now has an explicit way to release shared futures.
These minor changes also happened:
- Formalized the "linter ID", which is a semi-durable identifier for the cache.
- Removed linter -> exception explicit mapping, which was unused. We now just collect exceptions.
- We do the `canRun()` checks first (and separately) now.
- Share more service call profiling code.
- Fix an issue where the test harness would use the path on disk, even if configuration set a different path.
Test Plan:
- Ran `arc lint --everything` in `arcanist/`.
- With no chunking, saw **unstable** memory usage with a peak at 941 MB.
- With chunk size 32, saw **stable** memory usage with a peak at 269 MB.
- With chunk size 8, saw **stable** memory usage with a peak at 180 MB.
- Ran with `--trace` and saw profiling information.
- Created this diff.
Reviewers: joshuaspence
Reviewed By: joshuaspence
Subscribers: epriestley
Maniphest Tasks: T5097
Differential Revision: https://secure.phabricator.com/D12501
2015-04-22 14:15:57 +02:00
|
|
|
final public function lintPath($path) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
final public function didLintPaths(array $paths) {
|
|
|
|
if (!$this->futures) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
$map = array();
|
|
|
|
foreach ($this->futures as $path => $future) {
|
|
|
|
$this->setActivePath($path);
|
|
|
|
$this->resolveFuture($path, $future);
|
|
|
|
$map[$path] = $future;
|
2013-02-15 01:16:08 +01:00
|
|
|
}
|
Split large path lists into blocks when linting
Summary:
Fixes T5097. When linting a large list of paths (e.g., with `--everything`), do this internally:
$chunks = array_chunk($paths, 32);
foreach ($chunks as $chunk) {
$this->lintChunk($chunk);
}
This keeps the advantages of parallelism and artifact sharing for future-based linters, without having memory usage grow in an unbounded way.
These callbacks changed:
- `willLintPath()`: Useless, no meaningful implementations. Internalized the required side effect and broke the hook.
- `didRunLinters()`: Now useless, with no meaningful implementations. Broke the hook.
- `didLintPaths()`: New hook which executes opposite `willLintPaths()`.
- `lintPath()`: Linters no longer need to implement this method.
XHPAST now has an explicit way to release shared futures.
These minor changes also happened:
- Formalized the "linter ID", which is a semi-durable identifier for the cache.
- Removed linter -> exception explicit mapping, which was unused. We now just collect exceptions.
- We do the `canRun()` checks first (and separately) now.
- Share more service call profiling code.
- Fix an issue where the test harness would use the path on disk, even if configuration set a different path.
Test Plan:
- Ran `arc lint --everything` in `arcanist/`.
- With no chunking, saw **unstable** memory usage with a peak at 941 MB.
- With chunk size 32, saw **stable** memory usage with a peak at 269 MB.
- With chunk size 8, saw **stable** memory usage with a peak at 180 MB.
- Ran with `--trace` and saw profiling information.
- Created this diff.
Reviewers: joshuaspence
Reviewed By: joshuaspence
Subscribers: epriestley
Maniphest Tasks: T5097
Differential Revision: https://secure.phabricator.com/D12501
2015-04-22 14:15:57 +02:00
|
|
|
$this->futures = array();
|
|
|
|
|
|
|
|
$this->didResolveLinterFutures($map);
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Hook for cleaning up resources.
|
|
|
|
*
|
|
|
|
* This is invoked after a block of futures resolve, and allows linters to
|
|
|
|
* discard or clean up any shared resources they no longer need.
|
|
|
|
*
|
|
|
|
* @param map<string, Future> Map of paths to resolved futures.
|
|
|
|
* @return void
|
|
|
|
*/
|
|
|
|
protected function didResolveLinterFutures(array $futures) {
|
|
|
|
return;
|
2013-02-15 01:16:08 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
}
|