When queries overheat, raise an exception

Summary:
Ref T13259. Currently, queries set a flag and return a partial result set when they overheat. This is mostly okay:

  - It's very unusual for queries to overheat if they don't have a real viewer.
  - Overheating is rare in general.
  - In most cases where queries can overheat, the context is a SearchEngine UI, which handles this properly.

In T13259, we hit a case where a query with an omnipotent viewer can overheat: if you have more than 1,000 consecutive commits in the database with invalid `repositoryID` values, we'll overheat and bail out. This is pretty bad, since we don't process everything.

Change this beahvior:

  - Throw by default, so this stuff doesn't slip through the cracks.
  - Handle the SearchEngine case explicitly ("it's okay to overheat, we'll handle it").
  - Make `QueryIterator` disable overheating behavior: if we're iterating over all objects, we want to hit the whole table even if most of it is garbage.

There are some cases where this might cause new exception behavior that we don't necessarily want. For example, in Owners, each package shows "recent commits in this package". If you can't see the first 1,000 recent commits, you'd previously get a slow page with no results. Now you'll probably get an exception.

If these crop up, I think the best approach for now is to deal with them on a case-by-case basis and see how far we get. In the "Owners" case, it might be good to query by repositories you can see first, then query by commits in the package in those repositories. That should give us a better outcome than any generic behavior we could implement.

Test Plan:
  - Added 100000 to all repositoryID values for commits on my local install.
  - Before making changes, ran `bin/repository rebuild-identities --all --trace`. Saw the script process 1,000 rows and exit silently.
  - Applied the first part ("throw by default") and ran `bin/repository rebuild-identities`. Saw the script process 1,000 rows, then raise an exception.
  - Applied the second part ("disable for queryiterator") and ran the script again. Saw the script process all 15,000 rows without issues (although none are valid and none actually load).
  - Viewed Diffusion, saw appropriate NUX / "overheated" UIs.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13259

Differential Revision: https://secure.phabricator.com/D20294

src/applications/search/controller/PhabricatorApplicationSearchController.php

@@ -249,6 +249,8 @@ final class PhabricatorApplicationSearchController
         $pager = $engine->newPagerForSavedQuery($saved_query);
         $pager->readFromRequest($request);
 
+        $query->setReturnPartialResultsOnOverheat(true);
+
         $objects = $engine->executeQuery($query, $pager);
 
         $force_nux = $request->getBool('nux');
@@ -798,6 +800,7 @@ final class PhabricatorApplicationSearchController
       $object = $query
         ->setViewer(PhabricatorUser::getOmnipotentUser())
         ->setLimit(1)
+        ->setReturnPartialResultsOnOverheat(true)
         ->execute();
       if ($object) {
         return null;

src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php

@@ -45,6 +45,8 @@ abstract class PhabricatorPolicyAwareQuery extends PhabricatorOffsetPagedQuery {
    */
   private $raisePolicyExceptions;
   private $isOverheated;
+  private $returnPartialResultsOnOverheat;
+  private $disableOverheating;
 
 
 /* -(  Query Configuration  )------------------------------------------------ */
@@ -130,6 +132,16 @@ abstract class PhabricatorPolicyAwareQuery extends PhabricatorOffsetPagedQuery {
     return $this;
   }
 
+  final public function setReturnPartialResultsOnOverheat($bool) {
+    $this->returnPartialResultsOnOverheat = $bool;
+    return $this;
+  }
+
+  final public function setDisableOverheating($disable_overheating) {
+    $this->disableOverheating = $disable_overheating;
+    return $this;
+  }
+
 
 /* -(  Query Execution  )---------------------------------------------------- */
 
@@ -319,10 +331,23 @@ abstract class PhabricatorPolicyAwareQuery extends PhabricatorOffsetPagedQuery {
         break;
       }
 
+      if (!$this->disableOverheating) {
         if ($overheat_limit && ($total_seen >= $overheat_limit)) {
           $this->isOverheated = true;
+
+          if (!$this->returnPartialResultsOnOverheat) {
+            throw new Exception(
+              pht(
+                'Query (of class "%s") overheated: examined more than %s '.
+                'raw rows without finding %s visible objects.',
+                get_class($this),
+                new PhutilNumber($overheat_limit),
+                new PhutilNumber($need)));
+          }
+
           break;
         }
+      }
     } while (true);
 
     $results = $this->didLoadResults($results);

src/infrastructure/storage/lisk/PhabricatorQueryIterator.php

@@ -25,6 +25,8 @@ final class PhabricatorQueryIterator extends PhutilBufferedIterator {
     $pager = clone $this->pager;
     $query = clone $this->query;
 
+    $query->setDisableOverheating(true);
+
     $results = $query->executeWithCursorPager($pager);
 
     // If we got less than a full page of results, this was the last set of