Add aggregated facts to the Facts application

Summary:
Some facts are aggregations of other facts. For example, we may compute how many times each macro is used in each object as a "raw fact":

  Dnnn uses macro "psyduck" 6 times.

But we want to present this data in aggregate form, e.g. "order macros by popularity". We can do this at runtime and it probably won't be too awful a query, but we can also aggregate it cheaply:

  Macro "psyduck" is used 3920 times across all objects.

...and then do a query like "select macros ordered by usage".

"Aggregate" facts support facts like this. The aggregate facts I've implemented are:

  - Count of all objects.
  - Count of objects of type X.
  - Last time facts were updated.

These clearly fit the "aggregate" facts template well. I'm not 100% sure macros do. We can use this table to answer a question like "What are the most popular macros, ordered by use?" We can also use it to answer a question like "What are the most popular macros in the last 6 months?", if we build a specific fact for that. But we can't use it to answer a question like "What are the most popular macros between times X and Y?". Maybe that's important; maybe not.

This seems like a good fit for at least some types of facts.

I'll de-magic the keys a bit in the next diff.

Test Plan: Ran the engines and got some aggregated facts about other facts.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Maniphest Tasks: T1562

Differential Revision: https://secure.phabricator.com/D3089

src/__phutil_library_map__.php

@@ -624,10 +624,14 @@ phutil_register_library_map(array(
     'PhabricatorEventEngine' => 'infrastructure/events/PhabricatorEventEngine.php',
     'PhabricatorEventType' => 'infrastructure/events/constant/PhabricatorEventType.php',
     'PhabricatorExampleEventListener' => 'infrastructure/events/PhabricatorExampleEventListener.php',
+    'PhabricatorFactAggregate' => 'applications/fact/storage/PhabricatorFactAggregate.php',
+    'PhabricatorFactController' => 'applications/fact/controller/PhabricatorFactController.php',
     'PhabricatorFactCountEngine' => 'applications/fact/engine/PhabricatorFactCountEngine.php',
     'PhabricatorFactDAO' => 'applications/fact/storage/PhabricatorFactDAO.php',
     'PhabricatorFactDaemon' => 'applications/fact/daemon/PhabricatorFactDaemon.php',
     'PhabricatorFactEngine' => 'applications/fact/engine/PhabricatorFactEngine.php',
+    'PhabricatorFactHomeController' => 'applications/fact/controller/PhabricatorFactHomeController.php',
+    'PhabricatorFactLastUpdatedEngine' => 'applications/fact/engine/PhabricatorFactLastUpdatedEngine.php',
     'PhabricatorFactManagementAnalyzeWorkflow' => 'applications/fact/management/PhabricatorFactManagementAnalyzeWorkflow.php',
     'PhabricatorFactManagementDestroyWorkflow' => 'applications/fact/management/PhabricatorFactManagementDestroyWorkflow.php',
     'PhabricatorFactManagementListWorkflow' => 'applications/fact/management/PhabricatorFactManagementListWorkflow.php',
@@ -1660,9 +1664,13 @@ phutil_register_library_map(array(
     'PhabricatorEvent' => 'PhutilEvent',
     'PhabricatorEventType' => 'PhutilEventType',
     'PhabricatorExampleEventListener' => 'PhutilEventListener',
+    'PhabricatorFactAggregate' => 'PhabricatorFactDAO',
+    'PhabricatorFactController' => 'PhabricatorController',
     'PhabricatorFactCountEngine' => 'PhabricatorFactEngine',
     'PhabricatorFactDAO' => 'PhabricatorLiskDAO',
     'PhabricatorFactDaemon' => 'PhabricatorDaemon',
+    'PhabricatorFactHomeController' => 'PhabricatorFactController',
+    'PhabricatorFactLastUpdatedEngine' => 'PhabricatorFactEngine',
     'PhabricatorFactManagementAnalyzeWorkflow' => 'PhabricatorFactManagementWorkflow',
     'PhabricatorFactManagementDestroyWorkflow' => 'PhabricatorFactManagementWorkflow',
     'PhabricatorFactManagementListWorkflow' => 'PhabricatorFactManagementWorkflow',

src/aphront/configuration/AphrontDefaultApplicationConfiguration.php

@@ -463,6 +463,10 @@ class AphrontDefaultApplicationConfiguration
 
       '/emailverify/(?P<code>[^/]+)/' =>
         'PhabricatorEmailVerificationController',
+
+      '/fact/' => array(
+        '' => 'PhabricatorFactHomeController',
+      ),
     );
   }
 

src/applications/fact/controller/PhabricatorFactController.php

@@ -0,0 +1,34 @@
+<?php
+
+/*
+ * Copyright 2012 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+abstract class PhabricatorFactController extends PhabricatorController {
+
+  public function buildStandardPageResponse($view, array $data) {
+    $page = $this->buildStandardPageView();
+
+    $page->setBaseURI('/fact/');
+    $page->setTitle(idx($data, 'title'));
+
+    $page->setGlyph("\xCE\xA3");
+    $page->appendChild($view);
+
+    $response = new AphrontWebpageResponse();
+    return $response->setContent($page->render());
+  }
+
+}

src/applications/fact/controller/PhabricatorFactHomeController.php

@@ -0,0 +1,59 @@
+<?php
+
+/*
+ * Copyright 2012 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+final class PhabricatorFactHomeController extends PhabricatorFactController {
+
+  public function processRequest() {
+
+    $facts = id(new PhabricatorFactAggregate())->loadAllWhere(
+      'factType LIKE %> OR factType = %s',
+      '+N:',
+      'updated');
+
+    $rows = array();
+    foreach ($facts as $fact) {
+      $rows[] = array(
+        phutil_escape_html($fact->getFactType()),
+        (int)$fact->getValueX(),
+      );
+    }
+
+    $table = new AphrontTableView($rows);
+    $table->setHeaders(
+      array(
+        'Fact',
+        'Value',
+      ));
+    $table->setColumnClasses(
+      array(
+        'wide',
+        'n',
+      ));
+
+    $panel = new AphrontPanelView();
+    $panel->setHeader('Facts!');
+    $panel->appendChild($table);
+
+    return $this->buildStandardPageResponse(
+      $panel,
+      array(
+        'title' => 'Facts!',
+      ));
+  }
+
+}

src/applications/fact/daemon/PhabricatorFactDaemon.php

@@ -54,6 +54,22 @@ final class PhabricatorFactDaemon extends PhabricatorDaemon {
     return $result;
   }
 
+  public function processAggregates() {
+    $facts = $this->computeAggregateFacts();
+    $this->updateAggregateFacts($facts);
+  }
+
+  private function computeAggregateFacts() {
+    $facts = array();
+    foreach ($this->engines as $engine) {
+      if (!$engine->shouldComputeAggregateFacts()) {
+        continue;
+      }
+      $facts[] = $engine->computeAggregateFacts();
+    }
+    return array_mergev($facts);
+  }
+
   private function computeRawFacts(PhabricatorLiskDAO $object) {
     $facts = array();
     foreach ($this->engines as $engine) {
@@ -118,4 +134,34 @@ final class PhabricatorFactDaemon extends PhabricatorDaemon {
     $table->saveTransaction();
   }
 
+  private function updateAggregateFacts(array $facts) {
+    if (!$facts) {
+      return;
+    }
+
+    $table = new PhabricatorFactAggregate();
+    $conn = $table->establishConnection('w');
+    $table_name = $table->getTableName();
+
+    $sql = array();
+    foreach ($facts as $fact) {
+      $sql[] = qsprintf(
+        $conn,
+        '(%s, %s, %d)',
+        $fact->getFactType(),
+        $fact->getObjectPHID(),
+        $fact->getValueX());
+    }
+
+    foreach (array_chunk($sql, 256) as $chunk) {
+      queryfx(
+        $conn,
+        'INSERT INTO %T (factType, objectPHID, valueX) VALUES %Q
+          ON DUPLICATE KEY UPDATE valueX = VALUES(valueX)',
+        $table_name,
+        implode(', ', $chunk));
+    }
+
+  }
+
 }

src/applications/fact/engine/PhabricatorFactCountEngine.php

@@ -41,4 +41,31 @@ final class PhabricatorFactCountEngine extends PhabricatorFactEngine {
     return $facts;
   }
 
+  public function shouldComputeAggregateFacts() {
+    return true;
+  }
+
+  public function computeAggregateFacts() {
+    $table = new PhabricatorFactRaw();
+    $table_name = $table->getTableName();
+    $conn = $table->establishConnection('r');
+
+    $counts = queryfx_all(
+      $conn,
+      'SELECT factType, count(*) N FROM %T WHERE factType LIKE %>
+        GROUP BY factType',
+      $table_name,
+      'N:');
+
+    $facts = array();
+    foreach ($counts as $count) {
+      $facts[] = id(new PhabricatorFactAggregate())
+        ->setFactType('+'.$count['factType'])
+        ->setValueX($count['N']);
+    }
+
+    return $facts;
+  }
+
+
 }

src/applications/fact/engine/PhabricatorFactEngine.php

@@ -40,4 +40,12 @@ abstract class PhabricatorFactEngine {
     return array();
   }
 
+  public function shouldComputeAggregateFacts() {
+    return false;
+  }
+
+  public function computeAggregateFacts() {
+    return array();
+  }
+
 }

src/applications/fact/engine/PhabricatorFactLastUpdatedEngine.php

@@ -0,0 +1,38 @@
+<?php
+
+/*
+ * Copyright 2012 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Engine that records the time facts were last updated.
+ */
+final class PhabricatorFactLastUpdatedEngine extends PhabricatorFactEngine {
+
+  public function shouldComputeAggregateFacts() {
+    return true;
+  }
+
+  public function computeAggregateFacts() {
+    $facts = array();
+
+    $facts[] = id(new PhabricatorFactAggregate())
+      ->setFactType('updated')
+      ->setValueX(time());
+
+    return $facts;
+  }
+
+}

src/applications/fact/management/PhabricatorFactManagementAnalyzeWorkflow.php

@@ -41,6 +41,8 @@ final class PhabricatorFactManagementAnalyzeWorkflow
       $daemon->processIterator($iterator);
     }
 
+    $daemon->processAggregates();
+
     return 0;
   }
 

src/applications/fact/management/PhabricatorFactManagementDestroyWorkflow.php

@@ -40,6 +40,7 @@ final class PhabricatorFactManagementDestroyWorkflow
 
     $tables = array();
     $tables[] = new PhabricatorFactRaw();
+    $tables[] = new PhabricatorFactAggregate();
     foreach ($tables as $table) {
       $conn = $table->establishConnection('w');
       $name = $table->getTableName();

src/applications/fact/management/PhabricatorFactManagementStatusWorkflow.php

@@ -31,6 +31,7 @@ final class PhabricatorFactManagementStatusWorkflow
 
     $map = array(
       'raw' => new PhabricatorFactRaw(),
+      'agg' => new PhabricatorFactAggregate(),
     );
 
     foreach ($map as $type => $table) {
@@ -48,6 +49,9 @@ final class PhabricatorFactManagementStatusWorkflow
         case 'raw':
           $desc = pht('There are %d raw fact(s) in storage.', $n);
           break;
+        case 'agg':
+          $desc = pht('There are %d aggregate fact(s) in storage.', $n);
+          break;
       }
 
       $console->writeOut("%s\n", $desc);

src/applications/fact/storage/PhabricatorFactAggregate.php

@@ -0,0 +1,26 @@
+<?php
+
+/*
+ * Copyright 2012 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+final class PhabricatorFactAggregate extends PhabricatorFactDAO {
+
+  protected $id;
+  protected $factType;
+  protected $objectPHID;
+  protected $valueX;
+
+}

src/infrastructure/internationalization/PhabricatorBaseEnglishTranslation.php

@@ -126,6 +126,11 @@ abstract class PhabricatorBaseEnglishTranslation
         'There are %d raw facts in storage.',
       ),
 
+      'There are %d aggregate fact(s) in storage.' => array(
+        'There is %d aggregate fact in storage.',
+        'There are %d aggregate facts in storage.',
+      ),
+
     );
   }