From 7c934e41764f9640883b1a3e110adb72cbf81cd3 Mon Sep 17 00:00:00 2001 From: epriestley Date: Fri, 27 Jul 2012 13:34:21 -0700 Subject: [PATCH] Add a basic "fact" application Summary: Basic "Fact" application with some storage, part of a daemon, and a control binary. = Goals = The general idea is that we have various statistics we'd like to compute, like the frequency of image macros, reviewer responsiveness, task close rates, etc. Computing these on page load is expensive and messy. By building an ETL pipeline and running it in a daemon, we can precompute statistics and just pull them out of "stats" tables. One way to do this is just to completely hard-code everything, e.g. have a daemon that runs every hour which issues a big-ass query and dumps results into a table per-fact or per fact-group. But this has a bunch of drawbacks: adding new stuff to the pipeline is a pain, various fact aggregators can't share much code, updates are slow and expensive, we can never build generic graphs on top of it, etc. I'm hoping to build an ETL pipeline which is generic enough that we can use it for most things we're interested in without needing schema changes, and so that installs can use it also without needing schema changes, while still being specific enough that it's fast and we can build useful stuff on top of it. I'm not sure if this will actually work, but it would be cool if it does so I'm starting pretty generally and we'll see how far I get. I haven't built this exact sort of thing before so I might be way off. I'm basing the whole thing on analyzing entire objects, not analyzing changes to objects. So each part of the pipeline is handed an object and told "analyze this", not handed a change. It pretty much deletes all the old data about that thing and then writes new data. I think this is simpler to implement and understand, and it protects us from all sorts of weird issues where we end up with some kind of garbage in the DB and have to wipe the whole thing. = Facts = The general idea is that we extract "facts" out of objects, and then the various view interfaces just report those facts. This change has on type of fact, a "raw fact", which is directly derived from an object. These facts are concerete and relate specifically to the object they are derived from. Some examples of such facts might be: D123 has 9 comments. D123 uses macro "psyduck" 15 times. D123 adds 35 lines. D123 has 5 files. D123 has 1 object. D123 has 1 object of type "DREV". D123 was created at epoch timestamp 89812351235. D123 was accepted by @alincoln at epoch timestamp 8397981839. The fact storage looks like this: Currently, we supprot one optional secondary key (like a user PHID or macro PHID), two optional integer values, and an optional timestamp. We might add more later. Each fact type can use these fields if it wants. Some facts use them, others don't. For instance, this diff adds a "N:*" fact, which is just the count of total objects in the system. These facts just look like: <"N:*", "PHID-xxxx-yyyy", ...> ...where all other fields are ignored. But some of the more complex facts might look like: <"DREV:accept", "PHID-DREV-xxxx", "PHID-USER-yyyy", ..., ..., nnnn> # User 'yyyy' accepted at epoch 'nnnn'. <"FILE:macro", "PHID-DREV-xxxx", "PHID-MACR-yyyy", 17, ..., ...> # Object 'xxxx' uses macro 'yyyy' 17 times. Facts have no uniqueness constraints. For @vrana's reviewer responsiveness stuff, we can insert multiple rows for each reviewer, e.g. <"DREV:reviewed", "PHID-DREV-xxxx", "PHID-USER-yyyy", nnnn, ..., mmmm> # User 'yyyy' reviewed revision 'xxxx' after 'nnnn' seconds at 'mmmm'. The second value (valueY) is mostly because we need it if we sample anything (valueX = observed value, valueY = sample rate) but there might be other uses. We might need to add "objectB" at some point too -- currently we can't represent a fact like "User X used macro Y on revision Z", so it would be impossible to compute macro use rates //for a specific user// based on this schema. I think we can start here though and see how far we get. = Aggregated Facts = These aren't implemented yet, but the idea is that we can then take the "raw facts" and compute derived/aggregated/rollup facts based on the raw fact table. For example, the "count" fact can be aggregated to arrive at a count of all objects in the system. This stuff will live in a separate table which does have uniqueness constraints, and come in the next diff. We might need some kind of time series facts too, not sure about that. I think most of our use cases today are covered by raw facts + aggregated facts. Test Plan: Ran `bin/fact` commands and verified they seemed to do reasonable things. Reviewers: vrana, btrahan Reviewed By: vrana CC: aran, majak Maniphest Tasks: T1562 Differential Revision: https://secure.phabricator.com/D3078 --- bin/fact | 1 + scripts/fact/manage_facts.php | 42 ++++++ src/__phutil_library_map__.php | 23 +++- .../fact/daemon/PhabricatorFactDaemon.php | 121 ++++++++++++++++++ .../engine/PhabricatorFactCountEngine.php | 44 +++++++ .../fact/engine/PhabricatorFactEngine.php | 43 +++++++ .../PhabricatorFactUpdateIterator.php} | 2 +- ...abricatorFactManagementAnalyzeWorkflow.php | 47 +++++++ ...abricatorFactManagementDestroyWorkflow.php | 58 +++++++++ .../PhabricatorFactManagementListWorkflow.php | 40 ++++++ ...habricatorFactManagementStatusWorkflow.php | 59 +++++++++ .../PhabricatorFactManagementWorkflow.php | 26 ++++ .../fact/storage/PhabricatorFactDAO.php | 31 +++++ .../fact/storage/PhabricatorFactRaw.php | 32 +++++ .../PhabricatorBaseEnglishTranslation.php | 5 + 15 files changed, 571 insertions(+), 3 deletions(-) create mode 120000 bin/fact create mode 100755 scripts/fact/manage_facts.php create mode 100644 src/applications/fact/daemon/PhabricatorFactDaemon.php create mode 100644 src/applications/fact/engine/PhabricatorFactCountEngine.php create mode 100644 src/applications/fact/engine/PhabricatorFactEngine.php rename src/applications/{facts/extract/PhabricatorFactsUpdateIterator.php => fact/extract/PhabricatorFactUpdateIterator.php} (97%) create mode 100644 src/applications/fact/management/PhabricatorFactManagementAnalyzeWorkflow.php create mode 100644 src/applications/fact/management/PhabricatorFactManagementDestroyWorkflow.php create mode 100644 src/applications/fact/management/PhabricatorFactManagementListWorkflow.php create mode 100644 src/applications/fact/management/PhabricatorFactManagementStatusWorkflow.php create mode 100644 src/applications/fact/management/PhabricatorFactManagementWorkflow.php create mode 100644 src/applications/fact/storage/PhabricatorFactDAO.php create mode 100644 src/applications/fact/storage/PhabricatorFactRaw.php diff --git a/bin/fact b/bin/fact new file mode 120000 index 0000000000..9a84ebad20 --- /dev/null +++ b/bin/fact @@ -0,0 +1 @@ +../scripts/fact/manage_facts.php \ No newline at end of file diff --git a/scripts/fact/manage_facts.php b/scripts/fact/manage_facts.php new file mode 100755 index 0000000000..fd075896d3 --- /dev/null +++ b/scripts/fact/manage_facts.php @@ -0,0 +1,42 @@ +#!/usr/bin/env php +setTagline('manage fact configuration'); +$args->setSynopsis(<<parseStandardArguments(); + +$workflows = array( + new PhabricatorFactManagementDestroyWorkflow(), + new PhabricatorFactManagementAnalyzeWorkflow(), + new PhabricatorFactManagementStatusWorkflow(), + new PhabricatorFactManagementListWorkflow(), + new PhutilHelpArgumentWorkflow(), +); + +$args->parseWorkflows($workflows); diff --git a/src/__phutil_library_map__.php b/src/__phutil_library_map__.php index b72e2107eb..97cb7f8a83 100644 --- a/src/__phutil_library_map__.php +++ b/src/__phutil_library_map__.php @@ -624,7 +624,17 @@ phutil_register_library_map(array( 'PhabricatorEventEngine' => 'infrastructure/events/PhabricatorEventEngine.php', 'PhabricatorEventType' => 'infrastructure/events/constant/PhabricatorEventType.php', 'PhabricatorExampleEventListener' => 'infrastructure/events/PhabricatorExampleEventListener.php', - 'PhabricatorFactsUpdateIterator' => 'applications/facts/extract/PhabricatorFactsUpdateIterator.php', + 'PhabricatorFactCountEngine' => 'applications/fact/engine/PhabricatorFactCountEngine.php', + 'PhabricatorFactDAO' => 'applications/fact/storage/PhabricatorFactDAO.php', + 'PhabricatorFactDaemon' => 'applications/fact/daemon/PhabricatorFactDaemon.php', + 'PhabricatorFactEngine' => 'applications/fact/engine/PhabricatorFactEngine.php', + 'PhabricatorFactManagementAnalyzeWorkflow' => 'applications/fact/management/PhabricatorFactManagementAnalyzeWorkflow.php', + 'PhabricatorFactManagementDestroyWorkflow' => 'applications/fact/management/PhabricatorFactManagementDestroyWorkflow.php', + 'PhabricatorFactManagementListWorkflow' => 'applications/fact/management/PhabricatorFactManagementListWorkflow.php', + 'PhabricatorFactManagementStatusWorkflow' => 'applications/fact/management/PhabricatorFactManagementStatusWorkflow.php', + 'PhabricatorFactManagementWorkflow' => 'applications/fact/management/PhabricatorFactManagementWorkflow.php', + 'PhabricatorFactRaw' => 'applications/fact/storage/PhabricatorFactRaw.php', + 'PhabricatorFactUpdateIterator' => 'applications/fact/extract/PhabricatorFactUpdateIterator.php', 'PhabricatorFeedBuilder' => 'applications/feed/builder/PhabricatorFeedBuilder.php', 'PhabricatorFeedConstants' => 'applications/feed/constants/PhabricatorFeedConstants.php', 'PhabricatorFeedController' => 'applications/feed/controller/PhabricatorFeedController.php', @@ -1650,7 +1660,16 @@ phutil_register_library_map(array( 'PhabricatorEvent' => 'PhutilEvent', 'PhabricatorEventType' => 'PhutilEventType', 'PhabricatorExampleEventListener' => 'PhutilEventListener', - 'PhabricatorFactsUpdateIterator' => 'PhutilBufferedIterator', + 'PhabricatorFactCountEngine' => 'PhabricatorFactEngine', + 'PhabricatorFactDAO' => 'PhabricatorLiskDAO', + 'PhabricatorFactDaemon' => 'PhabricatorDaemon', + 'PhabricatorFactManagementAnalyzeWorkflow' => 'PhabricatorFactManagementWorkflow', + 'PhabricatorFactManagementDestroyWorkflow' => 'PhabricatorFactManagementWorkflow', + 'PhabricatorFactManagementListWorkflow' => 'PhabricatorFactManagementWorkflow', + 'PhabricatorFactManagementStatusWorkflow' => 'PhabricatorFactManagementWorkflow', + 'PhabricatorFactManagementWorkflow' => 'PhutilArgumentWorkflow', + 'PhabricatorFactRaw' => 'PhabricatorFactDAO', + 'PhabricatorFactUpdateIterator' => 'PhutilBufferedIterator', 'PhabricatorFeedController' => 'PhabricatorController', 'PhabricatorFeedDAO' => 'PhabricatorLiskDAO', 'PhabricatorFeedPublicStreamController' => 'PhabricatorFeedController', diff --git a/src/applications/fact/daemon/PhabricatorFactDaemon.php b/src/applications/fact/daemon/PhabricatorFactDaemon.php new file mode 100644 index 0000000000..4ef792e758 --- /dev/null +++ b/src/applications/fact/daemon/PhabricatorFactDaemon.php @@ -0,0 +1,121 @@ +engines = $engines; + return $this; + } + + public function processIterator($iterator) { + $result = null; + + $raw_facts = array(); + foreach ($iterator as $key => $object) { + $raw_facts[$object->getPHID()] = $this->computeRawFacts($object); + if (count($raw_facts) > self::RAW_FACT_BUFFER_LIMIT) { + $this->updateRawFacts($raw_facts); + $raw_facts = array(); + } + $result = $key; + } + + if ($raw_facts) { + $this->updateRawFacts($raw_facts); + $raw_facts = array(); + } + + return $result; + } + + private function computeRawFacts(PhabricatorLiskDAO $object) { + $facts = array(); + foreach ($this->engines as $engine) { + if (!$engine->shouldComputeRawFactsForObject($object)) { + continue; + } + $facts[] = $engine->computeRawFactsForObject($object); + } + + return array_mergev($facts); + } + + private function updateRawFacts(array $map) { + foreach ($map as $phid => $facts) { + assert_instances_of($facts, 'PhabricatorFactRaw'); + } + + $phids = array_keys($map); + if (!$phids) { + return; + } + + $table = new PhabricatorFactRaw(); + $conn = $table->establishConnection('w'); + $table_name = $table->getTableName(); + + $sql = array(); + foreach ($map as $phid => $facts) { + foreach ($facts as $fact) { + $sql[] = qsprintf( + $conn, + '(%s, %s, %s, %d, %d, %d)', + $fact->getFactType(), + $fact->getObjectPHID(), + $fact->getObjectA(), + $fact->getValueX(), + $fact->getValueY(), + $fact->getEpoch()); + } + } + + $table->openTransaction(); + + queryfx( + $conn, + 'DELETE FROM %T WHERE objectPHID IN (%Ls)', + $table_name, + $phids); + + if ($sql) { + foreach (array_chunk($sql, 256) as $chunk) { + queryfx( + $conn, + 'INSERT INTO %T + (factType, objectPHID, objectA, valueX, valueY, epoch) + VALUES %Q', + $table_name, + implode(', ', $chunk)); + } + } + + $table->saveTransaction(); + } + +} diff --git a/src/applications/fact/engine/PhabricatorFactCountEngine.php b/src/applications/fact/engine/PhabricatorFactCountEngine.php new file mode 100644 index 0000000000..d20df85ad8 --- /dev/null +++ b/src/applications/fact/engine/PhabricatorFactCountEngine.php @@ -0,0 +1,44 @@ +getPHID(); + $type = phid_get_type($phid); + + foreach (array('N:*', 'N:'.$type) as $fact_type) { + $facts[] = id(new PhabricatorFactRaw()) + ->setFactType($fact_type) + ->setObjectPHID($phid) + ->setEpoch($object->getDateCreated()); + } + + return $facts; + } + +} diff --git a/src/applications/fact/engine/PhabricatorFactEngine.php b/src/applications/fact/engine/PhabricatorFactEngine.php new file mode 100644 index 0000000000..cd951e913f --- /dev/null +++ b/src/applications/fact/engine/PhabricatorFactEngine.php @@ -0,0 +1,43 @@ +setAncestorClass(__CLASS__) + ->setConcreteOnly(true) + ->selectAndLoadSymbols(); + + $objects = array(); + foreach ($classes as $class) { + $objects[] = newv($class['name'], array()); + } + + return $objects; + } + + public function shouldComputeRawFactsForObject(PhabricatorLiskDAO $object) { + return false; + } + + public function computeRawFactsForObject(PhabricatorLiskDAO $object) { + return array(); + } + +} diff --git a/src/applications/facts/extract/PhabricatorFactsUpdateIterator.php b/src/applications/fact/extract/PhabricatorFactUpdateIterator.php similarity index 97% rename from src/applications/facts/extract/PhabricatorFactsUpdateIterator.php rename to src/applications/fact/extract/PhabricatorFactUpdateIterator.php index 19b1757396..b70a74a702 100644 --- a/src/applications/facts/extract/PhabricatorFactsUpdateIterator.php +++ b/src/applications/fact/extract/PhabricatorFactUpdateIterator.php @@ -21,7 +21,7 @@ * for "normal" Lisk objects: objects with an autoincrement ID and a * dateModified column. */ -final class PhabricatorFactsUpdateIterator extends PhutilBufferedIterator { +final class PhabricatorFactUpdateIterator extends PhutilBufferedIterator { private $cursor; private $object; diff --git a/src/applications/fact/management/PhabricatorFactManagementAnalyzeWorkflow.php b/src/applications/fact/management/PhabricatorFactManagementAnalyzeWorkflow.php new file mode 100644 index 0000000000..c864277741 --- /dev/null +++ b/src/applications/fact/management/PhabricatorFactManagementAnalyzeWorkflow.php @@ -0,0 +1,47 @@ +setName('analyze') + ->setSynopsis(pht('Manually invoke fact analyzers.')) + ->setArguments(array()); + } + + public function execute(PhutilArgumentParser $args) { + $console = PhutilConsole::getConsole(); + + $daemon = new PhabricatorFactDaemon(array()); + $daemon->setVerbose(true); + $daemon->setEngines(PhabricatorFactEngine::loadAllEngines()); + + $iterators = array( + new PhabricatorFactUpdateIterator(new DifferentialRevision()), + ); + + foreach ($iterators as $iterator) { + $daemon->processIterator($iterator); + } + + return 0; + } + +} diff --git a/src/applications/fact/management/PhabricatorFactManagementDestroyWorkflow.php b/src/applications/fact/management/PhabricatorFactManagementDestroyWorkflow.php new file mode 100644 index 0000000000..686c12fbbd --- /dev/null +++ b/src/applications/fact/management/PhabricatorFactManagementDestroyWorkflow.php @@ -0,0 +1,58 @@ +setName('destroy') + ->setSynopsis(pht('Destroy all facts.')) + ->setArguments(array()); + } + + public function execute(PhutilArgumentParser $args) { + $console = PhutilConsole::getConsole(); + + $question = pht( + 'Really destroy all facts? They will need to be rebuilt through '. + 'analysis, which may take some time.'); + + $ok = $console->confirm($question, $default = false); + if (!$ok) { + return 1; + } + + $tables = array(); + $tables[] = new PhabricatorFactRaw(); + foreach ($tables as $table) { + $conn = $table->establishConnection('w'); + $name = $table->getTableName(); + + $console->writeOut("%s\n", pht("Destroying table '%s'...", $name)); + + queryfx( + $conn, + 'TRUNCATE TABLE %T', + $name); + } + + $console->writeOut("%s\n", pht('Done.')); + } + +} diff --git a/src/applications/fact/management/PhabricatorFactManagementListWorkflow.php b/src/applications/fact/management/PhabricatorFactManagementListWorkflow.php new file mode 100644 index 0000000000..b98a19cbc9 --- /dev/null +++ b/src/applications/fact/management/PhabricatorFactManagementListWorkflow.php @@ -0,0 +1,40 @@ +setName('list') + ->setSynopsis(pht('Show a list of fact engines.')) + ->setArguments(array()); + } + + public function execute(PhutilArgumentParser $args) { + $console = PhutilConsole::getConsole(); + + $engines = PhabricatorFactEngine::loadAllEngines(); + foreach ($engines as $engine) { + $console->writeOut("%s\n", get_class($engine)); + } + + return 0; + } + +} diff --git a/src/applications/fact/management/PhabricatorFactManagementStatusWorkflow.php b/src/applications/fact/management/PhabricatorFactManagementStatusWorkflow.php new file mode 100644 index 0000000000..114484f244 --- /dev/null +++ b/src/applications/fact/management/PhabricatorFactManagementStatusWorkflow.php @@ -0,0 +1,59 @@ +setName('status') + ->setSynopsis(pht('Show status of fact data.')) + ->setArguments(array()); + } + + public function execute(PhutilArgumentParser $args) { + $console = PhutilConsole::getConsole(); + + $map = array( + 'raw' => new PhabricatorFactRaw(), + ); + + foreach ($map as $type => $table) { + $conn = $table->establishConnection('r'); + $name = $table->getTableName(); + + $row = queryfx_one( + $conn, + 'SELECT COUNT(*) N FROM %T', + $name); + + $n = $row['N']; + + switch ($type) { + case 'raw': + $desc = pht('There are %d raw fact(s) in storage.', $n); + break; + } + + $console->writeOut("%s\n", $desc); + } + + return 0; + } + +} diff --git a/src/applications/fact/management/PhabricatorFactManagementWorkflow.php b/src/applications/fact/management/PhabricatorFactManagementWorkflow.php new file mode 100644 index 0000000000..5400a4ff0a --- /dev/null +++ b/src/applications/fact/management/PhabricatorFactManagementWorkflow.php @@ -0,0 +1,26 @@ + false, + ) + parent::getConfiguration(); + } + +} diff --git a/src/applications/fact/storage/PhabricatorFactRaw.php b/src/applications/fact/storage/PhabricatorFactRaw.php new file mode 100644 index 0000000000..446d8f5afc --- /dev/null +++ b/src/applications/fact/storage/PhabricatorFactRaw.php @@ -0,0 +1,32 @@ + 'changed revisions, added %3$s; removed %5$s', + 'There are %d raw fact(s) in storage.' => array( + 'There is %d raw fact in storage.', + 'There are %d raw facts in storage.', + ), + ); }