Add "Autoplans" to Harbormaster

Summary:
Ref T8095. Two general problems:

  - I want Harbormaster to own all lint and unit test results.
  - I don't want users to have to configure anything for `arc` to keep working automatically.

These are in conflict because generic lint/unit test ownership in Harbormaster requires that build targets exist which we can attach build results to. However, we can't currently create build targets on demand: Harbormaster assumes it is responsible for creating targets, then running code or making third-party service calls to actually run the builds.

I considered two broad approaches to let `arc` push results into Harbormaster without requiring administrators to configure some kind of "arc results" build plan:

  # Add magic target PHIDs like `PHID-MAGIC-this-is-really-arc-unit`.
  # Add new code to build real targets with real PHIDs.

(1) is probably a bit less work to get off the ground, but I think it's worse overall and very likely to create more problems in the long run. I particularly worry that it will lead to a small amount of special casing in a very large number of places, which seems more fragile.

(2) is more work upfront but I think does a better job of putting all the special casing in one place that we can, e.g., more reasonably unit test, and letting the rest of the code rarely/never care about this case since it's just dealing with normal plans/steps/targets as far as it can tell.

This diff introduces "autoplans", which are source templates for plans/steps. This let us "push" these targets into Harbormaster. Hypthetically, any process "like" arc can use autoplans to upload test/lint/etc results. In practice, probably only `arc` will ever use this, but I think it's still quite a bit cleaner than the alternative despite all the generality.

Workflow is basically:

  - `arc` creates a diff.
  - `arc` calls `harbormaster.queryautotargets`, passing the diff PHID and saying "I have some lint and unit results I want to stick on this thing".
  - Harbormaster builds the plan, steps, and targets (if any of them don't already exist), and hands back the target PHIDs so `arc` has a completely standard-looking place to put results.
  - `arc` uploads the test results to the right targets, as though Harbormaster had asked it to run unit/lint in the first place.

(This doesn't actually do any of that yet, just sets things up.)

I'll maybe doc turn that ^^^^^^ into a doc for posterity since I think it's hard to guess what an "autotarget" is, but I'm going to grab some lunch first.

Test Plan:
  - Added unit tests to make sure we can build these things properly.
  - Used `harbormaster.queryautotargets` to build autotargets for a bunch of diffs.
  - Verified targets come up in "waiting for message" state.
  - Verified plans and steps are not editable.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: hach-que, epriestley

Maniphest Tasks: T8095

Differential Revision: https://secure.phabricator.com/D13345

resources/sql/autopatches/20150618.harbor.1.planauto.sql

@@ -0,0 +1,2 @@
+ALTER TABLE {$NAMESPACE}_harbormaster.harbormaster_buildplan
+  ADD planAutoKey VARCHAR(32) COLLATE {$COLLATE_TEXT};

resources/sql/autopatches/20150618.harbor.2.stepauto.sql

@@ -0,0 +1,2 @@
+ALTER TABLE {$NAMESPACE}_harbormaster.harbormaster_buildstep
+  ADD stepAutoKey VARCHAR(32) COLLATE {$COLLATE_TEXT};

resources/sql/autopatches/20150618.harbor.3.buildauto.sql

@@ -0,0 +1,2 @@
+ALTER TABLE {$NAMESPACE}_harbormaster.harbormaster_build
+  ADD planAutoKey VARCHAR(32) COLLATE {$COLLATE_TEXT};

src/__phutil_library_map__.php

@@ -357,6 +357,7 @@ phutil_register_library_map(array(
     'DifferentialDiffTableOfContentsView' => 'applications/differential/view/DifferentialDiffTableOfContentsView.php',
     'DifferentialDiffTestCase' => 'applications/differential/storage/__tests__/DifferentialDiffTestCase.php',
     'DifferentialDiffTransaction' => 'applications/differential/storage/DifferentialDiffTransaction.php',
+    'DifferentialDiffTransactionQuery' => 'applications/differential/query/DifferentialDiffTransactionQuery.php',
     'DifferentialDiffViewController' => 'applications/differential/controller/DifferentialDiffViewController.php',
     'DifferentialDoorkeeperRevisionFeedStoryPublisher' => 'applications/differential/doorkeeper/DifferentialDoorkeeperRevisionFeedStoryPublisher.php',
     'DifferentialDraft' => 'applications/differential/storage/DifferentialDraft.php',
@@ -823,11 +824,16 @@ phutil_register_library_map(array(
     'FundInitiativeTransactionQuery' => 'applications/fund/query/FundInitiativeTransactionQuery.php',
     'FundInitiativeViewController' => 'applications/fund/controller/FundInitiativeViewController.php',
     'FundSchemaSpec' => 'applications/fund/storage/FundSchemaSpec.php',
+    'HarbormasterArcLintBuildStepImplementation' => 'applications/harbormaster/step/HarbormasterArcLintBuildStepImplementation.php',
+    'HarbormasterArcUnitBuildStepImplementation' => 'applications/harbormaster/step/HarbormasterArcUnitBuildStepImplementation.php',
+    'HarbormasterAutotargetsTestCase' => 'applications/harbormaster/__tests__/HarbormasterAutotargetsTestCase.php',
     'HarbormasterBuild' => 'applications/harbormaster/storage/build/HarbormasterBuild.php',
     'HarbormasterBuildAbortedException' => 'applications/harbormaster/exception/HarbormasterBuildAbortedException.php',
     'HarbormasterBuildActionController' => 'applications/harbormaster/controller/HarbormasterBuildActionController.php',
+    'HarbormasterBuildArcanistAutoplan' => 'applications/harbormaster/autoplan/HarbormasterBuildArcanistAutoplan.php',
     'HarbormasterBuildArtifact' => 'applications/harbormaster/storage/build/HarbormasterBuildArtifact.php',
     'HarbormasterBuildArtifactQuery' => 'applications/harbormaster/query/HarbormasterBuildArtifactQuery.php',
+    'HarbormasterBuildAutoplan' => 'applications/harbormaster/autoplan/HarbormasterBuildAutoplan.php',
     'HarbormasterBuildCommand' => 'applications/harbormaster/storage/HarbormasterBuildCommand.php',
     'HarbormasterBuildDependencyDatasource' => 'applications/harbormaster/typeahead/HarbormasterBuildDependencyDatasource.php',
     'HarbormasterBuildEngine' => 'applications/harbormaster/engine/HarbormasterBuildEngine.php',
@@ -897,6 +903,7 @@ phutil_register_library_map(array(
     'HarbormasterPlanRunController' => 'applications/harbormaster/controller/HarbormasterPlanRunController.php',
     'HarbormasterPlanViewController' => 'applications/harbormaster/controller/HarbormasterPlanViewController.php',
     'HarbormasterPublishFragmentBuildStepImplementation' => 'applications/harbormaster/step/HarbormasterPublishFragmentBuildStepImplementation.php',
+    'HarbormasterQueryAutotargetsConduitAPIMethod' => 'applications/harbormaster/conduit/HarbormasterQueryAutotargetsConduitAPIMethod.php',
     'HarbormasterQueryBuildablesConduitAPIMethod' => 'applications/harbormaster/conduit/HarbormasterQueryBuildablesConduitAPIMethod.php',
     'HarbormasterQueryBuildsConduitAPIMethod' => 'applications/harbormaster/conduit/HarbormasterQueryBuildsConduitAPIMethod.php',
     'HarbormasterRemarkupRule' => 'applications/harbormaster/remarkup/HarbormasterRemarkupRule.php',
@@ -907,6 +914,7 @@ phutil_register_library_map(array(
     'HarbormasterStepAddController' => 'applications/harbormaster/controller/HarbormasterStepAddController.php',
     'HarbormasterStepDeleteController' => 'applications/harbormaster/controller/HarbormasterStepDeleteController.php',
     'HarbormasterStepEditController' => 'applications/harbormaster/controller/HarbormasterStepEditController.php',
+    'HarbormasterTargetEngine' => 'applications/harbormaster/engine/HarbormasterTargetEngine.php',
     'HarbormasterTargetWorker' => 'applications/harbormaster/worker/HarbormasterTargetWorker.php',
     'HarbormasterThrowExceptionBuildStep' => 'applications/harbormaster/step/HarbormasterThrowExceptionBuildStep.php',
     'HarbormasterUIEventListener' => 'applications/harbormaster/event/HarbormasterUIEventListener.php',
@@ -3704,6 +3712,7 @@ phutil_register_library_map(array(
     'DifferentialDiffTableOfContentsView' => 'AphrontView',
     'DifferentialDiffTestCase' => 'PhutilTestCase',
     'DifferentialDiffTransaction' => 'PhabricatorApplicationTransaction',
+    'DifferentialDiffTransactionQuery' => 'PhabricatorApplicationTransactionQuery',
     'DifferentialDiffViewController' => 'DifferentialController',
     'DifferentialDoorkeeperRevisionFeedStoryPublisher' => 'DoorkeeperFeedStoryPublisher',
     'DifferentialDraft' => 'DifferentialDAO',
@@ -4235,6 +4244,9 @@ phutil_register_library_map(array(
     'FundInitiativeTransactionQuery' => 'PhabricatorApplicationTransactionQuery',
     'FundInitiativeViewController' => 'FundController',
     'FundSchemaSpec' => 'PhabricatorConfigSchemaSpec',
+    'HarbormasterArcLintBuildStepImplementation' => 'HarbormasterBuildStepImplementation',
+    'HarbormasterArcUnitBuildStepImplementation' => 'HarbormasterBuildStepImplementation',
+    'HarbormasterAutotargetsTestCase' => 'PhabricatorTestCase',
     'HarbormasterBuild' => array(
       'HarbormasterDAO',
       'PhabricatorApplicationTransactionInterface',
@@ -4242,11 +4254,13 @@ phutil_register_library_map(array(
     ),
     'HarbormasterBuildAbortedException' => 'Exception',
     'HarbormasterBuildActionController' => 'HarbormasterController',
+    'HarbormasterBuildArcanistAutoplan' => 'HarbormasterBuildAutoplan',
     'HarbormasterBuildArtifact' => array(
       'HarbormasterDAO',
       'PhabricatorPolicyInterface',
     ),
     'HarbormasterBuildArtifactQuery' => 'PhabricatorCursorPagedPolicyAwareQuery',
+    'HarbormasterBuildAutoplan' => 'Phobject',
     'HarbormasterBuildCommand' => 'HarbormasterDAO',
     'HarbormasterBuildDependencyDatasource' => 'PhabricatorTypeaheadDatasource',
     'HarbormasterBuildEngine' => 'Phobject',
@@ -4342,6 +4356,7 @@ phutil_register_library_map(array(
     'HarbormasterPlanRunController' => 'HarbormasterController',
     'HarbormasterPlanViewController' => 'HarbormasterPlanController',
     'HarbormasterPublishFragmentBuildStepImplementation' => 'HarbormasterBuildStepImplementation',
+    'HarbormasterQueryAutotargetsConduitAPIMethod' => 'HarbormasterConduitAPIMethod',
     'HarbormasterQueryBuildablesConduitAPIMethod' => 'HarbormasterConduitAPIMethod',
     'HarbormasterQueryBuildsConduitAPIMethod' => 'HarbormasterConduitAPIMethod',
     'HarbormasterRemarkupRule' => 'PhabricatorObjectRemarkupRule',
@@ -4352,6 +4367,7 @@ phutil_register_library_map(array(
     'HarbormasterStepAddController' => 'HarbormasterController',
     'HarbormasterStepDeleteController' => 'HarbormasterController',
     'HarbormasterStepEditController' => 'HarbormasterController',
+    'HarbormasterTargetEngine' => 'Phobject',
     'HarbormasterTargetWorker' => 'HarbormasterWorker',
     'HarbormasterThrowExceptionBuildStep' => 'HarbormasterBuildStepImplementation',
     'HarbormasterUIEventListener' => 'PhabricatorEventListener',

src/applications/differential/query/DifferentialDiffTransactionQuery.php

@@ -0,0 +1,10 @@
+<?php
+
+final class DifferentialDiffTransactionQuery
+  extends PhabricatorApplicationTransactionQuery {
+
+  public function getTemplateApplicationTransaction() {
+    return new DifferentialDiffTransaction();
+  }
+
+}

src/applications/differential/storage/DifferentialDiff.php

@@ -381,6 +381,7 @@ final class DifferentialDiff
     $results = array();
 
     $results['buildable.diff'] = $this->getID();
+    if ($this->revisionID) {
       $revision = $this->getRevision();
       $results['buildable.revision'] = $revision->getID();
       $repo = $revision->getRepository();
@@ -390,6 +391,7 @@ final class DifferentialDiff
         $results['repository.vcs'] = $repo->getVersionControlSystem();
         $results['repository.uri'] = $repo->getPublicCloneURI();
       }
+    }
 
     return $results;
   }

src/applications/harbormaster/__tests__/HarbormasterAutotargetsTestCase.php

@@ -0,0 +1,61 @@
+<?php
+
+final class HarbormasterAutotargetsTestCase extends PhabricatorTestCase {
+
+  protected function getPhabricatorTestCaseConfiguration() {
+    return array(
+      self::PHABRICATOR_TESTCONFIG_BUILD_STORAGE_FIXTURES => true,
+    );
+  }
+
+  public function testGenerateHarbormasterAutotargets() {
+    $viewer = $this->generateNewTestUser();
+
+    $raw_diff = <<<EODIFF
+diff --git a/fruit b/fruit
+new file mode 100644
+index 0000000..1c0f49d
+--- /dev/null
++++ b/fruit
+@@ -0,0 +1,2 @@
++apal
++banan
+EODIFF;
+
+    $parser = new ArcanistDiffParser();
+    $changes = $parser->parseDiff($raw_diff);
+
+    $diff = DifferentialDiff::newFromRawChanges($viewer, $changes)
+      ->setLintStatus(DifferentialLintStatus::LINT_AUTO_SKIP)
+      ->setUnitStatus(DifferentialUnitStatus::UNIT_AUTO_SKIP)
+      ->attachRevision(null)
+      ->save();
+
+    $params = array(
+      'objectPHID' => $diff->getPHID(),
+      'targetKeys' => array(
+        HarbormasterArcLintBuildStepImplementation::STEPKEY,
+        HarbormasterArcUnitBuildStepImplementation::STEPKEY,
+      ),
+    );
+
+    // Creation of autotargets should work from an empty state.
+    $result = id(new ConduitCall('harbormaster.queryautotargets', $params))
+      ->setUser($viewer)
+      ->execute();
+
+    $targets = idx($result, 'targetMap');
+    foreach ($params['targetKeys'] as $target_key) {
+      $this->assertTrue((bool)$result['targetMap'][$target_key]);
+    }
+
+    // Querying the same autotargets again should produce the same results,
+    // not make new ones.
+    $retry = id(new ConduitCall('harbormaster.queryautotargets', $params))
+      ->setUser($viewer)
+      ->execute();
+
+    $this->assertEqual($result, $retry);
+  }
+
+}

src/applications/harbormaster/autoplan/HarbormasterBuildArcanistAutoplan.php

@@ -0,0 +1,16 @@
+<?php
+
+final class HarbormasterBuildArcanistAutoplan
+  extends HarbormasterBuildAutoplan {
+
+  const PLANKEY = 'arcanist';
+
+  public function getAutoplanPlanKey() {
+    return self::PLANKEY;
+  }
+
+  public function getAutoplanName() {
+    return pht('Arcanist Client Results');
+  }
+
+}

src/applications/harbormaster/autoplan/HarbormasterBuildAutoplan.php

@@ -0,0 +1,44 @@
+<?php
+
+abstract class HarbormasterBuildAutoplan extends Phobject {
+
+  abstract public function getAutoplanPlanKey();
+  abstract public function getAutoplanName();
+
+  public static function getAutoplan($key) {
+    return idx(self::getAllAutoplans(), $key);
+  }
+
+  public static function getAllAutoplans() {
+    static $plans;
+
+    if ($plans === null) {
+      $objects = id(new PhutilSymbolLoader())
+        ->setAncestorClass(__CLASS__)
+        ->loadObjects();
+
+      $map = array();
+      foreach ($objects as $object) {
+        $key = $object->getAutoplanPlanKey();
+        if (!empty($map[$key])) {
+          $other = $map[$key];
+          throw new Exception(
+            pht(
+              'Two build autoplans (of classes "%s" and "%s") define the same '.
+              'key ("%s"). Each autoplan must have a unique key.',
+              get_class($other),
+              get_class($object),
+              $key));
+        }
+        $map[$key] = $object;
+      }
+
+      ksort($map);
+
+      $plans = $map;
+    }
+
+    return $plans;
+  }
+
+}

src/applications/harbormaster/conduit/HarbormasterQueryAutotargetsConduitAPIMethod.php

@@ -0,0 +1,76 @@
+<?php
+
+final class HarbormasterQueryAutotargetsConduitAPIMethod
+  extends HarbormasterConduitAPIMethod {
+
+  public function getAPIMethodName() {
+    return 'harbormaster.queryautotargets';
+  }
+
+  public function getMethodDescription() {
+    return pht('Load or create build autotargets.');
+  }
+
+  protected function defineParamTypes() {
+    return array(
+      'objectPHID' => 'phid',
+      'targetKeys' => 'list<string>',
+    );
+  }
+
+  protected function defineReturnType() {
+    return 'map<string, phid>';
+  }
+
+  protected function execute(ConduitAPIRequest $request) {
+    $viewer = $request->getUser();
+
+    $phid = $request->getValue('objectPHID');
+
+    // NOTE: We use withNames() to let monograms like "D123" work, which makes
+    // this a little easier to test. Real PHIDs will still work as expected.
+
+    $object = id(new PhabricatorObjectQuery())
+      ->setViewer($viewer)
+      ->withNames(array($phid))
+      ->executeOne();
+    if (!$object) {
+      throw new Exception(
+        pht(
+          'No such object "%s" exists.',
+          $phid));
+    }
+
+    if (!($object instanceof HarbormasterBuildableInterface)) {
+      throw new Exception(
+        pht(
+          'Object "%s" does not implement interface "%s". Autotargets may '.
+          'only be queried for buildable objects.',
+          $phid,
+          'HarbormasterBuildableInterface'));
+    }
+
+    $autotargets = $request->getValue('targetKeys', array());
+
+    if ($autotargets) {
+      $targets = id(new HarbormasterTargetEngine())
+        ->setViewer($viewer)
+        ->setObject($object)
+        ->setAutoTargetKeys($autotargets)
+        ->buildTargets();
+    } else {
+      $targets = array();
+    }
+
+    // Reorder the results according to the request order so we can make test
+    // assertions that subsequent calls return the same results.
+
+    $map = mpull($targets, 'getPHID');
+    $map = array_select_keys($map, $autotargets);
+
+    return array(
+      'targetMap' => $map,
+    );
+  }
+
+}

src/applications/harbormaster/controller/HarbormasterStepAddController.php

@@ -24,10 +24,17 @@ final class HarbormasterStepAddController extends HarbormasterController {
     $plan_id = $plan->getID();
     $cancel_uri = $this->getApplicationURI("plan/{$plan_id}/");
 
+    $all = HarbormasterBuildStepImplementation::getImplementations();
+    foreach ($all as $key => $impl) {
+      if ($impl->shouldRequireAutotargeting()) {
+        unset($all[$key]);
+      }
+    }
+
     $errors = array();
     if ($request->isFormPost()) {
       $class = $request->getStr('class');
-      if (!HarbormasterBuildStepImplementation::getImplementation($class)) {
+      if (empty($all[$class])) {
         $errors[] = pht('Choose the type of build step you want to add.');
       }
       if (!$errors) {
@@ -39,7 +46,6 @@ final class HarbormasterStepAddController extends HarbormasterController {
     $control = id(new AphrontFormRadioButtonControl())
       ->setName('class');
 
-    $all = HarbormasterBuildStepImplementation::getImplementations();
     foreach ($all as $class => $implementation) {
       $control->addButton(
         $class,

src/applications/harbormaster/controller/HarbormasterStepEditController.php

@@ -47,6 +47,11 @@ final class HarbormasterStepEditController extends HarbormasterController {
         return new Aphront404Response();
       }
 
+      if ($impl->shouldRequireAutotargeting()) {
+        // No manual creation of autotarget steps.
+        return new Aphront404Response();
+      }
+
       $step = HarbormasterBuildStep::initializeNewStep($viewer)
         ->setBuildPlanPHID($plan->getPHID())
         ->setClassName($class);

src/applications/harbormaster/engine/HarbormasterTargetEngine.php

@@ -0,0 +1,251 @@
+<?php
+
+final class HarbormasterTargetEngine extends Phobject {
+
+  private $viewer;
+  private $object;
+  private $autoTargetKeys;
+
+  public function setViewer($viewer) {
+    $this->viewer = $viewer;
+    return $this;
+  }
+
+  public function getViewer() {
+    return $this->viewer;
+  }
+
+  public function setObject(HarbormasterBuildableInterface $object) {
+    $this->object = $object;
+    return $this;
+  }
+
+  public function getObject() {
+    return $this->object;
+  }
+
+  public function setAutoTargetKeys(array $auto_keys) {
+    $this->autoTargetKeys = $auto_keys;
+    return $this;
+  }
+
+  public function getAutoTargetKeys() {
+    return $this->autoTargetKeys;
+  }
+
+  public function buildTargets() {
+    $object = $this->getObject();
+    $viewer = $this->getViewer();
+
+    $step_map = $this->generateBuildStepMap($this->getAutoTargetKeys());
+
+    $buildable = HarbormasterBuildable::createOrLoadExisting(
+      $viewer,
+      $object->getHarbormasterBuildablePHID(),
+      $object->getHarbormasterContainerPHID());
+
+    $target_map = $this->generateBuildTargetMap($buildable, $step_map);
+
+    return $target_map;
+  }
+
+
+  /**
+   * Get a map of the @{class:HarbormasterBuildStep} objects for a list of
+   * autotarget keys.
+   *
+   * This method creates the steps if they do not yet exist.
+   *
+   * @param list<string> Autotarget keys, like `"core.arc.lint"`.
+   * @return map<string, object> Map of keys to step objects.
+   */
+  private function generateBuildStepMap(array $autotargets) {
+    $viewer = $this->getViewer();
+
+    $autosteps = $this->getAutosteps($autotargets);
+    $autosteps = mgroup($autosteps, 'getBuildStepAutotargetPlanKey');
+
+    $plans = id(new HarbormasterBuildPlanQuery())
+      ->setViewer($viewer)
+      ->withPlanAutoKeys(array_keys($autosteps))
+      ->needBuildSteps(true)
+      ->execute();
+    $plans = mpull($plans, null, 'getPlanAutoKey');
+
+    // NOTE: When creating the plan and steps, we save the autokeys as the
+    // names. These won't actually be shown in the UI, but make the data more
+    // consistent for secondary consumers like typeaheads.
+
+    $step_map = array();
+    foreach ($autosteps as $plan_key => $steps) {
+      $plan = idx($plans, $plan_key);
+      if (!$plan) {
+        $plan = HarbormasterBuildPlan::initializeNewBuildPlan($viewer)
+          ->setName($plan_key)
+          ->setPlanAutoKey($plan_key);
+      }
+
+      $current = $plan->getBuildSteps();
+      $current = mpull($current, null, 'getStepAutoKey');
+      $new_steps = array();
+
+      foreach ($steps as $step_key => $step) {
+        if (isset($current[$step_key])) {
+          $step_map[$step_key] = $current[$step_key];
+          continue;
+        }
+
+        $new_step = HarbormasterBuildStep::initializeNewStep($viewer)
+          ->setName($step_key)
+          ->setClassName(get_class($step))
+          ->setStepAutoKey($step_key);
+
+        $new_steps[$step_key] = $new_step;
+      }
+
+      if ($new_steps) {
+        $plan->openTransaction();
+          if (!$plan->getPHID()) {
+            $plan->save();
+          }
+          foreach ($new_steps as $step_key => $step) {
+            $step->setBuildPlanPHID($plan->getPHID());
+            $step->save();
+
+            $step->attachBuildPlan($plan);
+            $step_map[$step_key] = $step;
+          }
+        $plan->saveTransaction();
+      }
+    }
+
+    return $step_map;
+  }
+
+
+  /**
+   * Get all of the @{class:HarbormasterBuildStepImplementation} objects for
+   * a list of autotarget keys.
+   *
+   * @param list<string> Autotarget keys, like `"core.arc.lint"`.
+   * @return map<string, object> Map of keys to implementations.
+   */
+  private function getAutosteps(array $autotargets) {
+    $all_steps = HarbormasterBuildStepImplementation::getImplementations();
+    $all_steps = mpull($all_steps, null, 'getBuildStepAutotargetStepKey');
+
+    // Make sure all the targets really exist.
+    foreach ($autotargets as $autotarget) {
+      if (empty($all_steps[$autotarget])) {
+        throw new Exception(
+          pht(
+            'No build step provides autotarget "%s"!',
+            $autotarget));
+      }
+    }
+
+    return array_select_keys($all_steps, $autotargets);
+  }
+
+
+  /**
+   * Get a list of @{class:HarbormasterBuildTarget} objects for a list of
+   * autotarget keys.
+   *
+   * If some targets or builds do not exist, they are created.
+   *
+   * @param HarbormasterBuildable A buildable.
+   * @param map<string, object> Map of keys to steps.
+   * @return map<string, object> Map of keys to targets.
+   */
+  private function generateBuildTargetMap(
+    HarbormasterBuildable $buildable,
+    array $step_map) {
+
+    $viewer = $this->getViewer();
+    $plan_map = mgroup($step_map, 'getBuildPlanPHID');
+
+    $builds = id(new HarbormasterBuildQuery())
+      ->setViewer($viewer)
+      ->withBuildablePHIDs(array($buildable->getPHID()))
+      ->withBuildPlanPHIDs(array_keys($plan_map))
+      ->needBuildTargets(true)
+      ->execute();
+
+    $autobuilds = array();
+    foreach ($builds as $build) {
+      $plan_key = $build->getBuildPlan()->getPlanAutoKey();
+      $autobuilds[$plan_key] = $build;
+    }
+
+    $new_builds = array();
+    foreach ($plan_map as $plan_phid => $steps) {
+      $plan = head($steps)->getBuildPlan();
+      $plan_key = $plan->getPlanAutoKey();
+
+      $build = idx($autobuilds, $plan_key);
+      if ($build) {
+        // We already have a build for this set of targets, so we don't need
+        // to do any work. (It's possible the build is an older build that
+        // doesn't have all of the right targets if new autotargets were
+        // recently introduced, but we don't currently try to construct them.)
+        continue;
+      }
+
+      // NOTE: Normally, `applyPlan()` does not actually generate targets.
+      // We need to apply the plan in-process to perform target generation.
+      // This is fine as long as autotargets are empty containers that don't
+      // do any work, which they always should be.
+
+      PhabricatorWorker::setRunAllTasksInProcess(true);
+      try {
+
+        // NOTE: We might race another process here to create the same build
+        // with the same `planAutoKey`. The database will prevent this and
+        // using autotargets only currently makes sense if you just created the
+        // resource and "own" it, so we don't try to handle this, but may need
+        // to be more careful here if use of autotargets expands.
+
+        $build = $buildable->applyPlan($plan);
+        PhabricatorWorker::setRunAllTasksInProcess(false);
+      } catch (Exception $ex) {
+        PhabricatorWorker::setRunAllTasksInProcess(false);
+        throw $ex;
+      }
+
+      $new_builds[] = $build;
+    }
+
+    if ($new_builds) {
+      $all_targets = id(new HarbormasterBuildTargetQuery())
+        ->setViewer($viewer)
+        ->withBuildPHIDs(mpull($new_builds, 'getPHID'))
+        ->execute();
+    } else {
+      $all_targets = array();
+    }
+
+    foreach ($builds as $build) {
+      foreach ($build->getBuildTargets() as $target) {
+        $all_targets[] = $target;
+      }
+    }
+
+    $target_map = array();
+    foreach ($all_targets as $target) {
+      $target_key = $target
+        ->getImplementation()
+        ->getBuildStepAutotargetStepKey();
+      if (!$target_key) {
+        continue;
+      }
+      $target_map[$target_key] = $target;
+    }
+
+    $target_map = array_select_keys($target_map, array_keys($step_map));
+
+    return $target_map;
+  }
+
+
+}

src/applications/harbormaster/query/HarbormasterBuildPlanQuery.php

@@ -7,6 +7,8 @@ final class HarbormasterBuildPlanQuery
   private $phids;
   private $statuses;
   private $datasourceQuery;
+  private $planAutoKeys;
+  private $needBuildSteps;
 
   public function withIDs(array $ids) {
     $this->ids = $ids;
@@ -28,6 +30,16 @@ final class HarbormasterBuildPlanQuery
     return $this;
   }
 
+  public function withPlanAutoKeys(array $keys) {
+    $this->planAutoKeys = $keys;
+    return $this;
+  }
+
+  public function needBuildSteps($need) {
+    $this->needBuildSteps = $need;
+    return $this;
+  }
+
   public function newResultObject() {
     return new HarbormasterBuildPlan();
   }
@@ -36,6 +48,26 @@ final class HarbormasterBuildPlanQuery
     return $this->loadStandardPage($this->newResultObject());
   }
 
+  protected function didFilterPage(array $page) {
+    if ($this->needBuildSteps) {
+      $plan_phids = mpull($page, 'getPHID');
+
+      $steps = id(new HarbormasterBuildStepQuery())
+        ->setParentQuery($this)
+        ->setViewer($this->getViewer())
+        ->withBuildPlanPHIDs($plan_phids)
+        ->execute();
+      $steps = mgroup($steps, 'getBuildPlanPHID');
+
+      foreach ($page as $plan) {
+        $plan_steps = idx($steps, $plan->getPHID(), array());
+        $plan->attachBuildSteps($plan_steps);
+      }
+    }
+
+    return $page;
+  }
+
   protected function buildWhereClauseParts(AphrontDatabaseConnection $conn) {
     $where = parent::buildWhereClauseParts($conn);
 
@@ -67,6 +99,13 @@ final class HarbormasterBuildPlanQuery
         $this->datasourceQuery);
     }
 
+    if ($this->planAutoKeys !== null) {
+      $where[] = qsprintf(
+        $conn,
+        'planAutoKey IN (%Ls)',
+        $this->planAutoKeys);
+    }
+
     return $where;
   }
 

src/applications/harbormaster/query/HarbormasterBuildPlanSearchEngine.php

@@ -88,6 +88,10 @@ final class HarbormasterBuildPlanSearchEngine
         $item->setDisabled(true);
       }
 
+      if ($plan->isAutoplan()) {
+        $item->addIcon('fa-lock grey', pht('Autoplan'));
+      }
+
       $item->setHref($this->getApplicationURI("plan/{$id}/"));
 
       $list->addItem($item);

src/applications/harbormaster/step/HarbormasterArcLintBuildStepImplementation.php

@@ -0,0 +1,38 @@
+<?php
+
+final class HarbormasterArcLintBuildStepImplementation
+  extends HarbormasterBuildStepImplementation {
+
+  const STEPKEY = 'arcanist.lint';
+
+  public function getBuildStepAutotargetPlanKey() {
+    return HarbormasterBuildArcanistAutoplan::PLANKEY;
+  }
+
+  public function getBuildStepAutotargetStepKey() {
+    return self::STEPKEY;
+  }
+
+  public function shouldRequireAutotargeting() {
+    return true;
+  }
+
+  public function getName() {
+    return pht('Arcanist Lint Results');
+  }
+
+  public function getGenericDescription() {
+    return pht('Automatic `arc lint` step.');
+  }
+
+  public function execute(
+    HarbormasterBuild $build,
+    HarbormasterBuildTarget $build_target) {
+    return;
+  }
+
+  public function shouldWaitForMessage(HarbormasterBuildTarget $target) {
+    return true;
+  }
+
+}

src/applications/harbormaster/step/HarbormasterArcUnitBuildStepImplementation.php

@@ -0,0 +1,38 @@
+<?php
+
+final class HarbormasterArcUnitBuildStepImplementation
+  extends HarbormasterBuildStepImplementation {
+
+  const STEPKEY = 'arcanist.unit';
+
+  public function getBuildStepAutotargetPlanKey() {
+    return HarbormasterBuildArcanistAutoplan::PLANKEY;
+  }
+
+  public function getBuildStepAutotargetStepKey() {
+    return self::STEPKEY;
+  }
+
+  public function shouldRequireAutotargeting() {
+    return true;
+  }
+
+  public function getName() {
+    return pht('Arcanist Unit Results');
+  }
+
+  public function getGenericDescription() {
+    return pht('Automatic `arc unit` step.');
+  }
+
+  public function execute(
+    HarbormasterBuild $build,
+    HarbormasterBuildTarget $build_target) {
+    return;
+  }
+
+  public function shouldWaitForMessage(HarbormasterBuildTarget $target) {
+    return true;
+  }
+
+}

src/applications/harbormaster/step/HarbormasterBuildStepImplementation.php

@@ -1,5 +1,8 @@
 <?php
 
+/**
+ * @task autotarget Automatic Targets
+ */
 abstract class HarbormasterBuildStepImplementation extends Phobject {
 
   private $settings;
@@ -249,4 +252,20 @@ abstract class HarbormasterBuildStepImplementation extends Phobject {
     }
   }
 
+
+/* -(  Automatic Targets  )-------------------------------------------------- */
+
+
+  public function getBuildStepAutotargetStepKey() {
+    return null;
+  }
+
+  public function getBuildStepAutotargetPlanKey() {
+    throw new PhutilMethodNotImplementedException();
+  }
+
+  public function shouldRequireAutotargeting() {
+    return false;
+  }
+
 }

src/applications/harbormaster/storage/HarbormasterBuildable.php

@@ -141,8 +141,14 @@ final class HarbormasterBuildable extends HarbormasterDAO
     $build = HarbormasterBuild::initializeNewBuild($viewer)
       ->setBuildablePHID($this->getPHID())
       ->setBuildPlanPHID($plan->getPHID())
-      ->setBuildStatus(HarbormasterBuild::STATUS_PENDING)
-      ->save();
+      ->setBuildStatus(HarbormasterBuild::STATUS_PENDING);
+
+    $auto_key = $plan->getPlanAutoKey();
+    if ($auto_key) {
+      $build->setPlanAutoKey($auto_key);
+    }
+
+    $build->save();
 
     PhabricatorWorker::scheduleTask(
       'HarbormasterBuildWorker',

src/applications/harbormaster/storage/build/HarbormasterBuild.php

@@ -9,6 +9,7 @@ final class HarbormasterBuild extends HarbormasterDAO
   protected $buildPlanPHID;
   protected $buildStatus;
   protected $buildGeneration;
+  protected $planAutoKey;
 
   private $buildable = self::ATTACHABLE;
   private $buildPlan = self::ATTACHABLE;
@@ -148,6 +149,7 @@ final class HarbormasterBuild extends HarbormasterDAO
       self::CONFIG_COLUMN_SCHEMA => array(
         'buildStatus' => 'text32',
         'buildGeneration' => 'uint32',
+        'planAutoKey' => 'text32?',
       ),
       self::CONFIG_KEY_SCHEMA => array(
         'key_buildable' => array(
@@ -159,6 +161,10 @@ final class HarbormasterBuild extends HarbormasterDAO
         'key_status' => array(
           'columns' => array('buildStatus'),
         ),
+        'key_planautokey' => array(
+          'columns' => array('buildablePHID', 'planAutoKey'),
+          'unique' => true,
+        ),
       ),
     ) + parent::getConfiguration();
   }

src/applications/harbormaster/storage/build/HarbormasterBuildTarget.php

@@ -175,8 +175,16 @@ final class HarbormasterBuildTarget extends HarbormasterDAO
     return $this->implementation;
   }
 
+  public function isAutotarget() {
+    try {
+      return (bool)$this->getImplementation()->getBuildStepAutotargetPlanKey();
+    } catch (Exception $e) {
+      return false;
+    }
+  }
+
   public function getName() {
-    if (strlen($this->name)) {
+    if (strlen($this->name) && !$this->isAutotarget()) {
       return $this->name;
     }
 

src/applications/harbormaster/storage/configuration/HarbormasterBuildPlan.php

@@ -1,5 +1,8 @@
 <?php
 
+/**
+ * @task autoplan Autoplans
+ */
 final class HarbormasterBuildPlan extends HarbormasterDAO
   implements
     PhabricatorApplicationTransactionInterface,
@@ -8,6 +11,7 @@ final class HarbormasterBuildPlan extends HarbormasterDAO
 
   protected $name;
   protected $planStatus;
+  protected $planAutoKey;
 
   const STATUS_ACTIVE   = 'active';
   const STATUS_DISABLED = 'disabled';
@@ -16,7 +20,9 @@ final class HarbormasterBuildPlan extends HarbormasterDAO
 
   public static function initializeNewBuildPlan(PhabricatorUser $actor) {
     return id(new HarbormasterBuildPlan())
-      ->setPlanStatus(self::STATUS_ACTIVE);
+      ->setName('')
+      ->setPlanStatus(self::STATUS_ACTIVE)
+      ->attachBuildSteps(array());
   }
 
   protected function getConfiguration() {
@@ -25,6 +31,7 @@ final class HarbormasterBuildPlan extends HarbormasterDAO
       self::CONFIG_COLUMN_SCHEMA => array(
         'name' => 'sort128',
         'planStatus' => 'text32',
+        'planAutoKey' => 'text32?',
       ),
       self::CONFIG_KEY_SCHEMA => array(
         'key_status' => array(
@@ -33,6 +40,10 @@ final class HarbormasterBuildPlan extends HarbormasterDAO
         'key_name' => array(
           'columns' => array('name'),
         ),
+        'key_planautokey' => array(
+          'columns' => array('planAutoKey'),
+          'unique' => true,
+        ),
       ),
     ) + parent::getConfiguration();
   }
@@ -57,6 +68,33 @@ final class HarbormasterBuildPlan extends HarbormasterDAO
   }
 
 
+/* -(  Autoplans  )---------------------------------------------------------- */
+
+
+  public function isAutoplan() {
+    return ($this->getPlanAutoKey() !== null);
+  }
+
+
+  public function getAutoplan() {
+    if (!$this->isAutoplan()) {
+      return null;
+    }
+
+    return HarbormasterBuildAutoplan::getAutoplan($this->getPlanAutoKey());
+  }
+
+
+  public function getName() {
+    $autoplan = $this->getAutoplan();
+    if ($autoplan) {
+      return $autoplan->getAutoplanName();
+    }
+
+    return parent::getName();
+  }
+
+
 /* -(  PhabricatorSubscribableInterface  )----------------------------------- */
 
 
@@ -113,6 +151,11 @@ final class HarbormasterBuildPlan extends HarbormasterDAO
       case PhabricatorPolicyCapability::CAN_EDIT:
         // NOTE: In practice, this policy is always limited by the "Mangage
         // Build Plans" policy.
+
+        if ($this->isAutoplan()) {
+          return PhabricatorPolicies::POLICY_NOONE;
+        }
+
         return PhabricatorPolicies::getMostOpenPolicy();
     }
   }
@@ -122,7 +165,19 @@ final class HarbormasterBuildPlan extends HarbormasterDAO
   }
 
   public function describeAutomaticCapability($capability) {
-    return null;
+    $messages = array();
+
+    switch ($capability) {
+      case PhabricatorPolicyCapability::CAN_EDIT:
+        if ($this->isAutoplan()) {
+          $messages[] = pht(
+            'This is an autoplan (a builtin plan provided by an application) '.
+            'so it can not be edited.');
+        }
+        break;
+    }
+
+    return $messages;
   }
 
 }

src/applications/harbormaster/storage/configuration/HarbormasterBuildStep.php

@@ -12,13 +12,16 @@ final class HarbormasterBuildStep extends HarbormasterDAO
   protected $className;
   protected $details = array();
   protected $sequence = 0;
+  protected $stepAutoKey;
 
   private $buildPlan = self::ATTACHABLE;
   private $customFields = self::ATTACHABLE;
   private $implementation;
 
   public static function initializeNewStep(PhabricatorUser $actor) {
-    return id(new HarbormasterBuildStep());
+    return id(new HarbormasterBuildStep())
+      ->setName('')
+      ->setDescription('');
   }
 
   protected function getConfiguration() {
@@ -37,11 +40,16 @@ final class HarbormasterBuildStep extends HarbormasterDAO
         // which predated editable names. These should be backfilled with
         // default names, then the code for handling `null` shoudl be removed.
         'name' => 'text255?',
+        'stepAutoKey' => 'text32?',
       ),
       self::CONFIG_KEY_SCHEMA => array(
         'key_plan' => array(
           'columns' => array('buildPlanPHID'),
         ),
+        'key_stepautokey' => array(
+          'columns' => array('buildPlanPHID', 'stepAutoKey'),
+          'unique' => true,
+        ),
       ),
     ) + parent::getConfiguration();
   }
@@ -88,6 +96,10 @@ final class HarbormasterBuildStep extends HarbormasterDAO
     return $this->implementation;
   }
 
+  public function isAutostep() {
+    return ($this->getStepAutoKey() !== null);
+  }
+
 
 /* -(  PhabricatorApplicationTransactionInterface  )------------------------- */