Implement a rough AlmanacService blueprint in Drydock

Summary:
Ref T9253. Broadly, this realigns Allocator behavior to be more consistent and straightforward and amenable to intended future changes.

This attempts to make language more consistent: resources are "allocated" and leases are "acquired".

This prepares for (but does not implement) optimistic "slot locking", as discussed in D10304. Although I suspect some blueprints will need to perform other locking eventually, this does feel like a good fit for most of the locking blueprints need to do.

In particular, I've made the blueprint operations on `$resource` and `$lease` objects more purposeful: they need to invoke an activator on the appropriate object to be implemented correctly. Before they invoke this activator method, they configure the object. In a future diff, this configuration will include specifying slot locks that the lease or resource must acquire. So the API will be something like:

  $lease
    ->setActivateWhenAcquired(true)
    ->needSlotLock('x')
    ->needSlotLock('y')
    ->acquireOnResource($resource);

In the common case where slot locks are a good fit, I think this should make correct blueprint implementation very straightforward.

This prepares for (but does not implement) resources and leases which need significant setup steps. I've basically carved out two modes:

  - The "activate immediately" mode, as here, immediately opens the resource or activates the lease. This is appropriate if little or no setup is required. I expect many leases to operate in this mode, although I expect many resources will operate in the other mode.
  - The "allocate now, activate later" mode, which is not fully implemented yet. This will queue setup workers when the allocator exits. Overall, this will work very similarly to Harbormaster.
  - This new structure makes it acceptable for blueprints to sleep as long as they want during resource allocation and lease acquisition, so long as they are not waiting on anything which needs to be completed by the queue. Putting a `sleep(15 * 60)` in your EC2Blueprint to wait for EC2 to bring a machine up will perform worse than using delayed activation, but won't deadlock the queue or block any locks.

Overall, this flow is more similar to Harbormaster's flow. Having consistency between Harbormaster's model and Drydock's model is good, and I think Harbormaster's model is also simply much better than Drydock's (what exists today in Drydock was implemented a long time ago, and we had more support and infrastructure by the time Harbormaster was implemented, as well as a more clearly defined problem).

The particular strength of Harbormaster is that objects always (or almost always, at least) have a single, clearly defined writer. Ensuring objects have only one writer prevents races and makes reasoning about everything easier.

Drydock does not currently have a clearly defined single writer, but this moves us in that direction. We'll probably need more primitives eventually to flesh this out, like Harbormaster's command queue for messaging objects which you can't write to.

This blueprint was originally implemented in D13843. This makes a few changes to the blueprint itself:

  - A bunch of code from that (e.g., interfaces) doesn't exist yet.
  - I let the blueprint have multiple services. This simplifies the code a little and seems like it costs us nothing.

This also removes `bin/drydock create-resource`, which no longer makes sense to expose. It won't get locking, leasing, etc., correct, and can not be made correct.

NOTE: This technically works but doesn't do anything useful yet.

Test Plan: Used `bin/drydock lease --type host` to acquire leases against these blueprints.

Reviewers: hach-que, chad

Reviewed By: hach-que, chad

Subscribers: Mnkras

Maniphest Tasks: T9253

Differential Revision: https://secure.phabricator.com/D14117

src/__phutil_library_map__.php

@@ -797,6 +797,7 @@ phutil_register_library_map(array(
     'DoorkeeperTagView' => 'applications/doorkeeper/view/DoorkeeperTagView.php',
     'DoorkeeperTagsController' => 'applications/doorkeeper/controller/DoorkeeperTagsController.php',
     'DrydockAllocatorWorker' => 'applications/drydock/worker/DrydockAllocatorWorker.php',
+    'DrydockAlmanacServiceHostBlueprintImplementation' => 'applications/drydock/blueprint/DrydockAlmanacServiceHostBlueprintImplementation.php',
     'DrydockApacheWebrootInterface' => 'applications/drydock/interface/webroot/DrydockApacheWebrootInterface.php',
     'DrydockBlueprint' => 'applications/drydock/storage/DrydockBlueprint.php',
     'DrydockBlueprintController' => 'applications/drydock/controller/DrydockBlueprintController.php',
@@ -845,7 +846,6 @@ phutil_register_library_map(array(
     'DrydockLogQuery' => 'applications/drydock/query/DrydockLogQuery.php',
     'DrydockLogSearchEngine' => 'applications/drydock/query/DrydockLogSearchEngine.php',
     'DrydockManagementCloseWorkflow' => 'applications/drydock/management/DrydockManagementCloseWorkflow.php',
-    'DrydockManagementCreateResourceWorkflow' => 'applications/drydock/management/DrydockManagementCreateResourceWorkflow.php',
     'DrydockManagementLeaseWorkflow' => 'applications/drydock/management/DrydockManagementLeaseWorkflow.php',
     'DrydockManagementReleaseWorkflow' => 'applications/drydock/management/DrydockManagementReleaseWorkflow.php',
     'DrydockManagementWorkflow' => 'applications/drydock/management/DrydockManagementWorkflow.php',
@@ -4502,6 +4502,7 @@ phutil_register_library_map(array(
     'DoorkeeperTagView' => 'AphrontView',
     'DoorkeeperTagsController' => 'PhabricatorController',
     'DrydockAllocatorWorker' => 'PhabricatorWorker',
+    'DrydockAlmanacServiceHostBlueprintImplementation' => 'DrydockBlueprintImplementation',
     'DrydockApacheWebrootInterface' => 'DrydockWebrootInterface',
     'DrydockBlueprint' => array(
       'DrydockDAO',
@@ -4564,7 +4565,6 @@ phutil_register_library_map(array(
     'DrydockLogQuery' => 'DrydockQuery',
     'DrydockLogSearchEngine' => 'PhabricatorApplicationSearchEngine',
     'DrydockManagementCloseWorkflow' => 'DrydockManagementWorkflow',
-    'DrydockManagementCreateResourceWorkflow' => 'DrydockManagementWorkflow',
     'DrydockManagementLeaseWorkflow' => 'DrydockManagementWorkflow',
     'DrydockManagementReleaseWorkflow' => 'DrydockManagementWorkflow',
     'DrydockManagementWorkflow' => 'PhabricatorManagementWorkflow',

src/applications/drydock/blueprint/DrydockAlmanacServiceHostBlueprintImplementation.php

@@ -0,0 +1,234 @@
+<?php
+
+final class DrydockAlmanacServiceHostBlueprintImplementation
+  extends DrydockBlueprintImplementation {
+
+  private $services;
+  private $freeBindings;
+
+  public function isEnabled() {
+    $almanac_app = 'PhabricatorAlmanacApplication';
+    return PhabricatorApplication::isClassInstalled($almanac_app);
+  }
+
+  public function getBlueprintName() {
+    return pht('Almanac Hosts');
+  }
+
+  public function getDescription() {
+    return pht(
+      'Allows Drydock to lease existing hosts defined in an Almanac service '.
+      'pool.');
+  }
+
+  public function canAnyBlueprintEverAllocateResourceForLease(
+    DrydockLease $lease) {
+    return true;
+  }
+
+  public function canEverAllocateResourceForLease(
+    DrydockBlueprint $blueprint,
+    DrydockLease $lease) {
+    $services = $this->loadServices($blueprint);
+    $bindings = $this->loadAllBindings($services);
+
+    if (!$bindings) {
+      // If there are no devices bound to the services for this blueprint,
+      // we can not allocate resources.
+      return false;
+    }
+
+    return true;
+  }
+
+  public function canAllocateResourceForLease(
+    DrydockBlueprint $blueprint,
+    DrydockLease $lease) {
+
+    // We will only allocate one resource per unique device bound to the
+    // services for this blueprint. Make sure we have a free device somewhere.
+    $free_bindings = $this->loadFreeBindings($blueprint);
+    if (!$free_bindings) {
+      return false;
+    }
+
+    return true;
+  }
+
+  public function allocateResource(
+    DrydockBlueprint $blueprint,
+    DrydockLease $lease) {
+
+    $free_bindings = $this->loadFreeBindings($blueprint);
+    shuffle($free_bindings);
+
+    $exceptions = array();
+    foreach ($free_bindings as $binding) {
+      $device = $binding->getDevice();
+      $device_name = $device->getName();
+
+      $resource = $this->newResourceTemplate($blueprint, $device_name)
+        ->setActivateWhenAllocated(true)
+        ->setAttribute('almanacServicePHID', $binding->getServicePHID())
+        ->setAttribute('almanacBindingPHID', $binding->getPHID());
+
+      // TODO: This algorithm can race, and the "free" binding may not be
+      // free by the time we acquire it. Do slot-locking here if that works
+      // out, or some other kind of locking if it does not.
+
+      try {
+        return $resource->allocateResource(DrydockResourceStatus::STATUS_OPEN);
+      } catch (Exception $ex) {
+        $exceptions[] = $ex;
+      }
+    }
+
+    throw new PhutilAggregateException(
+      pht('Unable to allocate any binding as a resource.'),
+      $exceptions);
+  }
+
+  public function canAcquireLeaseOnResource(
+    DrydockBlueprint $blueprint,
+    DrydockResource $resource,
+    DrydockLease $lease) {
+
+    // TODO: We'll currently lease each resource an unlimited number of times,
+    // but should stop doing that.
+
+    return true;
+  }
+
+  public function acquireLease(
+    DrydockBlueprint $blueprint,
+    DrydockResource $resource,
+    DrydockLease $lease) {
+
+    // TODO: Once we have limit rules, we should perform slot locking (or other
+    // kinds of locking) here.
+
+    $lease
+      ->setActivateWhenAcquired(true)
+      ->acquireOnResource($resource);
+  }
+
+  public function getType() {
+    return 'host';
+  }
+
+  public function getInterface(
+    DrydockResource $resource,
+    DrydockLease $lease,
+    $type) {
+    // TODO: Actually do stuff here, this needs work and currently makes this
+    // entire exercise pointless.
+  }
+
+  public function getFieldSpecifications() {
+    return array(
+      'almanacServicePHIDs' => array(
+        'name' => pht('Almanac Services'),
+        'type' => 'datasource',
+        'datasource.class' => 'AlmanacServiceDatasource',
+        'datasource.parameters' => array(
+          'serviceClasses' => $this->getAlmanacServiceClasses(),
+        ),
+        'required' => true,
+      ),
+      'credentialPHID' => array(
+        'name' => pht('Credentials'),
+        'type' => 'credential',
+        'credential.provides' =>
+          PassphraseSSHPrivateKeyCredentialType::PROVIDES_TYPE,
+        'credential.type' =>
+          PassphraseSSHPrivateKeyTextCredentialType::CREDENTIAL_TYPE,
+      ),
+    ) + parent::getFieldSpecifications();
+  }
+
+  private function loadServices(DrydockBlueprint $blueprint) {
+    if (!$this->services) {
+      $service_phids = $blueprint->getFieldValue('almanacServicePHIDs');
+      if (!$service_phids) {
+        throw new Exception(
+          pht(
+            'This blueprint ("%s") does not define any Almanac Service PHIDs.',
+            $blueprint->getBlueprintName()));
+      }
+
+      $viewer = PhabricatorUser::getOmnipotentUser();
+      $services = id(new AlmanacServiceQuery())
+        ->setViewer($viewer)
+        ->withPHIDs($service_phids)
+        ->withServiceClasses($this->getAlmanacServiceClasses())
+        ->needBindings(true)
+        ->execute();
+      $services = mpull($services, null, 'getPHID');
+
+      if (count($services) != count($service_phids)) {
+        $missing_phids = array_diff($service_phids, array_keys($services));
+        throw new Exception(
+          pht(
+            'Some of the Almanac Services defined by this blueprint '.
+            'could not be loaded. They may be invalid, no longer exist, '.
+            'or be of the wrong type: %s.',
+            implode(', ', $missing_phids)));
+      }
+
+      $this->services = $services;
+    }
+
+    return $this->services;
+  }
+
+  private function loadAllBindings(array $services) {
+    assert_instances_of($services, 'AlmanacService');
+    $bindings = array_mergev(mpull($services, 'getBindings'));
+    return mpull($bindings, null, 'getPHID');
+  }
+
+  private function loadFreeBindings(DrydockBlueprint $blueprint) {
+    if ($this->freeBindings === null) {
+      $viewer = PhabricatorUser::getOmnipotentUser();
+
+      $pool = id(new DrydockResourceQuery())
+        ->setViewer($viewer)
+        ->withBlueprintPHIDs(array($blueprint->getPHID()))
+        ->withStatuses(
+          array(
+            DrydockResourceStatus::STATUS_PENDING,
+            DrydockResourceStatus::STATUS_OPEN,
+            DrydockResourceStatus::STATUS_CLOSED,
+          ))
+        ->execute();
+
+      $allocated_phids = array();
+      foreach ($pool as $resource) {
+        $allocated_phids[] = $resource->getAttribute('almanacDevicePHID');
+      }
+      $allocated_phids = array_fuse($allocated_phids);
+
+      $services = $this->loadServices($blueprint);
+      $bindings = $this->loadAllBindings($services);
+
+      $free = array();
+      foreach ($bindings as $binding) {
+        if (empty($allocated_phids[$binding->getPHID()])) {
+          $free[] = $binding;
+        }
+      }
+
+      $this->freeBindings = $free;
+    }
+
+    return $this->freeBindings;
+  }
+
+  private function getAlmanacServiceClasses() {
+    return array(
+      'AlmanacDrydockPoolServiceType',
+    );
+  }
+
+
+}

src/applications/drydock/blueprint/DrydockBlueprintImplementation.php

@@ -60,10 +60,6 @@ abstract class DrydockBlueprintImplementation extends Phobject {
     return array();
   }
 
-  public function getDetail($key, $default = null) {
-    return $this->getInstance()->getDetail($key, $default);
-  }
-
 
 /* -(  Lease Acquisition  )-------------------------------------------------- */
 
@@ -86,171 +82,29 @@ abstract class DrydockBlueprintImplementation extends Phobject {
    * @return bool True if the resource and lease are compatible.
    * @task lease
    */
-  abstract public function canAllocateLeaseOnResource(
+  abstract public function canAcquireLeaseOnResource(
     DrydockBlueprint $blueprint,
     DrydockResource $resource,
     DrydockLease $lease);
 
 
   /**
-   * @task lease
+   * Acquire a lease. Allows resources to peform setup as leases are brought
-   */
+   * online.
-  final public function allocateLease(
-    DrydockResource $resource,
-    DrydockLease $lease) {
-
-    $scope = $this->pushActiveScope($resource, $lease);
-
-    $this->log(pht('Trying to Allocate Lease'));
-
-    $lease->setStatus(DrydockLeaseStatus::STATUS_ACQUIRING);
-    $lease->setResourceID($resource->getID());
-    $lease->attachResource($resource);
-
-    $ephemeral_lease = id(clone $lease)->makeEphemeral();
-
-    $allocated = false;
-    $allocation_exception = null;
-
-    $resource->openTransaction();
-      $resource->beginReadLocking();
-        $resource->reload();
-
-        // TODO: Policy stuff.
-        $other_leases = id(new DrydockLease())->loadAllWhere(
-          'status IN (%Ld) AND resourceID = %d',
-          array(
-            DrydockLeaseStatus::STATUS_ACQUIRING,
-            DrydockLeaseStatus::STATUS_ACTIVE,
-          ),
-          $resource->getID());
-
-        try {
-          $allocated = $this->shouldAllocateLease(
-            $resource,
-            $ephemeral_lease,
-            $other_leases);
-        } catch (Exception $ex) {
-          $allocation_exception = $ex;
-        }
-
-        if ($allocated) {
-          $lease->save();
-        }
-      $resource->endReadLocking();
-    if ($allocated) {
-      $resource->saveTransaction();
-      $this->log(pht('Allocated Lease'));
-    } else {
-      $resource->killTransaction();
-      $this->log(pht('Failed to Allocate Lease'));
-    }
-
-    if ($allocation_exception) {
-      $this->logException($allocation_exception);
-    }
-
-    return $allocated;
-  }
-
-
-  /**
-   * Enforce lease limits on resources. Allows resources to reject leases if
-   * they would become over-allocated by accepting them.
-   *
-   * For example, if a resource represents disk space, this method might check
-   * how much space the lease is asking for (say, 200MB) and how much space is
-   * left unallocated on the resource. It could grant the lease (return true)
-   * if it has enough remaining space (more than 200MB), and reject the lease
-   * (return false) if it does not (less than 200MB).
-   *
-   * A resource might also allow only exclusive leases. In this case it could
-   * accept a new lease (return true) if there are no active leases, or reject
-   * the new lease (return false) if there any other leases.
-   *
-   * A lock is held on the resource while this method executes to prevent
-   * multiple processes from allocating leases on the resource simultaneously.
-   * However, this means you should implement the method as cheaply as possible.
-   * In particular, do not perform any actual acquisition or setup in this
-   * method.
-   *
-   * If allocation is permitted, the lease will be moved to `ACQUIRING` status
-   * and @{method:executeAcquireLease} will be called to actually perform
-   * acquisition.
-   *
-   * General compatibility checks unrelated to resource limits and capacity are
-   * better implemented in @{method:canAllocateLease}, which serves as a
-   * cheap filter before lock acquisition.
-   *
-   * @param   DrydockResource     Candidate resource to allocate the lease on.
-   * @param   DrydockLease        Pending lease that wants to allocate here.
-   * @param   list<DrydockLease>  Other allocated and acquired leases on the
-   *                              resource. The implementation can inspect them
-   *                              to verify it can safely add the new lease.
-   * @return  bool                True to allocate the lease on the resource;
-   *                              false to reject it.
-   * @task lease
-   */
-  abstract protected function shouldAllocateLease(
-    DrydockResource $resource,
-    DrydockLease $lease,
-    array $other_leases);
-
-
-  /**
-   * @task lease
-   */
-  final public function acquireLease(
-    DrydockResource $resource,
-    DrydockLease $lease) {
-
-    $scope = $this->pushActiveScope($resource, $lease);
-
-    $this->log(pht('Acquiring Lease'));
-    $lease->setStatus(DrydockLeaseStatus::STATUS_ACTIVE);
-    $lease->setResourceID($resource->getID());
-    $lease->attachResource($resource);
-
-    $ephemeral_lease = id(clone $lease)->makeEphemeral();
-
-    try {
-      $this->executeAcquireLease($resource, $ephemeral_lease);
-    } catch (Exception $ex) {
-      $this->logException($ex);
-      throw $ex;
-    }
-
-    $lease->setAttributes($ephemeral_lease->getAttributes());
-    $lease->save();
-    $this->log(pht('Acquired Lease'));
-  }
-
-
-  /**
-   * Acquire and activate an allocated lease. Allows resources to peform setup
-   * as leases are brought online.
-   *
-   * Following a successful call to @{method:canAllocateLease}, a lease is moved
-   * to `ACQUIRING` status and this method is called after resource locks are
-   * released. Nothing is locked while this method executes; the implementation
-   * is free to perform expensive operations like writing files and directories,
-   * executing commands, etc.
-   *
-   * After this method executes, the lease status is moved to `ACTIVE` and the
-   * original leasee may access it.
    *
    * If acquisition fails, throw an exception.
    *
-   * @param   DrydockResource   Resource to acquire a lease on.
+   * @param DrydockBlueprint Blueprint which built the resource.
-   * @param   DrydockLease      Lease to acquire.
+   * @param DrydockResource Resource to acquire a lease on.
-   * @return  void
+   * @param DrydockLease Requested lease.
+   * @return void
+   * @task lease
    */
-  abstract protected function executeAcquireLease(
+  abstract public function acquireLease(
+    DrydockBlueprint $blueprint,
     DrydockResource $resource,
     DrydockLease $lease);
 
-
-
   final public function releaseLease(
     DrydockResource $resource,
     DrydockLease $lease) {
@@ -352,6 +206,7 @@ abstract class DrydockBlueprintImplementation extends Phobject {
    * @param DrydockLease Requested lease.
    * @return bool True if this blueprint appears likely to be able to allocate
    *   a suitable resource.
+   * @task resource
    */
   abstract public function canAllocateResourceForLease(
     DrydockBlueprint $blueprint,
@@ -369,34 +224,12 @@ abstract class DrydockBlueprintImplementation extends Phobject {
    * @param DrydockBlueprint The blueprint which should allocate a resource.
    * @param DrydockLease Requested lease.
    * @return DrydockResource Allocated resource.
+   * @task resource
    */
-  abstract protected function executeAllocateResource(
+  abstract public function allocateResource(
     DrydockBlueprint $blueprint,
     DrydockLease $lease);
 
-  final public function allocateResource(
-    DrydockBlueprint $blueprint,
-    DrydockLease $lease) {
-
-    $scope = $this->pushActiveScope(null, $lease);
-
-    $this->log(
-      pht(
-        "Blueprint '%s': Allocating Resource for '%s'",
-        $this->getBlueprintClass(),
-        $lease->getLeaseName()));
-
-    try {
-      $resource = $this->executeAllocateResource($blueprint, $lease);
-      $this->validateAllocatedResource($resource);
-    } catch (Exception $ex) {
-      $this->logException($ex);
-      throw $ex;
-    }
-
-    return $resource;
-  }
-
 
 /* -(  Logging  )------------------------------------------------------------ */
 
@@ -454,14 +287,15 @@ abstract class DrydockBlueprintImplementation extends Phobject {
     return idx(self::getAllBlueprintImplementations(), $class);
   }
 
-  protected function newResourceTemplate($name) {
+  protected function newResourceTemplate(
+    DrydockBlueprint $blueprint,
+    $name) {
+
     $resource = id(new DrydockResource())
-      ->setBlueprintPHID($this->getInstance()->getPHID())
+      ->setBlueprintPHID($blueprint->getPHID())
-      ->setBlueprintClass($this->getBlueprintClass())
       ->setType($this->getType())
       ->setStatus(DrydockResourceStatus::STATUS_PENDING)
-      ->setName($name)
+      ->setName($name);
-      ->save();
 
     $this->activeResource = $resource;
 
@@ -473,39 +307,6 @@ abstract class DrydockBlueprintImplementation extends Phobject {
     return $resource;
   }
 
-  /**
-   * Sanity checks that the blueprint is implemented properly.
-   */
-  private function validateAllocatedResource($resource) {
-    $blueprint = $this->getBlueprintClass();
-
-    if (!($resource instanceof DrydockResource)) {
-      throw new Exception(
-        pht(
-          "Blueprint '%s' is not properly implemented: %s must return an ".
-          "object of type %s or throw, but returned something else.",
-          $blueprint,
-          'executeAllocateResource()',
-          'DrydockResource'));
-    }
-
-    $current_status = $resource->getStatus();
-    $req_status = DrydockResourceStatus::STATUS_OPEN;
-    if ($current_status != $req_status) {
-      $current_name = DrydockResourceStatus::getNameForStatus($current_status);
-      $req_name = DrydockResourceStatus::getNameForStatus($req_status);
-      throw new Exception(
-        pht(
-          "Blueprint '%s' is not properly implemented: %s must return a %s ".
-          "with status '%s', but returned one with status '%s'.",
-          $blueprint,
-          'executeAllocateResource()',
-          'DrydockResource',
-          $req_name,
-          $current_name));
-    }
-  }
-
   private function pushActiveScope(
     DrydockResource $resource = null,
     DrydockLease $lease = null) {

src/applications/drydock/blueprint/DrydockWorkingCopyBlueprintImplementation.php

@@ -35,7 +35,7 @@ final class DrydockWorkingCopyBlueprintImplementation
     return true;
   }
 
-  public function canAllocateLeaseOnResource(
+  public function canAcquireLeaseOnResource(
     DrydockBlueprint $blueprint,
     DrydockResource $resource,
     DrydockLease $lease) {
@@ -47,15 +47,7 @@ final class DrydockWorkingCopyBlueprintImplementation
     return ($resource_repo && $lease_repo && ($resource_repo == $lease_repo));
   }
 
-  protected function shouldAllocateLease(
+  public function allocateResource(
-    DrydockResource $resource,
-    DrydockLease $lease,
-    array $other_leases) {
-    // TODO: These checks are out of date.
-    return !$other_leases;
-  }
-
-  protected function executeAllocateResource(
     DrydockBlueprint $blueprint,
     DrydockLease $lease) {
 
@@ -105,6 +97,7 @@ final class DrydockWorkingCopyBlueprintImplementation
     $this->log(pht('Complete.'));
 
     $resource = $this->newResourceTemplate(
+      $blueprint,
       pht(
         'Working Copy (%s)',
         $repository->getCallsign()));
@@ -117,7 +110,8 @@ final class DrydockWorkingCopyBlueprintImplementation
     return $resource;
   }
 
-  protected function executeAcquireLease(
+  public function acquireLease(
+    DrydockBlueprint $blueprint,
     DrydockResource $resource,
     DrydockLease $lease) {
     return;

src/applications/drydock/constants/DrydockLeaseStatus.php

@@ -3,7 +3,7 @@
 final class DrydockLeaseStatus extends DrydockConstants {
 
   const STATUS_PENDING      = 0;
-  const STATUS_ACQUIRING    = 5;
+  const STATUS_ACQUIRED     = 5;
   const STATUS_ACTIVE       = 1;
   const STATUS_RELEASED     = 2;
   const STATUS_BROKEN       = 3;
@@ -12,7 +12,7 @@ final class DrydockLeaseStatus extends DrydockConstants {
   public static function getNameForStatus($status) {
     $map = array(
       self::STATUS_PENDING    => pht('Pending'),
-      self::STATUS_ACQUIRING  => pht('Acquiring'),
+      self::STATUS_ACQUIRED   => pht('Acquired'),
       self::STATUS_ACTIVE     => pht('Active'),
       self::STATUS_RELEASED   => pht('Released'),
       self::STATUS_BROKEN     => pht('Broken'),
@@ -25,7 +25,7 @@ final class DrydockLeaseStatus extends DrydockConstants {
   public static function getAllStatuses() {
     return array(
       self::STATUS_PENDING,
-      self::STATUS_ACQUIRING,
+      self::STATUS_ACQUIRED,
       self::STATUS_ACTIVE,
       self::STATUS_RELEASED,
       self::STATUS_BROKEN,

src/applications/drydock/customfield/DrydockBlueprintCoreCustomField.php

@@ -40,4 +40,8 @@ final class DrydockBlueprintCoreCustomField
     return;
   }
 
+  public function getBlueprintFieldValue() {
+    return $this->getProxy()->getFieldValue();
+  }
+
 }

src/applications/drydock/customfield/DrydockBlueprintCustomField.php

@@ -1,4 +1,8 @@
 <?php
 
 abstract class DrydockBlueprintCustomField
-  extends PhabricatorCustomField {}
+  extends PhabricatorCustomField {
+
+  abstract public function getBlueprintFieldValue();
+
+}

src/applications/drydock/management/DrydockManagementCreateResourceWorkflow.php

@@ -1,81 +0,0 @@
-<?php
-
-final class DrydockManagementCreateResourceWorkflow
-  extends DrydockManagementWorkflow {
-
-  protected function didConstruct() {
-    $this
-      ->setName('create-resource')
-      ->setSynopsis(pht('Create a resource manually.'))
-      ->setArguments(
-        array(
-          array(
-            'name'      => 'name',
-            'param'     => 'resource_name',
-            'help'      => pht('Resource name.'),
-          ),
-          array(
-            'name'      => 'blueprint',
-            'param'     => 'blueprint_id',
-            'help'      => pht('Blueprint ID.'),
-          ),
-          array(
-            'name'      => 'attributes',
-            'param'     => 'name=value,...',
-            'help'      => pht('Resource attributes.'),
-          ),
-        ));
-  }
-
-  public function execute(PhutilArgumentParser $args) {
-    $console = PhutilConsole::getConsole();
-
-    $resource_name = $args->getArg('name');
-    if (!$resource_name) {
-      throw new PhutilArgumentUsageException(
-        pht(
-          'Specify a resource name with `%s`.',
-          '--name'));
-    }
-
-    $blueprint_id = $args->getArg('blueprint');
-    if (!$blueprint_id) {
-      throw new PhutilArgumentUsageException(
-        pht(
-          'Specify a blueprint ID with `%s`.',
-          '--blueprint'));
-    }
-
-    $attributes = $args->getArg('attributes');
-    if ($attributes) {
-      $options = new PhutilSimpleOptions();
-      $options->setCaseSensitive(true);
-      $attributes = $options->parse($attributes);
-    }
-
-    $viewer = $this->getViewer();
-
-    $blueprint = id(new DrydockBlueprintQuery())
-      ->setViewer($viewer)
-      ->withIDs(array($blueprint_id))
-      ->executeOne();
-    if (!$blueprint) {
-      throw new PhutilArgumentUsageException(
-        pht('Specified blueprint does not exist.'));
-    }
-
-    $resource = id(new DrydockResource())
-      ->setBlueprintPHID($blueprint->getPHID())
-      ->setType($blueprint->getImplementation()->getType())
-      ->setName($resource_name)
-      ->setStatus(DrydockResourceStatus::STATUS_OPEN);
-    if ($attributes) {
-      $resource->setAttributes($attributes);
-    }
-    $resource->save();
-
-    $console->writeOut("%s\n", pht('Created Resource %s', $resource->getID()));
-    return 0;
-  }
-
-}

src/applications/drydock/query/DrydockLeaseSearchEngine.php

@@ -74,7 +74,7 @@ final class DrydockLeaseSearchEngine
           'statuses',
           array(
             DrydockLeaseStatus::STATUS_PENDING,
-            DrydockLeaseStatus::STATUS_ACQUIRING,
+            DrydockLeaseStatus::STATUS_ACQUIRED,
             DrydockLeaseStatus::STATUS_ACTIVE,
           ));
       case 'all':

src/applications/drydock/storage/DrydockBlueprint.php

@@ -1,5 +1,9 @@
 <?php
 
+/**
+ * @task resource Allocating Resources
+ * @task lease Acquiring Leases
+ */
 final class DrydockBlueprint extends DrydockDAO
   implements
     PhabricatorApplicationTransactionInterface,
@@ -14,6 +18,7 @@ final class DrydockBlueprint extends DrydockDAO
 
   private $implementation = self::ATTACHABLE;
   private $customFields = self::ATTACHABLE;
+  private $fields = null;
 
   public static function initializeNewBlueprint(PhabricatorUser $actor) {
     $app = id(new PhabricatorApplicationQuery())
@@ -68,27 +73,96 @@ final class DrydockBlueprint extends DrydockDAO
     return $this;
   }
 
+  public function getFieldValue($key) {
+    $key = "std:drydock:core:{$key}";
+    $fields = $this->loadCustomFields();
+
+    $field = idx($fields, $key);
+    if (!$field) {
+      throw new Exception(
+        pht(
+          'Unknown blueprint field "%s"!',
+          $key));
+    }
+
+    return $field->getBlueprintFieldValue();
+  }
+
+  private function loadCustomFields() {
+    if ($this->fields === null) {
+      $field_list = PhabricatorCustomField::getObjectFields(
+        $this,
+        PhabricatorCustomField::ROLE_VIEW);
+      $field_list->readFieldsFromStorage($this);
+
+      $this->fields = $field_list->getFields();
+    }
+    return $this->fields;
+  }
+
+
+/* -(  Allocating Resources  )----------------------------------------------- */
+
+
+  /**
+   * @task resource
+   */
   public function canEverAllocateResourceForLease(DrydockLease $lease) {
     return $this->getImplementation()->canEverAllocateResourceForLease(
       $this,
       $lease);
   }
 
+
+  /**
+   * @task resource
+   */
   public function canAllocateResourceForLease(DrydockLease $lease) {
     return $this->getImplementation()->canAllocateResourceForLease(
       $this,
       $lease);
   }
 
-  public function canAllocateLeaseOnResource(
+
+  /**
+   * @task resource
+   */
+  public function allocateResource(DrydockLease $lease) {
+    return $this->getImplementation()->allocateResource(
+      $this,
+      $lease);
+  }
+
+
+/* -(  Acquiring Leases  )--------------------------------------------------- */
+
+
+  /**
+   * @task lease
+   */
+  public function canAcquireLeaseOnResource(
     DrydockResource $resource,
     DrydockLease $lease) {
-    return $this->getImplementation()->canAllocateLeaseOnResource(
+    return $this->getImplementation()->canAcquireLeaseOnResource(
       $this,
       $resource,
       $lease);
   }
 
+
+  /**
+   * @task lease
+   */
+  public function acquireLease(
+    DrydockResource $resource,
+    DrydockLease $lease) {
+    return $this->getImplementation()->acquireLease(
+      $this,
+      $resource,
+      $lease);
+  }
+
+
 /* -(  PhabricatorApplicationTransactionInterface  )------------------------- */
 
 

src/applications/drydock/storage/DrydockLease.php

@@ -13,6 +13,8 @@ final class DrydockLease extends DrydockDAO
 
   private $resource = self::ATTACHABLE;
   private $releaseOnDestruction;
+  private $isAcquired = false;
+  private $activateWhenAcquired = false;
 
   /**
    * Flag this lease to be released when its destructor is called. This is
@@ -133,8 +135,8 @@ final class DrydockLease extends DrydockDAO
 
   public function isActive() {
     switch ($this->status) {
+      case DrydockLeaseStatus::STATUS_ACQUIRED:
       case DrydockLeaseStatus::STATUS_ACTIVE:
-      case DrydockLeaseStatus::STATUS_ACQUIRING:
         return true;
     }
     return false;
@@ -171,7 +173,7 @@ final class DrydockLease extends DrydockDAO
           case DrydockLeaseStatus::STATUS_BROKEN:
             throw new Exception(pht('Lease has been broken!'));
           case DrydockLeaseStatus::STATUS_PENDING:
-          case DrydockLeaseStatus::STATUS_ACQUIRING:
+          case DrydockLeaseStatus::STATUS_ACQUIRED:
             break;
           default:
             throw new Exception(pht('Unknown status??'));
@@ -199,6 +201,53 @@ final class DrydockLease extends DrydockDAO
     return $this;
   }
 
+  public function setActivateWhenAcquired($activate) {
+    $this->activateWhenAcquired = true;
+    return $this;
+  }
+
+  public function acquireOnResource(DrydockResource $resource) {
+    $expect_status = DrydockLeaseStatus::STATUS_PENDING;
+    $actual_status = $this->getStatus();
+    if ($actual_status != $expect_status) {
+      throw new Exception(
+        pht(
+          'Trying to acquire a lease on a resource which is in the wrong '.
+          'state: status must be "%s", actually "%s".',
+          $expect_status,
+          $actual_status));
+    }
+
+    if ($this->activateWhenAcquired) {
+      $new_status = DrydockLeaseStatus::STATUS_ACTIVE;
+    } else {
+      $new_status = DrydockLeaseStatus::STATUS_PENDING;
+    }
+
+    if ($new_status === DrydockLeaseStatus::STATUS_ACTIVE) {
+      if ($resource->getStatus() === DrydockResourceStatus::STATUS_PENDING) {
+        throw new Exception(
+          pht(
+            'Trying to acquire an active lease on a pending resource. '.
+            'You can not immediately activate leases on resources which '.
+            'need time to start up.'));
+      }
+    }
+
+    $this
+      ->setResourceID($resource->getID())
+      ->setStatus($new_status)
+      ->save();
+
+    $this->isAcquired = true;
+
+    return $this;
+  }
+
+  public function isAcquiredLease() {
+    return $this->isAcquired;
+  }
+
 
 /* -(  PhabricatorPolicyInterface  )----------------------------------------- */
 

src/applications/drydock/storage/DrydockResource.php

@@ -15,6 +15,8 @@ final class DrydockResource extends DrydockDAO
   protected $ownerPHID;
 
   private $blueprint = self::ATTACHABLE;
+  private $isAllocated = false;
+  private $activateWhenAllocated = false;
 
   protected function getConfiguration() {
     return array(
@@ -73,10 +75,47 @@ final class DrydockResource extends DrydockDAO
     return $this;
   }
 
-  public function canAllocateLease(DrydockLease $lease) {
+  public function setActivateWhenAllocated($activate) {
-    return $this->getBlueprint()->canAllocateLeaseOnResource(
+    $this->activateWhenAllocated = $activate;
-      $this,
+    return $this;
-      $lease);
+  }
+
+  public function allocateResource($status) {
+    if ($this->getID()) {
+      throw new Exception(
+        pht(
+          'Trying to allocate a resource which has already been persisted. '.
+          'Only new resources may be allocated.'));
+    }
+
+    $expect_status = DrydockResourceStatus::STATUS_PENDING;
+    $actual_status = $this->getStatus();
+    if ($actual_status != $expect_status) {
+      throw new Exception(
+        pht(
+          'Trying to allocate a resource from the wrong status. Status must '.
+          'be "%s", actually "%s".',
+          $expect_status,
+          $actual_status));
+    }
+
+    if ($this->activateWhenAllocated) {
+      $new_status = DrydockResourceStatus::STATUS_OPEN;
+    } else {
+      $new_status = DrydockResourceStatus::STATUS_PENDING;
+    }
+
+    $this
+      ->setStatus($new_status)
+      ->save();
+
+    $this->didAllocate = true;
+
+    return $this;
+  }
+
+  public function isAllocatedResource() {
+    return $this->isAllocated;
   }
 
   public function closeResource() {

src/applications/drydock/worker/DrydockAllocatorWorker.php

@@ -1,5 +1,10 @@
 <?php
 
+/**
+ * @task allocate Allocator
+ * @task resource Managing Resources
+ * @task lease Managing Leases
+ */
 final class DrydockAllocatorWorker extends PhabricatorWorker {
 
   private function getViewer() {
@@ -27,10 +32,22 @@ final class DrydockAllocatorWorker extends PhabricatorWorker {
 
   protected function doWork() {
     $lease = $this->loadLease();
-    $this->allocateLease($lease);
+    $this->allocateAndAcquireLease($lease);
   }
 
-  private function allocateLease(DrydockLease $lease) {
+
+/* -(  Allocator  )---------------------------------------------------------- */
+
+
+  /**
+   * Find or build a resource which can satisfy a given lease request, then
+   * acquire the lease.
+   *
+   * @param DrydockLease Requested lease.
+   * @return void
+   * @task allocator
+   */
+  private function allocateAndAcquireLease(DrydockLease $lease) {
     $blueprints = $this->loadBlueprintsForAllocatingLease($lease);
 
     // If we get nothing back, that means no blueprint is defined which can
@@ -72,7 +89,8 @@ final class DrydockAllocatorWorker extends PhabricatorWorker {
       $exceptions = array();
       foreach ($usable_blueprints as $blueprint) {
         try {
-          $resources[] = $blueprint->allocateResource($lease);
+          $resources[] = $this->allocateResource($blueprint, $lease);
+
           // Bail after allocating one resource, we don't need any more than
           // this.
           break;
@@ -106,7 +124,7 @@ final class DrydockAllocatorWorker extends PhabricatorWorker {
     $allocated = false;
     foreach ($resources as $resource) {
       try {
-        $blueprint->allocateLease($resource, $lease);
+        $this->acquireLease($resource, $lease);
         $allocated = true;
         break;
       } catch (Exception $ex) {
@@ -129,88 +147,54 @@ final class DrydockAllocatorWorker extends PhabricatorWorker {
 
 
   /**
-   * Load a list of all resources which a given lease can possibly be
+   * Get all the @{class:DrydockBlueprintImplementation}s which can possibly
-   * allocated against.
+   * build a resource to satisfy a lease.
+   *
+   * This method returns blueprints which might, at some time, be able to
+   * build a resource which can satisfy the lease. They may not be able to
+   * build that resource right now.
    *
-   * @param list<DrydockBlueprint> Blueprints which may produce suitable
-   *   resources.
    * @param DrydockLease Requested lease.
-   * @return list<DrydockResource> Resources which may be able to allocate
+   * @return list<DrydockBlueprintImplementation> List of qualifying blueprint
-   *   the lease.
+   *   implementations.
+   * @task allocator
    */
-  private function loadResourcesForAllocatingLease(
+  private function loadBlueprintImplementationsForAllocatingLease(
-    array $blueprints,
     DrydockLease $lease) {
-    assert_instances_of($blueprints, 'DrydockBlueprint');
-    $viewer = $this->getViewer();
 
-    $resources = id(new DrydockResourceQuery())
+    $impls = DrydockBlueprintImplementation::getAllBlueprintImplementations();
-      ->setViewer($viewer)
-      ->withBlueprintPHIDs(mpull($blueprints, 'getPHID'))
-      ->withTypes(array($lease->getResourceType()))
-      ->withStatuses(
-        array(
-          DrydockResourceStatus::STATUS_PENDING,
-          DrydockResourceStatus::STATUS_OPEN,
-        ))
-      ->execute();
 
     $keep = array();
-    foreach ($resources as $key => $resource) {
+    foreach ($impls as $key => $impl) {
-      if (!$resource->canAllocateLease($lease)) {
+      // Don't use disabled blueprint types.
+      if (!$impl->isEnabled()) {
         continue;
       }
 
-      $keep[$key] = $resource;
+      // Don't use blueprint types which can't allocate the correct kind of
+      // resource.
+      if ($impl->getType() != $lease->getResourceType()) {
+        continue;
+      }
+
+      if (!$impl->canAnyBlueprintEverAllocateResourceForLease($lease)) {
+        continue;
+      }
+
+      $keep[$key] = $impl;
     }
 
     return $keep;
   }
 
 
-  /**
-   * Rank blueprints by suitability for building a new resource for a
-   * particular lease.
-   *
-   * @param list<DrydockBlueprint> List of blueprints.
-   * @param DrydockLease Requested lease.
-   * @return list<DrydockBlueprint> Ranked list of blueprints.
-   */
-  private function rankBlueprints(array $blueprints, DrydockLease $lease) {
-    assert_instances_of($blueprints, 'DrydockBlueprint');
-
-    // TODO: Implement improvements to this ranking algorithm if they become
-    // available.
-    shuffle($blueprints);
-
-    return $blueprints;
-  }
-
-
-  /**
-   * Rank resources by suitability for allocating a particular lease.
-   *
-   * @param list<DrydockResource> List of resources.
-   * @param DrydockLease Requested lease.
-   * @return list<DrydockResource> Ranked list of resources.
-   */
-  private function rankResources(array $resources, DrydockLease $lease) {
-    assert_instances_of($resources, 'DrydockResource');
-
-    // TODO: Implement improvements to this ranking algorithm if they become
-    // available.
-    shuffle($resources);
-
-    return $resources;
-  }
-
-
   /**
    * Get all the concrete @{class:DrydockBlueprint}s which can possibly
    * build a resource to satisfy a lease.
    *
    * @param DrydockLease Requested lease.
    * @return list<DrydockBlueprint> List of qualifying blueprints.
+   * @task allocator
    */
   private function loadBlueprintsForAllocatingLease(
     DrydockLease $lease) {
@@ -243,40 +227,42 @@ final class DrydockAllocatorWorker extends PhabricatorWorker {
 
 
   /**
-   * Get all the @{class:DrydockBlueprintImplementation}s which can possibly
+   * Load a list of all resources which a given lease can possibly be
-   * build a resource to satisfy a lease.
+   * allocated against.
-   *
-   * This method returns blueprints which might, at some time, be able to
-   * build a resource which can satisfy the lease. They may not be able to
-   * build that resource right now.
    *
+   * @param list<DrydockBlueprint> Blueprints which may produce suitable
+   *   resources.
    * @param DrydockLease Requested lease.
-   * @return list<DrydockBlueprintImplementation> List of qualifying blueprint
+   * @return list<DrydockResource> Resources which may be able to allocate
-   *   implementations.
+   *   the lease.
+   * @task allocator
    */
-  private function loadBlueprintImplementationsForAllocatingLease(
+  private function loadResourcesForAllocatingLease(
+    array $blueprints,
     DrydockLease $lease) {
+    assert_instances_of($blueprints, 'DrydockBlueprint');
+    $viewer = $this->getViewer();
 
-    $impls = DrydockBlueprintImplementation::getAllBlueprintImplementations();
+    $resources = id(new DrydockResourceQuery())
+      ->setViewer($viewer)
+      ->withBlueprintPHIDs(mpull($blueprints, 'getPHID'))
+      ->withTypes(array($lease->getResourceType()))
+      ->withStatuses(
+        array(
+          DrydockResourceStatus::STATUS_PENDING,
+          DrydockResourceStatus::STATUS_OPEN,
+        ))
+      ->execute();
 
     $keep = array();
-    foreach ($impls as $key => $impl) {
+    foreach ($resources as $key => $resource) {
-      // Don't use disabled blueprint types.
+      $blueprint = $resource->getBlueprint();
-      if (!$impl->isEnabled()) {
+
+      if (!$blueprint->canAcquireLeaseOnResource($resource, $lease)) {
         continue;
       }
 
-      // Don't use blueprint types which can't allocate the correct kind of
+      $keep[$key] = $resource;
-      // resource.
-      if ($impl->getType() != $lease->getResourceType()) {
-        continue;
-      }
-
-      if (!$impl->canAnyBlueprintEverAllocateResourceForLease($lease)) {
-        continue;
-      }
-
-      $keep[$key] = $impl;
     }
 
     return $keep;
@@ -288,8 +274,9 @@ final class DrydockAllocatorWorker extends PhabricatorWorker {
    * a lease from a list of blueprints.
    *
    * @param list<DrydockBlueprint> List of blueprints.
-   * @param list<DrydockBlueprint> List with fully allocated blueprints
+   * @return list<DrydockBlueprint> List with blueprints that can not allocate
-   *   removed.
+   *   a resource for the lease right now removed.
+   * @task allocator
    */
   private function removeOverallocatedBlueprints(
     array $blueprints,
@@ -308,4 +295,182 @@ final class DrydockAllocatorWorker extends PhabricatorWorker {
     return $keep;
   }
 
+
+  /**
+   * Rank blueprints by suitability for building a new resource for a
+   * particular lease.
+   *
+   * @param list<DrydockBlueprint> List of blueprints.
+   * @param DrydockLease Requested lease.
+   * @return list<DrydockBlueprint> Ranked list of blueprints.
+   * @task allocator
+   */
+  private function rankBlueprints(array $blueprints, DrydockLease $lease) {
+    assert_instances_of($blueprints, 'DrydockBlueprint');
+
+    // TODO: Implement improvements to this ranking algorithm if they become
+    // available.
+    shuffle($blueprints);
+
+    return $blueprints;
+  }
+
+
+  /**
+   * Rank resources by suitability for allocating a particular lease.
+   *
+   * @param list<DrydockResource> List of resources.
+   * @param DrydockLease Requested lease.
+   * @return list<DrydockResource> Ranked list of resources.
+   * @task allocator
+   */
+  private function rankResources(array $resources, DrydockLease $lease) {
+    assert_instances_of($resources, 'DrydockResource');
+
+    // TODO: Implement improvements to this ranking algorithm if they become
+    // available.
+    shuffle($resources);
+
+    return $resources;
+  }
+
+
+/* -(  Managing Resources  )------------------------------------------------- */
+
+
+  /**
+   * Perform an actual resource allocation with a particular blueprint.
+   *
+   * @param DrydockBlueprint The blueprint to allocate a resource from.
+   * @param DrydockLease Requested lease.
+   * @return DrydockResource Allocated resource.
+   * @task resource
+   */
+  private function allocateResource(
+    DrydockBlueprint $blueprint,
+    DrydockLease $lease) {
+    $resource = $blueprint->allocateResource($lease);
+    $this->validateAllocatedResource($resource);
+    return $resource;
+  }
+
+
+  /**
+   * Check that the resource a blueprint allocated is roughly the sort of
+   * object we expect.
+   *
+   * @param DrydockBlueprint Blueprint which built the resource.
+   * @param wild Thing which the blueprint claims is a valid resource.
+   * @param DrydockLease Lease the resource was allocated for.
+   * @return void
+   * @task resource
+   */
+  private function validateAllocatedResource(
+    DrydockBlueprint $blueprint,
+    $resource,
+    DrydockLease $lease) {
+    $blueprint = $this->getBlueprintClass();
+
+    if (!($resource instanceof DrydockResource)) {
+      throw new Exception(
+        pht(
+          'Blueprint "%s" (of type "%s") is not properly implemented: %s must '.
+          'return an object of type %s or throw, but returned something else.',
+          $blueprint->getBlueprintName(),
+          $blueprint->getClassName(),
+          'allocateResource()',
+          'DrydockResource'));
+    }
+
+    if (!$resource->isAllocatedResource()) {
+      throw new Exception(
+        pht(
+          'Blueprint "%s" (of type "%s") is not properly implemented: %s '.
+          'must actually allocate the resource it returns.',
+          $blueprint->getBlueprintName(),
+          $blueprint->getClassName(),
+          'allocateResource()'));
+    }
+
+    $resource_type = $resource->getType();
+    $lease_type = $lease->getResourceType();
+
+    if ($resource_type !== $lease_type) {
+      // TODO: Destroy the resource here?
+
+      throw new Exception(
+        pht(
+          'Blueprint "%s" (of type "%s") is not properly implemented: it '.
+          'built a resource of type "%s" to satisfy a lease requesting a '.
+          'resource of type "%s".',
+          $blueprint->getBlueprintName(),
+          $blueprint->getClassName(),
+          $resource_type,
+          $lease_type));
+    }
+  }
+
+
+/* -(  Managing Leases  )---------------------------------------------------- */
+
+
+  /**
+   * Perform an actual lease acquisition on a particular resource.
+   *
+   * @param DrydockResource Resource to acquire a lease on.
+   * @param DrydockLease Lease to acquire.
+   * @return void
+   * @task lease
+   */
+  private function acquireLease(
+    DrydockResource $resource,
+    DrydockLease $lease) {
+
+    $blueprint = $resource->getBlueprint();
+    $blueprint->acquireLease($resource, $lease);
+
+    $this->validateAcquiredLease($blueprint, $resource, $lease);
+  }
+
+
+  /**
+   * Make sure that a lease was really acquired properly.
+   *
+   * @param DrydockBlueprint Blueprint which created the resource.
+   * @param DrydockResource Resource which was acquired.
+   * @param DrydockLease The lease which was supposedly acquired.
+   * @return void
+   * @task lease
+   */
+  private function validateAcquiredLease(
+    DrydockBlueprint $blueprint,
+    DrydockResource $resource,
+    DrydockLease $lease) {
+
+    if (!$lease->isAcquiredLease()) {
+      throw new Exception(
+        pht(
+          'Blueprint "%s" (of type "%s") is not properly implemented: it '.
+          'returned from "%s" without acquiring a lease.',
+          $blueprint->getBlueprintName(),
+          $blueprint->getClassName(),
+          'acquireLease()'));
+    }
+
+    $lease_id = $lease->getResourceID();
+    $resource_id = $resource->getID();
+
+    if ($lease_id !== $resource_id) {
+      // TODO: Destroy the lease?
+      throw new Exception(
+        pht(
+          'Blueprint "%s" (of type "%s") is not properly implemented: it '.
+          'returned from "%s" with a lease acquired on the wrong resource.',
+          $blueprint->getBlueprintName(),
+          $blueprint->getClassName(),
+          'acquireLease()'));
+    }
+  }
+
+
 }

src/applications/passphrase/controller/PassphraseCredentialEditController.php

@@ -31,7 +31,7 @@ final class PassphraseCredentialEditController extends PassphraseController {
         throw new Exception(
           pht(
             'Credential has noncreateable type "%s"!',
-            $credential->getCredentialType()));
+            $type_const));
       }
 
       $credential = PassphraseCredential::initializeNewCredential($viewer)