Fix an issue where newly created Drydock resources could be improperly acquired

Summary:
Ref T9252. This is mostly a fix for an edge case from D14236. Here's the setup:

  - There are no resources.
  - A request for a new resource arrives.
  - We build a new resource.

Now, if we were leasing an existing resource, we'd call `canAcquireLeaseOnResource()` before acquiring a lease on the new resource.

However, for new resources we don't do that: we just acquire a lease immediately. This is wrong, because we now allow and expect some resources to be unleasable when created.

In a more complex workflow, this can also produce the wrong result and leave the lease acquired sub-optimally (and, today, deadlocked).

Make the "can we acquire?" pathway consistent for new and existing resources, so we always do the same set of checks.

Test Plan:
  - Started daemons.
  - Deleted all working copy resources.
  - Ran two working-copy-using build plans at the same time.
  - Before this change, one would often [1] acquire a lease on a pending resource which never allocated, then deadlock.
  - After this change, the same thing happens except that the lease remains pending and the work completes.

[1] Although the race this implies is allowed (resource pool limits are soft/advisory, and it is expected that we may occasionally run over them), it's MUCH easier to hit right now than I would expect it to be, so I think there's probably at least one more small bug here somewhere. I'll see if I can root it out after this change.

Reviewers: chad, hach-que

Reviewed By: hach-que

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14272

src/applications/drydock/worker/DrydockLeaseUpdateWorker.php

@@ -211,10 +211,19 @@ final class DrydockLeaseUpdateWorker extends DrydockWorker {
           $exceptions);
       }
 
+      $resources = $this->removeUnacquirableResources($resources, $lease);
+      if (!$resources) {
+        // If we make it here, we just built a resource but aren't allowed
+        // to acquire it. We expect this during routine operation if the
+        // resource prevents acquisition until it activates. Yield and wait
+        // for activation.
+        throw new PhabricatorWorkerYieldException(15);
+      }
+
       // NOTE: We have not acquired the lease yet, so it is possible that the
       // resource we just built will be snatched up by some other lease before
-      // we can. This is not problematic: we'll retry a little later and should
-      // suceed eventually.
+      // we can acquire it. This is not problematic: we'll retry a little later
+      // and should suceed eventually.
     }
 
     $resources = $this->rankResources($resources, $lease);
@@ -382,6 +391,22 @@ final class DrydockLeaseUpdateWorker extends DrydockWorker {
         ))
       ->execute();
 
+    return $this->removeUnacquirableResources($resources, $lease);
+  }
+
+
+  /**
+   * Remove resources which can not be acquired by a given lease from a list.
+   *
+   * @param list<DrydockResource> Candidate resources.
+   * @param DrydockLease Acquiring lease.
+   * @return list<DrydockResource> Resources which the lease may be able to
+   *   acquire.
+   * @task allocator
+   */
+  private function removeUnacquirableResources(
+    array $resources,
+    DrydockLease $lease) {
     $keep = array();
     foreach ($resources as $key => $resource) {
       $blueprint = $resource->getBlueprint();