After a Drydock lease triggers a resource to be reclaimed, stop it from triggering another reclaim until the first one completes

Summary:
Depends on D19752. Ref T13210. If resources take a long time to reclaim/destroy (normally, more than 15 seconds) a single new lease may update several times during the reclaim/destroy process and end up reclaiming multiple resources.

Instead: after a lease triggers a reclaim, prevent it from triggering another reclaim as long as the resource it is reclaiming hasn't finished its reclaim/destroy cycle. Basically, each lease only gets to destroy one resource at a time.

Test Plan:
  - Added a `sleep(120)` to `destroyResource()` to simulate a long reclaim/destroy cycle.
  - Allocated A, A, A working copies. Leased a B working copy.
  - Before patch: saw "B" lease destroy all three "A" working copies after ~0, ~15, and ~30 seconds, then build a new "B" resource after ~120 seconds (when the first reclaim/destroy finished).
  - After patch: saw "B" lease destroy one "A" working copy after ~0 seconds, then wait patiently until it finished up, then build a new "B" resource.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19753

src/applications/drydock/worker/DrydockLeaseUpdateWorker.php

@@ -666,6 +666,26 @@ final class DrydockLeaseUpdateWorker extends DrydockWorker {
     DrydockLease $lease) {
     $viewer = $this->getViewer();
 
+    // If this lease is marked as already in the process of reclaiming a
+    // resource, don't let it reclaim another one until the first reclaim
+    // completes. This stops one lease from reclaiming a large number of
+    // resources if the reclaims take a while to complete.
+    $reclaiming_phid = $lease->getAttribute('drydock.reclaimingPHID');
+    if ($reclaiming_phid) {
+      $reclaiming_resource = id(new DrydockResourceQuery())
+        ->setViewer($viewer)
+        ->withPHIDs(array($reclaiming_phid))
+        ->withStatuses(
+          array(
+            DrydockResourceStatus::STATUS_ACTIVE,
+            DrydockResourceStatus::STATUS_RELEASED,
+          ))
+        ->executeOne();
+      if ($reclaiming_resource) {
+        return null;
+      }
+    }
+
     $resources = id(new DrydockResourceQuery())
       ->setViewer($viewer)
       ->withBlueprintPHIDs(array($blueprint->getPHID()))

src/applications/drydock/worker/DrydockWorker.php

@@ -241,16 +241,25 @@ abstract class DrydockWorker extends PhabricatorWorker {
     DrydockLease $lease) {
     $viewer = $this->getViewer();
 
+    // Mark the lease as reclaiming this resource. It won't be allowed to start
+    // another reclaim as long as this resource is still in the process of
+    // being reclaimed.
+    $lease->setAttribute('drydock.reclaimingPHID', $resource->getPHID());
+
     // When the resource releases, we we want to reawaken this task since it
-    // should be able to start building a new resource right away.
+    // should (usually) be able to start building a new resource right away.
     $worker_task_id = $this->getCurrentWorkerTaskID();
 
     $command = DrydockCommand::initializeNewCommand($viewer)
       ->setTargetPHID($resource->getPHID())
       ->setAuthorPHID($lease->getPHID())
       ->setCommand(DrydockCommand::COMMAND_RECLAIM)
-      ->setProperty('awakenTaskIDs', array($worker_task_id))
-      ->save();
+      ->setProperty('awakenTaskIDs', array($worker_task_id));
+
+    $lease->openTransaction();
+      $lease->save();
+      $command->save();
+    $lease->saveTransaction();
 
     $resource->scheduleUpdate();