mirror of
https://we.phorge.it/source/phorge.git
synced 2024-12-25 06:50:55 +01:00
Fix an issue where newly created Drydock resources could be improperly acquired
Summary: Ref T9252. This is mostly a fix for an edge case from D14236. Here's the setup: - There are no resources. - A request for a new resource arrives. - We build a new resource. Now, if we were leasing an existing resource, we'd call `canAcquireLeaseOnResource()` before acquiring a lease on the new resource. However, for new resources we don't do that: we just acquire a lease immediately. This is wrong, because we now allow and expect some resources to be unleasable when created. In a more complex workflow, this can also produce the wrong result and leave the lease acquired sub-optimally (and, today, deadlocked). Make the "can we acquire?" pathway consistent for new and existing resources, so we always do the same set of checks. Test Plan: - Started daemons. - Deleted all working copy resources. - Ran two working-copy-using build plans at the same time. - Before this change, one would often [1] acquire a lease on a pending resource which never allocated, then deadlock. - After this change, the same thing happens except that the lease remains pending and the work completes. [1] Although the race this implies is allowed (resource pool limits are soft/advisory, and it is expected that we may occasionally run over them), it's MUCH easier to hit right now than I would expect it to be, so I think there's probably at least one more small bug here somewhere. I'll see if I can root it out after this change. Reviewers: chad, hach-que Reviewed By: hach-que Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14272
This commit is contained in:
parent
4169d7bfd5
commit
083a321dad
1 changed files with 27 additions and 2 deletions
|
@ -211,10 +211,19 @@ final class DrydockLeaseUpdateWorker extends DrydockWorker {
|
||||||
$exceptions);
|
$exceptions);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
$resources = $this->removeUnacquirableResources($resources, $lease);
|
||||||
|
if (!$resources) {
|
||||||
|
// If we make it here, we just built a resource but aren't allowed
|
||||||
|
// to acquire it. We expect this during routine operation if the
|
||||||
|
// resource prevents acquisition until it activates. Yield and wait
|
||||||
|
// for activation.
|
||||||
|
throw new PhabricatorWorkerYieldException(15);
|
||||||
|
}
|
||||||
|
|
||||||
// NOTE: We have not acquired the lease yet, so it is possible that the
|
// NOTE: We have not acquired the lease yet, so it is possible that the
|
||||||
// resource we just built will be snatched up by some other lease before
|
// resource we just built will be snatched up by some other lease before
|
||||||
// we can. This is not problematic: we'll retry a little later and should
|
// we can acquire it. This is not problematic: we'll retry a little later
|
||||||
// suceed eventually.
|
// and should suceed eventually.
|
||||||
}
|
}
|
||||||
|
|
||||||
$resources = $this->rankResources($resources, $lease);
|
$resources = $this->rankResources($resources, $lease);
|
||||||
|
@ -382,6 +391,22 @@ final class DrydockLeaseUpdateWorker extends DrydockWorker {
|
||||||
))
|
))
|
||||||
->execute();
|
->execute();
|
||||||
|
|
||||||
|
return $this->removeUnacquirableResources($resources, $lease);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Remove resources which can not be acquired by a given lease from a list.
|
||||||
|
*
|
||||||
|
* @param list<DrydockResource> Candidate resources.
|
||||||
|
* @param DrydockLease Acquiring lease.
|
||||||
|
* @return list<DrydockResource> Resources which the lease may be able to
|
||||||
|
* acquire.
|
||||||
|
* @task allocator
|
||||||
|
*/
|
||||||
|
private function removeUnacquirableResources(
|
||||||
|
array $resources,
|
||||||
|
DrydockLease $lease) {
|
||||||
$keep = array();
|
$keep = array();
|
||||||
foreach ($resources as $key => $resource) {
|
foreach ($resources as $key => $resource) {
|
||||||
$blueprint = $resource->getBlueprint();
|
$blueprint = $resource->getBlueprint();
|
||||||
|
|
Loading…
Reference in a new issue