1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-29 02:02:41 +01:00
phorge-phorge/src/applications/drydock/worker/DrydockResourceUpdateWorker.php

302 lines
7.4 KiB
PHP
Raw Normal View History

Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
<?php
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
/**
* @task update Updating Resources
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
* @task command Processing Commands
* @task activate Activating Resources
* @task release Releasing Resources
* @task break Breaking Resources
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
* @task destroy Destroying Resources
*/
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
final class DrydockResourceUpdateWorker extends DrydockWorker {
protected function doWork() {
$resource_phid = $this->getTaskDataValue('resourcePHID');
$hash = PhabricatorHash::digestForIndex($resource_phid);
$lock_key = 'drydock.resource:'.$hash;
$lock = PhabricatorGlobalLock::newLock($lock_key)
->lock(1);
try {
$resource = $this->loadResource($resource_phid);
$this->handleUpdate($resource);
} catch (Exception $ex) {
$lock->unlock();
Fix an issue where Drydock followup tasks would not queue if the main task failed Summary: Ref T9994. This fixes the first issue discussed on that task, which is that when a merge fails after "arc land", we would not clean up all the leases properly. Specifically, when a merge fails, we use `queueTask()` to schedule a followup task. This followup destroys the lease and frees the underlying resource. However, the default behavior of `queueTask()` is to //not queue tasks// if the parent task fails. This is a reasonable, safe behavior that was originally introduced in D8774, where it kept us from sending too much mail if a task did "send some mail" and then failed a little later on and got retried. Since I think the default behavior is correct, I just special cased the behavior for Drydock to make it queue even on failure. These are the only types of followup tasks we currently want to queue on main task failure. (It's possible that future Blueprints might want some kind of more specialized behavior, where some tasks queue only on success, but we can cross that bridge when we come to it.) Test Plan: - See T9994#149878 for test case setup. - I ran that test case again with this patch, and saw the followup task queue properly in the `--trace` log, a correspoinding update task show up in `/daemon/`, and the lease get destroyed when I ran it a moment later. {F1029915} Reviewers: chad Reviewed By: chad Maniphest Tasks: T9994 Differential Revision: https://secure.phabricator.com/D14818
2015-12-18 15:38:02 +01:00
$this->flushDrydockTaskQueue();
throw $ex;
}
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
$lock->unlock();
}
/* -( Updating Resources )------------------------------------------------- */
/**
* Update a resource, handling exceptions thrown during the update.
*
* @param DrydockReosource Resource to update.
* @return void
* @task update
*/
private function handleUpdate(DrydockResource $resource) {
try {
$this->updateResource($resource);
} catch (Exception $ex) {
if ($this->isTemporaryException($ex)) {
$this->yieldResource($resource, $ex);
} else {
$this->breakResource($resource, $ex);
}
}
}
/**
* Update a resource.
*
* @param DrydockResource Resource to update.
* @return void
* @task update
*/
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
private function updateResource(DrydockResource $resource) {
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
$this->processResourceCommands($resource);
$resource_status = $resource->getStatus();
switch ($resource_status) {
case DrydockResourceStatus::STATUS_PENDING:
$this->activateResource($resource);
break;
case DrydockResourceStatus::STATUS_ACTIVE:
// Nothing to do.
break;
case DrydockResourceStatus::STATUS_RELEASED:
case DrydockResourceStatus::STATUS_BROKEN:
$this->destroyResource($resource);
break;
case DrydockResourceStatus::STATUS_DESTROYED:
// Nothing to do.
break;
}
$this->yieldIfExpiringResource($resource);
}
/**
* Convert a temporary exception into a yield.
*
* @param DrydockResource Resource to yield.
* @param Exception Temporary exception worker encountered.
* @task update
*/
private function yieldResource(DrydockResource $resource, Exception $ex) {
$duration = $this->getYieldDurationFromException($ex);
$resource->logEvent(
DrydockResourceActivationYieldLogType::LOGCONST,
array(
'duration' => $duration,
));
throw new PhabricatorWorkerYieldException($duration);
}
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
/* -( Processing Commands )------------------------------------------------ */
/**
* @task command
*/
private function processResourceCommands(DrydockResource $resource) {
if (!$resource->canReceiveCommands()) {
return;
}
$this->checkResourceExpiration($resource);
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
$commands = $this->loadCommands($resource->getPHID());
foreach ($commands as $command) {
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
if (!$resource->canReceiveCommands()) {
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
break;
}
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
$this->processResourceCommand($resource, $command);
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
$command
->setIsConsumed(true)
->save();
}
}
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
/**
* @task command
*/
private function processResourceCommand(
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
DrydockResource $resource,
DrydockCommand $command) {
switch ($command->getCommand()) {
case DrydockCommand::COMMAND_RELEASE:
Make Drydock reclaim unused resources when it reaches a resource limit Summary: Fixes T9994. Currently, when Drydock can't allocate a new resource because some limit has been reached, it waits patiently for a resource to become available. It is possible that no resource will ever become available. Particularly with "Working Copy" resources, the new lease may want a copy of `rB`, but the resource may already be maxed out on `rA`. Right now, no process exists to automatically reclaim the unused `rA`. When we encounter this situation, try to reclaim one of the other resources if it is just sitting there unused. Specifically: - Add a "reclaim" command which means "release this resource //if// it is completely unused". - Add a `bin/drydock reclaim` to send this command to every active resource. - When we try to acquire a resource and can't, but only because of some kind of limit / utilization problem, try to release an unused resource to free up some room. Test Plan: - Set "Working Copy" resource limit to 1. - Ran "Test Configuration" in `rA`, which worked. - Ran "Test Configuration" in `rB`, which hung forever. - Applied patch. - Ran "Test Configuration" in `rB`, saw it reclaim the `rA` resource, use the slot, then succeed. - Ran "Test Configuration" in `rA` again, saw it grab the slot back. - Ran `bin/drydock reclaim` and saw it reclaim a bunch of old orphaned resources. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9994 Differential Revision: https://secure.phabricator.com/D14819
2015-12-18 17:44:13 +01:00
$this->releaseResource($resource, null);
break;
case DrydockCommand::COMMAND_RECLAIM:
$reclaimer_phid = $command->getAuthorPHID();
$this->releaseResource($resource, $reclaimer_phid);
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
break;
}
}
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
/* -( Activating Resources )----------------------------------------------- */
/**
* @task activate
*/
private function activateResource(DrydockResource $resource) {
$blueprint = $resource->getBlueprint();
$blueprint->activateResource($resource);
$this->validateActivatedResource($blueprint, $resource);
}
/**
* @task activate
*/
private function validateActivatedResource(
DrydockBlueprint $blueprint,
DrydockResource $resource) {
if (!$resource->isActivatedResource()) {
throw new Exception(
pht(
'Blueprint "%s" (of type "%s") is not properly implemented: %s '.
'must actually allocate the resource it returns.',
$blueprint->getBlueprintName(),
$blueprint->getClassName(),
'allocateResource()'));
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
}
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
}
/* -( Releasing Resources )------------------------------------------------ */
/**
* @task release
*/
Make Drydock reclaim unused resources when it reaches a resource limit Summary: Fixes T9994. Currently, when Drydock can't allocate a new resource because some limit has been reached, it waits patiently for a resource to become available. It is possible that no resource will ever become available. Particularly with "Working Copy" resources, the new lease may want a copy of `rB`, but the resource may already be maxed out on `rA`. Right now, no process exists to automatically reclaim the unused `rA`. When we encounter this situation, try to reclaim one of the other resources if it is just sitting there unused. Specifically: - Add a "reclaim" command which means "release this resource //if// it is completely unused". - Add a `bin/drydock reclaim` to send this command to every active resource. - When we try to acquire a resource and can't, but only because of some kind of limit / utilization problem, try to release an unused resource to free up some room. Test Plan: - Set "Working Copy" resource limit to 1. - Ran "Test Configuration" in `rA`, which worked. - Ran "Test Configuration" in `rB`, which hung forever. - Applied patch. - Ran "Test Configuration" in `rB`, saw it reclaim the `rA` resource, use the slot, then succeed. - Ran "Test Configuration" in `rA` again, saw it grab the slot back. - Ran `bin/drydock reclaim` and saw it reclaim a bunch of old orphaned resources. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9994 Differential Revision: https://secure.phabricator.com/D14819
2015-12-18 17:44:13 +01:00
private function releaseResource(
DrydockResource $resource,
$reclaimer_phid) {
if ($reclaimer_phid) {
if (!$this->canReclaimResource($resource)) {
return;
}
$resource->logEvent(
DrydockResourceReclaimLogType::LOGCONST,
array(
'reclaimerPHID' => $reclaimer_phid,
));
}
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
$viewer = $this->getViewer();
$drydock_phid = id(new PhabricatorDrydockApplication())->getPHID();
$resource
->setStatus(DrydockResourceStatus::STATUS_RELEASED)
->save();
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
$statuses = array(
DrydockLeaseStatus::STATUS_PENDING,
DrydockLeaseStatus::STATUS_ACQUIRED,
DrydockLeaseStatus::STATUS_ACTIVE,
);
$leases = id(new DrydockLeaseQuery())
->setViewer($viewer)
->withResourcePHIDs(array($resource->getPHID()))
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
->withStatuses($statuses)
->execute();
foreach ($leases as $lease) {
$command = DrydockCommand::initializeNewCommand($viewer)
->setTargetPHID($lease->getPHID())
->setAuthorPHID($drydock_phid)
->setCommand(DrydockCommand::COMMAND_RELEASE)
->save();
$lease->scheduleUpdate();
}
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
$this->destroyResource($resource);
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
}
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
/* -( Breaking Resources )------------------------------------------------- */
/**
* @task break
*/
private function breakResource(DrydockResource $resource, Exception $ex) {
switch ($resource->getStatus()) {
case DrydockResourceStatus::STATUS_BROKEN:
case DrydockResourceStatus::STATUS_RELEASED:
case DrydockResourceStatus::STATUS_DESTROYED:
// If the resource was already broken, just throw a normal exception.
// This will retry the task eventually.
throw new PhutilProxyException(
pht(
'Unexpected failure while destroying resource ("%s").',
$resource->getPHID()),
$ex);
}
$resource
->setStatus(DrydockResourceStatus::STATUS_BROKEN)
->save();
$resource->scheduleUpdate();
$resource->logEvent(
DrydockResourceActivationFailureLogType::LOGCONST,
array(
'class' => get_class($ex),
'message' => $ex->getMessage(),
));
throw new PhabricatorWorkerPermanentFailureException(
pht(
'Permanent failure while activating resource ("%s"): %s',
$resource->getPHID(),
$ex->getMessage()));
}
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
/* -( Destroying Resources )----------------------------------------------- */
/**
* @task destroy
*/
private function destroyResource(DrydockResource $resource) {
$blueprint = $resource->getBlueprint();
$blueprint->destroyResource($resource);
DrydockSlotLock::releaseLocks($resource->getPHID());
Merge the DrydockResource workers into a single worker Summary: Ref T9252. Currently, Drydock Leases and Resources have several workers: - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker This is kind of a lot of stuff, and it creates some problems. In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it. Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands. A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler. This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers. Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly. Reviewers: chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 17:10:40 +02:00
$resource
->setStatus(DrydockResourceStatus::STATUS_DESTROYED)
->save();
}
Add a command queue to Drydock to manage lease/resource release Summary: Ref T9252. Broadly, Drydock currently races on releasing objects from the "active" state. To reproduce this: - Scatter some sleep()s pretty much anywhere in the release code. - Release several times from web UI or CLI in quick succession. Resources or leases will execute some release code twice or otherwise do inconsistent things. (I didn't chase down a detailed reproduction scenario for this since inspection of the code makes it clear that there are no meaningful locks or mechanisms preventing this.) Instead, add a Harbormaster-style command queue to resources and leases. When something wants to do a release, it adds a command to the queue and schedules a worker. The workers acquire a lock, then try to consume commands from the queue. This guarantees that only one process is responsible for writes to active resource/leases. This is the last major step to giving resources and leases a single writer during all states: - Resource, Unsaved: AllocatorWorker - Resource, Pending: ResourceWorker (Possible rename to "Allocated?") - Resource, Open: This diff, ResourceUpdateWorker. (Likely rename to "Active"). - Resource, Closed/Broken: Future destruction worker. (Likely rename to "Released" / "Broken"; maybe remove "Broken"). - Resource, Destroyed: No writes. - Lease, Unsaved: Whatever wants the lease. - Lease, Pending: AllocatorWorker - Lease, Acquired: LeaseWorker - Lease, Active: This diff, LeaseUpdateWorker. - Lease, Released/Broken: Future destruction worker (Maybe remove "Broken"?) - Lease, Expired: No writes. (Likely rename to "Destroyed"). In most phases, we can already guarantee that there is a single writer without doing any extra work. This is more complicated in the "Active" case because the release buttons on the web UI, the release tools on the CLI, the lease requestor itself, the garbage collector, and any other release process cleaning up related objects may try to effect a release. All of these could race one another (and, in many cases, race other processes from other phases because all of these get to act immediately) as this code is currently written. Using a queue here lets us make sure there's only a single writer in this phase. One thing which is notable is that whatever acquires a lease **can not write to it**! It is never the writer once it queues the lease for activation. It can not write to any resources, either. And, likewise, Blueprints can not write to resources while acquiring or releasing leases. We may need to provide a mechinism so that blueprints and/or resource/lease holders get to attach some storage to resources/leases for bookkeeping. For example, a blueprint might need to keep some kind of cache on a resource to help it manage state. But I think we can cross that bridge when we come to it, and nothing else would need to write to this storage so it's technically straightforward to introduce such a mechanism if we need one. Test Plan: - Viewed buttons in web UI, checked enabled/disabled states. - Clicked the buttons. - Saw commands show up in the command queue. - Saw some daemon stuff get scheduled. - Ran CLI tools, saw commands get consumed and resources/leases release. Reviewers: hach-que, chad Reviewed By: chad Maniphest Tasks: T9252 Differential Revision: https://secure.phabricator.com/D14143
2015-09-23 16:42:08 +02:00
}