1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-28 01:32:42 +01:00
phorge-phorge/src/applications/drydock/exception
epriestley a1baedbd9a Lock resources briefly while acquiring leases on them to prevent acquiring near-death resources
Summary:
Depends on D19078. Ref T13073. Currently, there is a narrow window where we can acquire a resource after a reclaim has started against it.

To prevent this, briefly lock resources before acquiring them and make sure they're still good. If a resource isn't good, throw the lease back in the pool.

Test Plan:
This is tricky. You need:

  - Hoax blueprint with limits and a rule where leases of a given "flavor" can only be satisfied by resources of the same flavor.
  - Reduce the 3-minute "wait before resources can be released" to 3 seconds.
  - Limit Hoaxes to 1.
  - Allocate one "cherry" flavored Hoax and release the lease.
  - Add a `sleep(15)` to `releaseResource()` in `DrydockResourceUpdateWorker`, after the `canReclaimResource()` check, with a `print`.

Now:

  - Run `bin/phd debug task` in two windows.
  - Run `bin/drydock lease --type host --attributes flavor=banana` in a third window.
  - This will start to reclaim the existing "cherry" resource. Once one of the `phd` windows prints the "RECLAIMING" message run `bin/drydock lease --type host --attributes flavor=cherry` in a fourth window.
  - Before patch: the "cherry" lease acquired immediately, then was released and destroyed moments later.
  - After patch: the "cherry" lease yields.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19080
2018-02-13 13:22:13 -08:00
..
DrydockAcquiredBrokenResourceException.php When a lease acquires a resource but the resource fails to activate, throw the lease back in the pool 2018-02-13 13:17:54 -08:00
DrydockCommandError.php Improve Drydock errors for empty commits and missing changes 2016-03-07 05:29:25 -08:00
DrydockResourceLockException.php Lock resources briefly while acquiring leases on them to prevent acquiring near-death resources 2018-02-13 13:22:13 -08:00
DrydockSlotLockException.php Add more Drydock log types and some additional logging 2015-10-01 08:11:42 -07:00