1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-23 15:22:41 +01:00
phorge-phorge/resources
epriestley 3ac99006bf Implement optimistic "slot locks" in Drydock
Summary:
See discussion in D10304. There's a lot of context there, but the general idea is:

  - Blueprints should manage locks in a granular way during the actual allocation/acquisition phase.
  - Optimistic "slot locks" might a pretty good primitive to make that easy to implement and reason about in most cases.

The way these locks work is that you just pick some name for the lock (like the PHID of a resource) and say that it needs to be acquired for the allocation/acquisition to work:

```
...
->needSlotLock("mylock(PHID-XYZQ-...)")
...
```

When you fire off the acquisition or allocation, it fails unless it could acquire the slot with that name. This is really simple (no explicit lock management) and a pretty good fit for most of the locking that blueprints and leases need to do.

If you need to do limit-based locks (e.g., maximum of 3 locks) you could acquire a lock like this:

```
mylock(whatever).slot(2)
```

Blueprints generally only contend with themselves, so it's normally OK for them to pick whatever strategy works best for them in naming locks.

This may not work as well if you have a huge number of slots (e.g., 100TB you want to give out in 1MB chunks), or other complex needs for locks (like you have to synchronize access to some external resource), but slot locks don't need to be the only mechanism that blueprints use. If they run into a problem that slot locks aren't a good fit for, they can use something else instead. For now, slot locks seem like a good fit for the problems we currently face and most of the problems I anticipate facing.

(The release workflows have other race issues which I'm not addressing here. They work fine if nothing races, but aren't race-safe.)

Test Plan:
To create a race where the same binding is allocated as a resource twice:

  - Add `sleep(10)` near the beginning of `allocateResource()`, after the free bindings are loaded but before resources are allocated.
  - (Comment out slot lock acquisition if you have this patch.)
  - Run `bin/drydock lease ...` in two windows, within 10 seconds of one another.

This will reliably double-allocate the binding because both blueprints see a view of the world where the binding is free.

To verify the lock works, un-comment it (or apply this patch) and run the same test again. Now, the lock fails in one process and only one resource is allocated.

Reviewers: hach-que, chad

Reviewed By: hach-que, chad

Differential Revision: https://secure.phabricator.com/D14118
2015-09-21 04:45:25 -07:00
..
builtin Make modular transforms handle exceptions gracefully 2015-05-13 11:38:45 -07:00
celerity Shrink aphront table headers 2015-09-19 19:42:19 -07:00
chatbot Remove PhabricatorBotDifferentialNotificationHandler reference in example_config.json for phabot 2014-06-11 09:36:30 -07:00
cows Use PHP implementation of Cowsay for cowsay rule 2015-09-13 12:27:30 -07:00
figlet/custom Include "Figlet" and PEAR "Text_Figlet" in externals 2015-09-13 12:30:48 -07:00
font Made Meme Generator 2013-01-19 18:43:43 -08:00
sprite Fix Bitbucket 2015-07-20 10:33:28 -07:00
sql Implement optimistic "slot locks" in Drydock 2015-09-21 04:45:25 -07:00
sshd Update repository hosting documentation for all the issues users have hit 2014-03-26 06:44:18 -07:00