1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-25 00:02:41 +01:00
phorge-phorge/src/applications
epriestley f0fdcf1a51 Undumb the Drydock resource allocator pipeline
Summary:
This was the major goal of D3859/D3855, and to a lesser degree D3854/D3852.

As Drydock is allocating a resource, it may need to allocate other resources first. For example, if it's allocating a working copy, it may need to allocate a host first.

Currently, we have the process basically queue up the allocation (insert a task into the queue) and sleep() until it finishes. This is problematic for a bunch of reasons, but the major one is that if allocation takes more resources (host, port, machine, DNS) than you have daemons, they could all end up sleeping and waiting for some other daemon to do their work. This is really stupid. Even if you only take up some of them, you're spending slots sleeping when you could be doing useful work.

To partially get around this and make the CLI experience less dumb, there's this goofy `synchronous` flag that gets passed around everywhere and pushes the workflow through a pile of special cases. Basically the `synchronous` flag causes us to do everything in-process. But this is dumb too because we'd rather do things in parallel if we can, and we have to have a lot of special case code to make it work at all.

Get rid of all of this. Instead of sleep()ing, try to work on the tasks that need to be worked on. If another daemon grabbed them already that's fine, but in the worst case we just gracefully degrade and do everything in process. So we get the best of both worlds: if we have parallelizable tasks and free daemons, things will execute in parallel. If we have nonparallelizable tasks or no free daemons, things will execute in process.

Test Plan: Ran `drydock_control.php --trace` and saw it perform cascading allocations without sleeping or special casing.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3861
2012-11-01 11:30:42 -07:00
..
audit Implement PhabricatorMarkupInterface in Diffusion/Audit comments 2012-10-23 17:46:44 -07:00
auth Declare used properties 2012-10-04 14:46:06 -07:00
base Remove "admin page" color/styles 2012-10-16 10:33:37 -07:00
cache/storage Add a generic multistep Markup cache 2012-07-09 15:20:56 -07:00
calendar Create a status tool by giving /calendar/ some teeth 2012-10-24 13:22:24 -07:00
chatlog Allow policy-aware queries to prefilter results 2012-10-23 12:01:11 -07:00
conduit add ids and phids to maniphest.query 2012-11-01 10:47:45 -07:00
countdown Require canonical numbers in routes 2012-10-05 18:07:54 -07:00
daemon Modernize worker task detail view 2012-10-31 15:22:32 -07:00
differential add a few more email preferences for differential and maniphest 2012-10-31 17:11:04 -07:00
diffusion make "browse in diffusion" action work for commits in branches other than master 2012-10-31 14:07:25 -07:00
directory/controller Optimize feed query 2012-10-04 16:07:39 -07:00
diviner Group applications into groups on /applications/ 2012-10-03 15:46:19 -07:00
draft/storage Don't store empty drafts 2012-10-01 16:05:39 -07:00
drydock Undumb the Drydock resource allocator pipeline 2012-11-01 11:30:42 -07:00
fact Allow Fact analysis of commits 2012-10-08 13:27:06 -07:00
feed Make feed stories properly respect object policies 2012-10-23 12:01:59 -07:00
files Add PhabricatorFileQuery 2012-10-31 09:57:46 -07:00
flag Require canonical numbers in routes 2012-10-05 18:07:54 -07:00
harbormaster/storage Add a generic object for unit tests 2012-07-09 10:39:14 -07:00
help/controller Move files in Phabricator one level up 2012-06-01 12:32:44 -07:00
herald Remove "admin page" color/styles 2012-10-16 10:33:37 -07:00
macro Improve image thumbnailing and increase the size of Macro thumbnails 2012-10-08 13:26:10 -07:00
mailinglists Require canonical numbers in routes 2012-10-05 18:07:54 -07:00
maniphest add ids and phids to maniphest.query 2012-11-01 10:47:45 -07:00
meta Use new application icons and intended side nav icons 2012-10-06 14:48:46 -07:00
metamta Factor out task execution and formalize permanent failures 2012-11-01 11:30:23 -07:00
notification Make feed stories properly respect object policies 2012-10-23 12:01:59 -07:00
oauthserver Remove all code referencing old tab navigation 2012-08-13 15:28:41 -07:00
owners Require canonical numbers in routes 2012-10-05 18:07:54 -07:00
paste Allow policy-aware queries to prefilter results 2012-10-23 12:01:11 -07:00
people Don't send the address verification email "From" the user in question 2012-10-31 09:57:40 -07:00
phame Allow blog resources to be served without Celerity 2012-10-17 08:37:05 -07:00
phid Fix a string in PhabricatorObjectHandleData 2012-10-22 11:31:37 -07:00
phortune Use pht() 2012-06-14 16:25:20 -07:00
phriction Raise a better error for trying to create an empty document in Phriction 2012-10-17 15:13:03 -07:00
policy Fix fatal with some pastes 2012-09-27 17:16:01 -07:00
ponder If users are on the email to Phabricator, do not send them the Phabricator reply. 2012-10-10 10:18:23 -07:00
project If users are on the email to Phabricator, do not send them the Phabricator reply. 2012-10-10 10:18:23 -07:00
repository Move completed tasks to an "archive" table and delete them in the GC 2012-10-31 15:22:16 -07:00
search If users are on the email to Phabricator, do not send them the Phabricator reply. 2012-10-10 10:18:23 -07:00
settings add a few more email preferences for differential and maniphest 2012-10-31 17:11:04 -07:00
slowvote Require canonical numbers in routes 2012-10-05 18:07:54 -07:00
status Move files in Phabricator one level up 2012-06-01 12:32:44 -07:00
subscriptions If users are on the email to Phabricator, do not send them the Phabricator reply. 2012-10-10 10:18:23 -07:00
typeahead Fix symbol search in typeahead 2012-10-20 05:03:41 -07:00
uiexample Show all available action list icons in UIExamples 2012-10-23 12:00:28 -07:00
xhpastview Require canonical numbers in routes 2012-10-05 18:07:54 -07:00
xhprof Do sampled profiling of requests 2012-09-17 10:53:45 -07:00