1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-13 09:06:14 +01:00
Commit graph

14 commits

Author SHA1 Message Date
Joshua Spence
368f359114 Use PhutilClassMapQuery instead of PhutilSymbolLoader
Summary: Use `PhutilClassMaQuery` instead of `PhutilSymbolLoader`, mostly for consistency. Depends on D13588.

Test Plan: Poked around a bunch of pages.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D13589
2015-08-14 07:49:01 +10:00
Joshua Spence
36e2d02d6e phtize all the things
Summary: `pht`ize a whole bunch of strings in rP.

Test Plan: Intense eyeballing.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: hach-que, Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D12797
2015-05-22 21:16:39 +10:00
epriestley
ce78bf1de4 Make all bin/* scripts locate their workflows dynamically
Summary:
Ref T2015. Not directly related to Drydock, but I bumped into this. All these scripts currently enumerate their workflows explicitly.

Instead, use `PhutilSymbolLoader` to automatically discover workflows. This reduces code duplication and errors (see all the bad `extends` this diff fixes) and lets third parties add new workflows (not clearly valuable?).

Test Plan: Ran `bin/x help` for each modified script.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D7840
2013-12-27 13:15:48 -08:00
epriestley
1650874004 Modernize Drydock CLI management of task execution
Summary:
Ref T2015. Currently, Drydock has a `wait-for-lease` workflow which is invoked in the background by the `lease` workflow.

The goal of this mechanism is to allow `bin/drydock lease` to print out logs as the lease is acquired. However, this predates the `runAllTasksInProcess` flags, and they provide a simpler and more robust way (potentially with `--trace` and `PhutilConsole`) to do synchronous execution and debug logging.

Simplify this whole mechanism: just run everything in-process in `bin/drydock lease`, and do logging via `--trace`. We could thread a `PhutilConsole` through things too, but this seems good enough for now.

Also various cleanup/etc.

Test Plan: Ran `bin/drydock lease`. Ran `bin/harbormaster build X --plan Y`, for `Y` being a Drydock-dependent build plan.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D7835
2013-12-27 13:15:12 -08:00
James Rhodes
7c3cb5948c Drydock blueprint for preallocated remote hosts
Summary:
This adds a Drydock blueprint for preallocated, remote hosts.  This will be used by the Harbormaster interface to allow users to specify remote hosts that builds can be run on.

This adds a `canAllocateResource` method to Drydock blueprints; it is used to detect whether a blueprint can allocate a resource for the given type and attributes.

Test Plan:
Ran:

```
bin/drydock lease --type host --attributes remote=true,preallocated=true,host=192.168.56.101,port=22,user=james,keyfile=,path=C:\\Build\\,platform=windows
```

and saw the "C:\Build\<id>" folder appear on the remote Windows machine.  Viewed the lease and resource in Drydock as well.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran, jamesr

Maniphest Tasks: T4111

Differential Revision: https://secure.phabricator.com/D7593
2013-11-22 14:34:10 -08:00
epriestley
e4bb9255be Allow leases to be explicitly released via web or CLI
Summary: Permit the forcible release of Drydock leases. The implementation isn't very exciting for now.

Test Plan: Released leases via web and CLI.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D4181
2012-12-14 15:42:58 -08:00
epriestley
5cbc31644b Add a "close" action to Drydock resources
Summary: This does nothing fancy, just closes the resource and releases/breaks leases. They'll get cleaned up in some to-be-written GC process.

Test Plan: Closed resources from web UI and CLI.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3998
2012-11-27 12:48:03 -08:00
vrana
ef85f49adc Delete license headers from files
Summary:
This commit doesn't change license of any file. It just makes the license implicit (inherited from LICENSE file in the root directory).

We are removing the headers for these reasons:

- It wastes space in editors, less code is visible in editor upon opening a file.
- It brings noise to diff of the first change of any file every year.
- It confuses Git file copy detection when creating small files.
- We don't have an explicit license header in other files (JS, CSS, images, documentation).
- Using license header in every file is not obligatory: http://www.apache.org/dev/apply-license.html#new.

This change is approved by Alma Chao (Lead Open Source and IP Counsel at Facebook).

Test Plan: Verified that the license survived only in LICENSE file and that it didn't modify externals.

Reviewers: epriestley, davidrecordon

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2035

Differential Revision: https://secure.phabricator.com/D3886
2012-11-05 11:16:51 -08:00
epriestley
07dc943215 Modernize the drydock script
Summary: Add a bin/drydock symlink and break it into workflows. Nothing too special here.

Test Plan: Ran `bin/drydock wait-for-lease`, `bin/drydock lease`, `bin/drydock help`, etc.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3867
2012-11-01 15:30:14 -07:00
epriestley
f0fdcf1a51 Undumb the Drydock resource allocator pipeline
Summary:
This was the major goal of D3859/D3855, and to a lesser degree D3854/D3852.

As Drydock is allocating a resource, it may need to allocate other resources first. For example, if it's allocating a working copy, it may need to allocate a host first.

Currently, we have the process basically queue up the allocation (insert a task into the queue) and sleep() until it finishes. This is problematic for a bunch of reasons, but the major one is that if allocation takes more resources (host, port, machine, DNS) than you have daemons, they could all end up sleeping and waiting for some other daemon to do their work. This is really stupid. Even if you only take up some of them, you're spending slots sleeping when you could be doing useful work.

To partially get around this and make the CLI experience less dumb, there's this goofy `synchronous` flag that gets passed around everywhere and pushes the workflow through a pile of special cases. Basically the `synchronous` flag causes us to do everything in-process. But this is dumb too because we'd rather do things in parallel if we can, and we have to have a lot of special case code to make it work at all.

Get rid of all of this. Instead of sleep()ing, try to work on the tasks that need to be worked on. If another daemon grabbed them already that's fine, but in the worst case we just gracefully degrade and do everything in process. So we get the best of both worlds: if we have parallelizable tasks and free daemons, things will execute in parallel. If we have nonparallelizable tasks or no free daemons, things will execute in process.

Test Plan: Ran `drydock_control.php --trace` and saw it perform cascading allocations without sleeping or special casing.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3861
2012-11-01 11:30:42 -07:00
vrana
1ebf9186b4 Depend on class autoloading
Test Plan:
Run setup.
/differential/

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Koolvin

Maniphest Tasks: T1103

Differential Revision: https://secure.phabricator.com/D2612
2012-05-30 16:57:21 -07:00
epriestley
914f044b62 More Drydock Stuff
Summary:
  - Still really really rough.
  - Adds a full synchronous mode for debugging.
  - Adds some logging.
  - It can now allocate EC2 machines and put webroots on them in a hacky, terrible way.
  - Adds a base query class.

Test Plan: oh hey look a test page? http://ec2-50-18-65-151.us-west-1.compute.amazonaws.com:2011/

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D2026
2012-03-26 20:54:26 -07:00
epriestley
7d1f62409d Move resource allocation to task queue
Summary: Run the actual resource allocation for Drydock out-of-process via the
task queue.

Test Plan: Ran "drydock_control.php", saw it insert a task and wait for task
completion. Ran "phd debug taskmaster" and saw it run the task.

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1470
2012-01-24 09:44:14 -08:00
epriestley
d1ee08b2df Drydock Rough Cut
Summary:
Rough cut of Drydock. This is very basic and doesn't do much of use yet (it
//does// allocate EC2 machines as host resources and expose interfaces to them),
but I think the overall structure is more or less reasonable.

== Interfaces

Vision: Applications interact with Drydock resources through DrydockInterfaces,
like **command**, **filesystem** and **httpd** interfaces. Each interface allows
applications to perform some kind of operation on the resource, like executing
commands, reading/writing files, or configuring a web server. Interfaces have a
concrete, specific API:

  // Filesystem Interface
  $fs = $lease->getInterface('filesystem'); // Constants, some day?
  $fs->writeFile('index.html', 'hello world!');

  // Command Interface
  $cmd = $lease->getInterface('command');
  echo $cmd->execx('uptime');

  // HTTPD Interface
  $httpd = $lease->getInterface('httpd');
  $httpd->restart();

Interfaces are mostly just stock, although installs might add new interfaces if
they expose different ways to interact with resources (for instance, a resource
might want to expose a new 'MongoDB' interface or whatever).

Currently: We have like part of a command interface.

== Leases

Vision: Leases keep track of which resources are in use, and what they're being
used for. They allow us to know when we need to allocate more resources (too
many sandcastles on the existing hosts, e.g.) and when we can release resources
(because they are no longer being used). They also give applications something
to hold while resources are being allocated.

  // EXAMPLE: How this should work some day.
  $allocator = new DrydockAllocator();
  $allocator->setResourceType('sandcastle');
  $allocator->setAttributes(
    array(
      'diffID' => $diff->getID(),
    ));
  $lease = $allocator->allocate();
  $diff->setSandcastleLeaseID($lease->getID());

  // ...

  if ($lease->getStatus() == DrydockLeaseStatus::STATUS_ACTIVE) {
    $sandcastle_link = $lease->getInterface('httpd')->getURI('/');
  } else {
    $sandcastle_link = 'Still building your sandcastle...';
  }
  echo "Sandcastle for this diff: ".$sandcastle_link;

  // EXAMPLE: How this actually works now.
  $allocator = new DrydockAllocator();
  $allocator->setResourceType('host');
  // NOTE: Allocation is currently synchronous but will be task-driven soon.
  $lease = $allocator->allocate();

Leases are completely stock, installs will not define new lease types.

Currently: Leases exist and work but are very very basic.

== Resources

Vision: Resources represent some actual thing we've put somewhere, whether it's
a host, a block of storage, a webroot, or whatever else. Applications interact
through resources by acquiring leases to them, and then getting interfaces
through these leases. The lease acquisition process has a side effect of
allocating new resources if a lease can't be acquired on existing resources
(e.g., the application wants storage but all storage resources are full) and
things are configured to autoscale.

Resources may themselves acquire leases in order to allocate. For instance, a
storage resource might first acquire a lease to a host resource. A 'test
scaffold' resource might lease a storage resource and a mysql resource.

Not all resources are auto-allocate: the entry-level version of Drydock is that
you manually allocate a couple boxes and configure them through the web console.
Then, e.g.,  'storage' / 'webroot' resources allocate on top of them, but the
host pool itself does not autoscale.

Resources are completely stock, they are abstract shells representing any
arbitrary thing.

Currently: Resource exist ('host' only) but are very very basic.

== Blueprints

Vision: Blueprints contain instructions for building interfaces to, (possibly)
allocating, updating, managing, and destroying a specific type of resource in a
specific location. One way to think of them is that they are scripts for
creating and deleting resources. For example, the LocalHost, RemoteHost and
EC2Host blueprints can all manage 'host' resources.

Eventually, we will support more types of resources (storage, webroot,
sandcastle, test scaffold, phacility deployment) and more providers for resource
types, some of which will be in the Phabricator mainline and some of which will
be custom.

Blueprints are very custom and specific to application types, so installs will
define new blueprints if they are making significant use of Drydock.

Currently: They exist but have few capabilities. The stock blueprints do nearly
nothing useful. There is a technically functional blueprint for host allocation
in EC2.

== Allocator

This is just the actual code to execute the lease acquisition process.

Test Plan: Ran "drydock_control.php" script, it allocated a machine in EC2,
acquired a lease on it, interfaced with it, and then released the lease. Ran it
again, got a fresh lease on the existing resource.

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D1454
2012-01-19 21:12:57 -08:00