1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-28 17:52:43 +01:00
Commit graph

70 commits

Author SHA1 Message Date
epriestley
f48a833704 Fix an issue with incorrect authorization handling in Working Copy build steps
Summary:
Fixes T9669. Two issues:

  - We were using `repositoryPHIDs` instead of `blueprintPHIDs` for the list of allowed blueprints. Use the correct value.
  - We weren't enforcing `allowedBlueprintPHIDs` fully correctly. We //did// require an authorization, so the net effect was correct in nearly all cases, but we could have selected from too large a pool in the case where the application itself was doing the authorization (e.g., from the command line).

Test Plan: Ran a build through Drydock/Harbormaster locally.

Reviewers: chad, tycho.tatitscheff

Reviewed By: chad, tycho.tatitscheff

Subscribers: tycho.tatitscheff

Maniphest Tasks: T9669

Differential Revision: https://secure.phabricator.com/D14368
2015-10-30 16:02:35 +00:00
epriestley
43569d4e27 Make WorkingCopy build step slightly more durable
Summary: Fixes T9631. Build steps created before I added this option may not have it specified, which could throw later. Make handling a little more robust.

Test Plan: Will ask @yelirekim to report back.

Reviewers: chad

Reviewed By: chad

Subscribers: yelirekim

Maniphest Tasks: T9631

Differential Revision: https://secure.phabricator.com/D14336
2015-10-25 14:53:24 -07:00
epriestley
1bdf225354 Use Drydock authorizations when acquiring leases
Summary:
Ref T9519. When acquiring leases on resources:

  - Only consider resources created by authorized blueprints.
  - Only consider authorized blueprints when creating new resources.
  - Fail with a tailored error if no blueprints are allowed.
  - Fail with a tailored error if missing authorizations are causing acquisition failure.

One somewhat-substantial issue with this is that it's pretty hard to figure out from the Harbormaster side. Specifically, the Build step UI does not show field value anywhere, so the presence of unapproved blueprints is not communicated. This is much more clear in Drydock. I'll plan to address this in future changes to Harbormaster, since there are other related/similar issues anyway.

Test Plan: {F872527}

Reviewers: hach-que, chad

Reviewed By: chad

Maniphest Tasks: T9519

Differential Revision: https://secure.phabricator.com/D14254
2015-10-12 17:02:35 -07:00
epriestley
4d5278af11 Put Drydock build steps into their own group in Harbormaster
Summary: Ref T9252. Move these into a new "Drydock" group.

Test Plan: Clicked "Add Build Step", saw Drydock steps in a Drydock group.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14237
2015-10-05 15:59:35 -07:00
epriestley
4cf1270ecd In Harbormaster, make sure artifacts are destroyed even if a build is aborted
Summary:
Ref T9252. Currently, Harbormaster and Drydock work like this in some cases:

  # Queue a lease for activation.
  # Then, a little later, save the lease PHID somewhere.
  # When the target/resource is destroyed, destroy the lease.

However, something can happen between (1) and (2). In Drydock this window is very short and the "something" would have to be a lighting strike or something similar, but in Harbormaster we wait until the resource activates to do (2) so the window can be many minutes long. In particular, a user can use "Abort Build" during those many minutes.

If they do, the target is destroyed but it doesn't yet have a record of the artifact, so the artifact isn't cleaned up.

Make these things work like this instead:

  # Create a new lease and pre-generate a PHID for it.
  # Save that PHID as something that needs to be cleaned up.
  # Queue the lease for activation.
  # When the target/resource is destroyed, destroy the lease if it exists.

This makes sure there's no step in the process where we might lose track of a lease/resource.

Also, clean up and standardize some other stuff I hit.

Test Plan:
  - Stopped daemons.
  - Restarted a build in Harbormaster.
  - Stepped through the build one stage at a time using `bin/worker execute ...`.
  - After the lease was queued, but before it activated, aborted the build.
  - Processed the Harbormaster side of things only.
  - Saw the lease get destroyed properly.

Reviewers: chad, hach-que

Reviewed By: hach-que

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14234
2015-10-05 05:58:53 -07:00
epriestley
14d6325394 Acccept any HTTP 2xx status as success in Harbormaster
Summary: Ref T9478. This should probably be configurable eventually, but for now treat any 200-block status as success. Also show the result code.

Test Plan:
  - Hit a bad URI, saw "HTTP 503" + failure.
  - Hit a good URI, saw "HTTP 200" + success.

Reviewers: chad, hach-que

Reviewed By: chad, hach-que

Maniphest Tasks: T9478

Differential Revision: https://secure.phabricator.com/D14226
2015-10-02 09:17:51 -07:00
epriestley
bb4667cb84 Fix WorkingCopy step to read correct commit variables
Summary: Ref T9252. This variable was always wrong but we fell back to just resetting to `HEAD` before. Use the correct variable name.

Test Plan: Verified variable name.

Reviewers: chad, hach-que

Reviewed By: hach-que

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14224
2015-10-02 06:37:17 -07:00
epriestley
2728a9f964 Allow builds to have parameters
Summary: Ref T9352. See D13635. Build targets can have variables already, but let builds have them too. This mostly enables future use cases (sub-builds, more sophisticated build triggers).

Test Plan: With a custom Herald rule + action like the one in T9352, updated a revision and saw it generate multiple builds with varying parameters.

Reviewers: chad, hach-que

Reviewed By: hach-que

Maniphest Tasks: T9352

Differential Revision: https://secure.phabricator.com/D14222
2015-10-02 06:32:08 -07:00
epriestley
4496176924 Add staging area support to Harbormaster/Drydock + various fixes
Summary:
Ref T9252. This primarily allows Harbormaster to request (and Drydock to fulfill) working copies with a patch from a staging area. Doing this means we can do builds on in-review changes from `arc diff`.

This is a little cobbled-together but should basically work.

Also fix some other issues:

  - Yielded, awakend workers are fine to update but could complain.
  - We can't log slot lock failures to resources if we don't end up saving them.
  - Killing the transaction would wipe out the log.
  - Fix some TODOs, etc.

Test Plan: Ran Harbormaster builds on a local revision.

Reviewers: hach-que, chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14214
2015-10-01 16:55:01 -07:00
epriestley
bfaa93aa9b Allow Harbormaster build plans to request additional working copies
Summary:
Ref T9123. To run upstream builds in Harbormaster/Drydock, we need to be able to check out `libphutil`, `arcanist` and `phabricator` next to one another.

This adds an "Also Clone: ..." field to Harbormaster working copy build steps so I can type all three repos into it and get a proper clone with everything we need.

This is somewhat upstream-centric and a bit narrow, but I don't think it's totally unreasonable, and most of the underlying stuff is relatively general.

This adds some more typechecking and improves data/type handling for custom fields, too. In particular, it prevents users from entering an invalid/restricted value in a field (for example, you can't "Also Clone" a repository you don't have permission to see).

Test Plan: Restarted build, got a Drydock resource with multiple repositories in it.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9123

Differential Revision: https://secure.phabricator.com/D14183
2015-09-28 17:57:41 -07:00
epriestley
33be8f719f Allow WorkingCopy resources to have multiple working copies
Summary:
Ref T9252. For building Phabricator itself, we need to have `libphutil/`, `arcanist/` and `phabricator/` next to one another on disk.

Expand the Drydock WorkingCopy resource so that it can have multiple repositories if the caller needs them.

I'm not sure if I'm going to put the actual config for this in Harbormaster or Drydock yet, but the WorkingCopy resource itself should work the same way in either case.

Test Plan: Restarted a Harbormaster build which leases a working copy, saw it build as expected.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14180
2015-09-28 09:35:58 -07:00
epriestley
9b29d46e60 Make Drydock lease infrastructure more nimble
Summary:
Ref T9252. Currently, Harbormaster does this when trying to acquire a working copy:

  - Ask for a working copy.
  - Yield for 15 seconds.
  - Check if we have a working copy yet.

That's OK, but Drydock takes ~1s to acquire a working copy lease if a resource is already available, so we end up doing this:

  - T+0: Ask for a working copy.
  - T+0: Yield for 15 seconds.
  - T+1: Working copy lease activates.
  - T+15: Working copy lease is used.
  - T+16: Build finishes.

So we end up spending about 2 seconds doing work and 14 seconds sleeping.

One way to fix this would be to fiddle with the yield duration, so we yield for 1, 2, 4, ... seconds or something. This probably isn't a bad idea for longer leases (i.e., wait for 15, 30, 45 ... seconds or similar) but it implies a lot of churn for short leases.

Instead, let tasks "awaken" other tasks when they complete. The "awaken" operation means: if a task is in a yielded state (no failures, no owner, explicitly yielded, future expires time), pretend it only yielded until right now instead of whenever it really yielded to.

Basically, this rewrites history so that even though Harbormaster did a `yield(15)`, we pretend it did a `yield(4)` after we activate the lease if lease activation took 4 seconds.

If this misses, it's fine: we fall back to the normal yield behavior and things move forward normally a few seconds later.

If it hits, we get a more nimble process pretty cleanly.

Test Plan:
  - Restarted a build plan (lease working copy + run `ls`) with this patch no-op'd, took about 16 seconds.
  - Restarted a build plan with this patch active, took about 1 second.

Reviewers: hach-que, chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14178
2015-09-28 09:35:40 -07:00
epriestley
d735c7adf2 Allow Harbormaster to run commands on Drydock working copies
Summary: Ref T9252. This mostly cleans up future and log handling, and edges us closer to being able to do useful work with Harbormaster / Drydock.

Test Plan:
  - Added a "Run `ls -alh`" step to my trivial build plan.
  - Ran it a bunch of times.
  - Worked great.
  - Also did an HTTP plan.

{F835227}

{F835228}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14161
2015-09-25 10:43:32 -07:00
epriestley
284fe0fe51 Allow Harbormaster to lease working copies from Drydock
Summary: Ref T9252. This is still crude in a few ways but basically works, at least for commits.

Test Plan:
  - Made a build plan with just this build step.
  - Ran `bin/harbormaster build --plan 10 ...` on a commit.
  - It actually built a working copy, leased it, took no action, and released the lease. MAGIC~~~

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14160
2015-09-24 17:29:47 -07:00
June Rhodes
254b394f2c Associate Harbormaster build target with leases
Summary: Ref T1049.  This ensures the Harbormaster build target is associated with leases, so in the future we can query things and find out whether builds are still running with associated leases.

Test Plan: Leased a host, checked the DB and saw the field populated.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: joshuaspence, Korvin, epriestley

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D10870
2015-08-24 17:20:59 +10:00
Chad Little
a1d1cf77a7 Remove call to loadDrydockLease
Summary: Fixes T9219... I think.

Test Plan: swagging a guess for epriestley

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T9219

Differential Revision: https://secure.phabricator.com/D13932
2015-08-18 17:52:55 -07:00
epriestley
b21270d498 Probably fix bad loadArtifactLease() call
Summary: Ref T9205. This is a likely fix.

Test Plan: This isn't straightforward to test in the upstream unless you have custom code on top of it.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9205

Differential Revision: https://secure.phabricator.com/D13928
2015-08-18 10:46:41 -07:00
epriestley
57b0353034 Add harbormaster.createartifact
Summary:
Ref T8659. In the general case, this eventually allows build processes to do things like:

  - Upload build results (like a ".app" or ".exe" or other binary).
  - Pass complex results between build steps (e.g., build step A does something hard and build step B uses it to do something else).

Today, we're a long way away from having the infrastructure for that. However, it is useful to let third party build processes (like Jenkins) upload URIs that link back to the external build results.

This adds `harbormaster.createartifact` so they can do that. The only useful thing to do with this method today is have your Jenkins build do this:

  params = array(
    "uri": "https://jenkins.mycompany.com/build/23923/details/",
    "name": "View Build Results in Jenkins",
    "ui.external": true,
  );
  harbormaster.createartifact(target, 'uri', params);

Then (after the next diff) we'll show a link in Differential and a prominent link in Harbormaster. I didn't actually do the UI stuff in this diff since it's already pretty big.

This change moves a lot of code around, too:

  - Adds PHIDs to artifacts.
  - It modularizes build artifact types (currently "file", "host" and "URI").
  - It formalizes build artifact parameters and construction:
    - This lets me generate usable documentation about how to create artifacts.
    - This prevents users from doing dangerous or policy-violating things.
  - It does some other general modernization.

Test Plan:
{F715633}

{F715634}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8659

Differential Revision: https://secure.phabricator.com/D13900
2015-08-15 07:28:56 -07:00
epriestley
8860f4724f Lump Harbormaster build steps into groups
Summary:
Ref T8089. We have a lot of broken/confusing/prototype build steps that I want to hide from users when we unprototype Harbormaster.

The dialog is also just kind of unwieldy.

Organize this UI a little better and put all the sketchy junk in a "prototypes" group that you can't see unless prototypes are enabled.

This doesn't break anything (the old steps will still work fine), but should reduce user confusion.

Test Plan:
Old UI:

{F691439}

New UI (prototypes off):

{F691440}

New UI (prototypes on):

{F691441}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8089

Differential Revision: https://secure.phabricator.com/D13803
2015-08-06 04:19:42 -07:00
Joshua Spence
f695dcea9e Use PhutilClassMapQuery
Summary: Use `PhutilClassMapQuery` where appropriate.

Test Plan: Browsed around the UI to verify things seemed somewhat working.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D13429
2015-07-07 22:51:57 +10:00
epriestley
76194a0dc1 Add "Autoplans" to Harbormaster
Summary:
Ref T8095. Two general problems:

  - I want Harbormaster to own all lint and unit test results.
  - I don't want users to have to configure anything for `arc` to keep working automatically.

These are in conflict because generic lint/unit test ownership in Harbormaster requires that build targets exist which we can attach build results to. However, we can't currently create build targets on demand: Harbormaster assumes it is responsible for creating targets, then running code or making third-party service calls to actually run the builds.

I considered two broad approaches to let `arc` push results into Harbormaster without requiring administrators to configure some kind of "arc results" build plan:

  # Add magic target PHIDs like `PHID-MAGIC-this-is-really-arc-unit`.
  # Add new code to build real targets with real PHIDs.

(1) is probably a bit less work to get off the ground, but I think it's worse overall and very likely to create more problems in the long run. I particularly worry that it will lead to a small amount of special casing in a very large number of places, which seems more fragile.

(2) is more work upfront but I think does a better job of putting all the special casing in one place that we can, e.g., more reasonably unit test, and letting the rest of the code rarely/never care about this case since it's just dealing with normal plans/steps/targets as far as it can tell.

This diff introduces "autoplans", which are source templates for plans/steps. This let us "push" these targets into Harbormaster. Hypthetically, any process "like" arc can use autoplans to upload test/lint/etc results. In practice, probably only `arc` will ever use this, but I think it's still quite a bit cleaner than the alternative despite all the generality.

Workflow is basically:

  - `arc` creates a diff.
  - `arc` calls `harbormaster.queryautotargets`, passing the diff PHID and saying "I have some lint and unit results I want to stick on this thing".
  - Harbormaster builds the plan, steps, and targets (if any of them don't already exist), and hands back the target PHIDs so `arc` has a completely standard-looking place to put results.
  - `arc` uploads the test results to the right targets, as though Harbormaster had asked it to run unit/lint in the first place.

(This doesn't actually do any of that yet, just sets things up.)

I'll maybe doc turn that ^^^^^^ into a doc for posterity since I think it's hard to guess what an "autotarget" is, but I'm going to grab some lunch first.

Test Plan:
  - Added unit tests to make sure we can build these things properly.
  - Used `harbormaster.queryautotargets` to build autotargets for a bunch of diffs.
  - Verified targets come up in "waiting for message" state.
  - Verified plans and steps are not editable.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: hach-que, epriestley

Maniphest Tasks: T8095

Differential Revision: https://secure.phabricator.com/D13345
2015-06-21 09:04:21 -07:00
epriestley
6aa494bd25 Fix missing property on HarbormasterBuildStepImplementation
Auditors: joshuaspence
2015-06-15 12:21:54 -07:00
Joshua Spence
1239cfdeaf Add a bunch of tests for subclass implementations
Summary: Add a bunch of tests to ensure that subclasses behave.

Test Plan: `arc unit`

Reviewers: eadler, #blessed_reviewers, epriestley

Reviewed By: eadler, #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D13272
2015-06-15 18:13:27 +10:00
Joshua Spence
b6d745b666 Extend from Phobject
Summary: All classes should extend from some other class. See D13275 for some explanation.

Test Plan: `arc unit`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D13283
2015-06-15 18:02:27 +10:00
Joshua Spence
ea7397f7e4 Rename PassphraseCredentialType subclasses for consistency
Summary: Ref T5655.

Test Plan: `arc unit` + `grep`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T5655

Differential Revision: https://secure.phabricator.com/D13266
2015-06-14 14:11:55 +10:00
Joshua Spence
bf81fda036 Linter fixes
Summary: Apply various minor linter fixes.

Test Plan: `arc lint`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D13109
2015-06-02 22:14:01 +10:00
Joshua Spence
36e2d02d6e phtize all the things
Summary: `pht`ize a whole bunch of strings in rP.

Test Plan: Intense eyeballing.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: hach-que, Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D12797
2015-05-22 21:16:39 +10:00
Joshua Spence
acb45968d8 Use __CLASS__ instead of hard-coding class names
Summary: Use `__CLASS__` instead of hard-coding class names. Depends on D12605.

Test Plan: Eyeball it.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: hach-que, Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D12806
2015-05-14 07:21:13 +10:00
Joshua Spence
39ca2fdf64 Use new FutureIterator instead of Futures
Summary: Ref T6829. Deprecate the `Futures()` function.

Test Plan: N/A

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6829

Differential Revision: https://secure.phabricator.com/D11077
2014-12-30 23:13:38 +11:00
Joshua Spence
3cf9a5820f Minor formatting changes
Summary: Apply some autofix linter rules.

Test Plan: `arc lint` and `arc unit`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin, hach-que

Differential Revision: https://secure.phabricator.com/D10585
2014-10-08 08:39:49 +11:00
James Rhodes
f015cb50fe Prevent "Wait for Build Commits" from creating billions of logs
Summary:
Resolves T5987.  This build step was at some point converted to use yielding, which meant that whenever the build step executes it will create a new log.  This checks to see if there is an existing log before creating a new one and uses that instead.

Long term we're going to need some way of attaching data to `PhabricatorWorkerYieldException` that can be read when the build step starts again; this will allow us to move more build steps off `while (...) { ... sleep(X); }` loops and onto yielding.

Test Plan: Tested locally.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley

Maniphest Tasks: T5987

Differential Revision: https://secure.phabricator.com/D10383
2014-08-30 02:11:45 +10:00
James Rhodes
51b34c0544 Abort previous build targets when a build is restarted
Summary: Ref T5936. This implements build implementations aborting early when the build has since been restarted.   Build steps now periodically poll to see if the build's current generation does not match their generation, and they throw a `HarbormasterBuildAbortedException` if that is the case.

Test Plan: Tested locally on my machine with the sleep build step.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T5936

Differential Revision: https://secure.phabricator.com/D10322
2014-08-26 20:46:23 +10:00
James Rhodes
ca8f7cdaa5 Execute commands under Powershell on Windows for Harbormaster
Summary:
Resolves T5831.  This modifies the Drydock SSH interface to execute commands under Powershell when the target host platform is Windows.  Powershell is far more featured than cmd.exe, and more closely resembles a UNIX shell.

Currently Powershell outputs stderr as an XML blob on a line, and while this code currently doesn't use that, it will allow us in the future (planned next week) to redirect that output to the stderr log instead of having it all merged in with stdout under cmd (where there is no way to distinguish it).

Test Plan:
Ran various native commands and PowerShell commands from a Harbormaster build, including things like:

```
Write-Host ("my test" + ${build.id})
```

and saw:

```
my test679
```

in the output.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Maniphest Tasks: T5831

Differential Revision: https://secure.phabricator.com/D10248
2014-08-13 12:48:52 +10:00
James Rhodes
aa87a524e2 Allow build steps to explicitly fail the build
Summary: We've received feedback that the "core - exception" is incredibly confusing, to the point where developers see this and write off the build failure as a Phabricator error that is unrelated to their changes.

Test Plan: Ran a build with a `exit 1` run step, didn't see the "core - exception" appear.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D10090
2014-08-01 08:08:28 +10:00
James Rhodes
0f355756f5 Make artifacts imply dependencies on build steps
Summary: This makes input artifacts imply the appropriate build step dependencies in the build plan.  That is, if you use a host artifact in a build step, it will then implicitly depend on the 'Lease Host' step.

Test Plan: Viewed the build plan with the artifacts, saw the dependencies.  Ran a build, saw everything execute in the correct order.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D10089
2014-07-31 12:27:37 +10:00
James Rhodes
cad41ea294 Implement build simulation; convert Harbormaster to be purely dependency based
Summary:
Depends on D9806.  This implements the build simulator, which is used to calculate the order of build steps in the plan editor.  This includes a migration script to convert existing plans from sequential based to dependency based, and then drops the sequence column.

Because build plans are now dependency based, the grippable and re-order behaviour has been removed.

Test Plan: Tested the migration, saw the dependencies appear correctly.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9847
2014-07-31 11:39:49 +10:00
Joshua Spence
0a62f13464 Change double quotes to single quotes.
Summary: Ran `arc lint --apply-patches --everything` over rP, mainly to change double quotes to single quotes where appropriate. These changes also validate that the `ArcanistXHPASTLinter::LINT_DOUBLE_QUOTE` rule is working as expected.

Test Plan: Eyeballed it.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin, hach-que

Differential Revision: https://secure.phabricator.com/D9431
2014-06-09 11:36:50 -07:00
epriestley
702f073f0a Fix some logic in WaitForPreviousBuildStep
Summary: Fixes T5062. See inlines.

Test Plan: Did not test whatsoever.

Reviewers: hach-que

Reviewed By: hach-que

Subscribers: epriestley

Maniphest Tasks: T5062

Differential Revision: https://secure.phabricator.com/D9132
2014-05-15 07:36:44 -07:00
epriestley
4a6d2e9c97 Allow tasks to yield to other tasks
Summary:
For Harbormaster tasks which want to poll or wait, this lets them say "try again a little later" without having to sleep and hold a queue slot.

This is basically the same as failing, except that we don't increment the failure counter. Instead, we just set the current lease to the correct length and then exit. The task will be retried after the lease expires.

Test Plan: Using both `bin/harbormaster` and `phd debug taskmaster`, ran a lot of waiting tasks through the queue, faking them to either yield or not yield in a controlled manner. The queue responded as expected, yielding tasks appropraitely and retrying them later.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D8792
2014-04-16 13:02:12 -07:00
epriestley
78bf266bde Allow Harbormaster build targets to wait for messages
Summary:
This hooks up all the pieces of the build pipeline so `harbormaster.sendmessage` actually works. Particularly:

  - Candidate build steps (i.e., those which interact with external systems) can now "Wait for Message". This pauses them indefinitely when they complete, until something calls `harbormaster.sendmessage`.
  - After processing a target, we check if we should move it to PASSED or WAITING.
  - Before updating a build, we move WAITING targets with pending messages to either PASSED or FAILED.
  - I added an explicit "Building" state, which doesn't affect workflows but communicates more information to human users.

A big part of this is avoiding races. I believe we get the correct behavior no matter which order events occur in:

  - We update builds after targets complete and after we receive messages, so we're guaranteed to update once both these conditions are true. This means messages can't be lost (even if they arrive before a build completes).
  - The minor changes to the build engine logic mean that firing additional build updates is always safe, no matter what the current state of the build is.
  - The build itself is protected by a lock in the build engine.
  - The target is not covered by an explicit lock, but for all states only the engine (waiting) //or// the worker (all other states) can interact with it. All of the interactions also move the target state forward to the same destination and have no other side effects.
  - Messages are only consumed inside the engine lock, so they don't need an explicit lock.

Test Plan:
  - Made an HTTP request wait after completion, then ran a pile of builds through it using `bin/harbormaster build` and the web UI.
  - Passed and failed message-awaiting builds with `harbormaster.sendmessage`.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, zeeg

Differential Revision: https://secure.phabricator.com/D8788
2014-04-16 13:01:46 -07:00
epriestley
803c50c1e7 Allow Harbormaster HTTP steps to pass credentials
Summary: Fixes T4590. Use the credentials custom field to allow Harbormaster HTTP requests to include usernames/passwords.

Test Plan: Ran a build plan with credentials, verified they were sent to the remote server.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4590

Differential Revision: https://secure.phabricator.com/D8786
2014-04-16 13:01:38 -07:00
epriestley
25f91567a7 Make various minor Harbormaster UI improvements
Summary: Ref T1049. Tweaks some of the UI and code to improve / clean it up a bit.

Test Plan: Ran build plans, browsed UI.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D8603
2014-03-25 16:10:50 -07:00
epriestley
cec8d10731 Rename concrete Harbormaster step implementations
Summary: Ref T1049. For consistency, rename these to "Harbormaster...".

Test Plan: Ran migration, ran builds, everything still works fine.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D8602
2014-03-25 16:09:51 -07:00
epriestley
281f06e281 Rename "BuildStepImplementation" to "HarbormasterBuildStepImplementation"
Summary: Ref T1049. D8588 already required custom code to change what it extends, so this is as good a time as we're going to get to move to more standard class name.

Test Plan: `arc liberate`; `arc lint`

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D8601
2014-03-25 16:09:21 -07:00
epriestley
a246c85c6b Use ApplicationTransactions and CustomField to implement build steps
Summary:
Ref T1049. Fixes T4602. Moves all the funky field stuff to CustomField. Uses ApplicationTransactions to apply and record edits.

This makes "artifact" fields a little less nice (but still perfectly usable). With D8599, I think they're reasonable overall. We can improve this in the future.

All other field types are better (e.g., fixes weird bugs with "bool", fixes lots of weird behavior around required fields), and this gives us access to many new field types.

Test Plan:
Made a bunch of step edits. Here's an example:

{F133694}

Note that:

  - "Required" fields work correctly.
  - the transaction record is shown at the bottom of the page.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4602, T1049

Differential Revision: https://secure.phabricator.com/D8600
2014-03-25 16:08:40 -07:00
epriestley
72337dedaf Make Harbormaster input and output artifacts more explicit
Summary:
Ref T1049. In Harbormaster, build steps may have various inputs (like a host they should run on) and outputs (like a reference to an uploaded file).

  - Currently, inputs aren't defined anywhere (except implicitly at runtime).
    - Instead, define inputs explicitly.
  - Currently, outputs are defined in a way that loses information when misconfigured (the keys will collide).
    - Instead, define inputs and outputs so they work whether a step is configured correctly or not.
  - Currently, there's no simple way to see a step's inputs and outputs.
    - Add some UI for this.
  - Currently, reordering steps has some surprising side effects.
    - Instead of invalidating steps after reordering them, validate them at display time and warn the user.

Test Plan:
{F133679}
{F133680}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, chad

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D8599
2014-03-25 16:02:34 -07:00
epriestley
5b74fa0a75 Make all build steps support variables
Summary: Ref T1049. This generally simplifies things. The steps which don't support variables generally don't make sense to support varaibles anyway.

Test Plan: Edited some steps.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D8588
2014-03-25 16:02:07 -07:00
epriestley
a2a4f4b3da Fix validation of Harbormaster HTTP methods
Summary: Precedence here was mucked up.

Test Plan: Plan with no explicit "method" now defaults to POST correctly.

Reviewers: dctrwatson, btrahan

Reviewed By: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D8559
2014-03-18 12:05:14 -07:00
William R. Otte
29436dfe37 Added 'method' field to the HTTP request build step.
Summary:
This revision adds a 'method' field to the HTTP request harbormaster build step.  This allows the user to specify GET, POST, DELETE, and PUT (limited by the underlying wrapper phabricator uses for HTTP requests).  I'm not sure how much sense PUT makes, but oh well.

Existing plans shouldn't break, as if this field is an empty string, we default to POST, which is the old behavior.

Fixes T4604

Test Plan: 1) Verified that the empty string does, in fact, issue a POST request.  Changed the method to be GET and observed that the problem described in T4604 is resolved.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: aran, epriestley

Maniphest Tasks: T4604

Differential Revision: https://secure.phabricator.com/D8520
2014-03-13 15:51:05 -07:00
epriestley
996930da2a Improve several exception behaviors for Harbormaster workers
Summary:
Ref T2015. Several fixes:

  - `checkForCancellation()` no longer exists, and isn't relevant for resumable stops. Throw it away for now.
  - Fix an issue where a build could pass even if the final step failed.
  - `phlog()` exceptions so they show up in `bin/harbormaster` and the daemon logs.
  - Write an exception log if a step fails.
  - Add a "throw an exception" step to debug this stuff more easily.

Test Plan:
  - Grepped for `checkForCancellation()`.
  - Ran a failing build where the final step caused the failure.
  - Observed `phlog()` in `bin/harbormaster` output.
  - Observed log in web UI:

{F101168}

Reviewers: btrahan, hach-que

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D7935
2014-01-13 12:21:49 -08:00