1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-12 00:26:13 +01:00
Commit graph

82 commits

Author SHA1 Message Date
epriestley
d4847c3eeb Convert simple query subclasses to use internal cursors
Summary:
Depends on D20291. Ref T13259. Move all the simple cases (where paging depends only on the partial object and does not depend on keys) to a simple wrapper.

This leaves a smaller set of more complex cases where we care about external data or which keys were requested that I'll convert in followups.

Test Plan: Poked at things, but a lot of stuff is still broken until everything is converted.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13259

Differential Revision: https://secure.phabricator.com/D20292
2019-03-19 13:00:27 -07:00
epriestley
620047bcfe Add a "Recent Builds" element to the Build Plan UI and tighten up a few odds and ends
Summary:
Depends on D20218. Ref T13258. It's somewhat cumbersome to get from build plans to related builds but this is a reasonable thing to want to do, so make it a little easier.

Also clean up / standardize / hint a few things a little better.

Test Plan: {F6244116}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13258

Differential Revision: https://secure.phabricator.com/D20219
2019-02-28 07:38:37 -08:00
epriestley
deea2f01f5 Allow unit tests to have arbitrarily long names (>255 characters)
Summary:
Depends on D20179. Ref T13088. See PHI351. See PHI1018. In various cases, unit tests names are 19 paths mashed together.

This is probably not an ideal name, and the test harness should probably pick a better name, but if users are fine with it and don't want to do the work to summarize on their own, accept them. We may summarize with "..." in some cases depending on how this fares in the UI.

The actual implementation is a separate "strings" table which is just `<hash-of-string, full-string>`. The unit message table can end up being mostly strings, so this should reduce storage requirements a bit.

For now, I'm not forcing a migration: new writes use the new table, existing rows retain the data. I plan to provide a migration tool, recommend migration, then force migration eventually.

Prior to that, I'm likely to move at least some other columns to use this table (e.g., lint names), since we have a lot of similar data (arbitrarily long user string constants that we are unlikely to need to search or filter).

Test Plan: {F6213819}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D20180
2019-02-19 11:21:42 -08:00
epriestley
c5e16f9bd9 Give HarbormasterBuildUnitMessage a real Query class
Summary: Ref T13088. Prepares for putting test names in a separate table to release the 255-character limit.

Test Plan: Viewed revisions, buildables, builds, test lists, specific tests.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D20179
2019-02-15 19:16:47 -08:00
epriestley
01c7be059d Add support for "harbormaster.target.search"
Summary: Ref T13222. See PHI986. See PHI896. Harbormaster build targets don't currently have a modern "*.search" API, but there's no reason not to provide one (even if some of the use cases are a little bit questionable).

Test Plan: {F6032423}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19841
2018-11-28 13:49:27 -08:00
epriestley
a817aa6c71 Add an "Abort Older Builds" build step to Harbormaster
Summary:
Ref T13124. See PHI531. When a revision is updated, builds against the older diff tend to stop being relevant. Add an option to abort outstanding older builds automatically.

At least for now, I'm adding this as a build step instead of some kind of special checkbox. An alternate implementation would be some kind of "Edit Options" action on plans with a checkbox like `[X] When this build starts, abort older builds.`

I think adding it as a build step is a bit simpler, and likely to lead to greater consistency and flexibility down the road, make it easier to add options, etc., and since we don't really have any other current use cases for "a bunch of checkboxes". This might change eventually if we add a bunch of checkboxes for some other reason.

The actual step activates //before// the build queues, so it doesn't need to wait in queue before it can actually act. T13088 discusses some plans here if this sticks.

Test Plan:
  - Created a "Sleep for 120 seconds" build plan and triggered it with Herald.
  - Added an "Abort Older Builds" step.
  - Updated a revision several times in a row, saw older builds abort.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13124

Differential Revision: https://secure.phabricator.com/D19376
2018-04-17 14:59:47 -07:00
epriestley
5a2213ef82 Provide API read access to Harbormaster build logs
Summary:
Depends on D19150. Ref T13088. Allow clients to retrieve information about build logs, including log data, over the API.

(To fetch log data, take the `filePHID` to `file.search`, then issue a normal GET against the URI. Use a `Content-Range` header to get part of the log.)

Test Plan: Ran `harbormaster.log.search`, got sensible-looking results.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19151
2018-02-28 12:36:03 -08:00
epriestley
f939a2b12e Make Harbormaster buildable status more of a nice flexible map and less of a bunch of switch statements
Summary: Depends on D19063. Ref T13054. Prepare for the addition of a new `PREPARING` status by getting rid of the "scattered mess of switch statements" pattern of status management.

Test Plan: Searched/browsed buildables. Viewed buildables. Viewed revisions. Grepped for all affected symbols.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13054

Differential Revision: https://secure.phabricator.com/D19064
2018-02-12 12:18:06 -08:00
epriestley
c42bbd6f5c Rename HarbormasterBuildMessage "buildTargetPHID" to "receiverPHID"
Summary: Ref T13054. Companion storage change for D19062.

Test Plan: Applied migration and adjustments. Viewed messages in Harbormaster; created them with `harbormaster.sendmessage`; processed them with `bin/phd debug task`.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13054

Differential Revision: https://secure.phabricator.com/D19063
2018-02-12 12:17:44 -08:00
epriestley
ed0ba41cd2 Allow a HarbormasterBuildMessage to be sent to any object
Summary:
See T13054. This prepares for Buildables to be sent messages ("attach", "done scheduling builds") to fix races between Harbormaster and Differential.

The `buildTargetPHID` is replaced with a `recipientPHID` in the API. An additional change will fix the storage.

In the future, this table could probably also replace `HarbormasterBuildCommand` now, which is approximately the same bus, but for Builds.

Test Plan: Viewed builds with messages. Sent messages with `harbormaster.sendmessage`. Processed messages with `bin/phd debug task`.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19062
2018-02-12 12:16:03 -08:00
epriestley
63e6b2553e Simply how Differential drafts ignore Harbormaster autobuilds
Summary:
Ref T2543. When a revision is created, we check if any builds are waiting/failed, and submit it for review immediately if we aren't waiting for anything.

In doing this, we ignore builds with only autotargets, since these are client-side and failures from local `arc lint` / `arc unit` should not count (the user has already chosen to ignore/skip them).

The way we do this has some issues:

  - Herald may have started builds, but they may still be PENDING and not have any targets yet. In this case, we'll see "no non-autotargets" and ignore the build, which is wrong.
  - We have to load targets but don't really care about them, which is more work than we really need to do.
  - And it's kind of complex, too.

Instead, just let `BuildQuery` filter out "autobuilds" (builds generated from autoplans) with a JOIN.

Test Plan: Ran `arc diff` with builds configured, got a clean "Draft" state instead of an incorrect promotion directly to "Needs Review".

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18721
2017-10-23 10:31:48 -07:00
epriestley
99889a6321 Execute Harbormaster buildable filtering properly from HarbormasterBuildSearchEngine
Summary:
Ref T11473. When running `harbormaster.build.search` with a `buildables` constraint, the constraint doesn't get passed to the Query and so currently has no effect.

This piece of logic was just accidentally omitted from D16356. It is probably not used anywhere today and doesn't show up in the UI, so it's easy to overlook (I missed it in review, too).

Test Plan: Ran `harbormaster.build.search` with a `buildables` constraint, got expected filtering.

Reviewers: yelirekim, chad

Reviewed By: chad

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11473

Differential Revision: https://secure.phabricator.com/D16395
2016-08-14 08:31:02 -07:00
Mike Riley
98492765d3 Subsume 'harbormaster.querybuilds' with a modern search API method
Summary: We deprecate the existing API method used to access build information from the API, but preserve its response structure after calling through to the new method.  I've cordoned off the fields I needed to define in order to meet the output structure by putting those fields in a search attachment.

Test Plan:
Used the API console and looked at the list view controller for builds.

Old output structure:

```lang=json
{
  "data": [
    {
      "id": "16823",
      "phid": "PHID-HMBD-xghrwfz6luoye5rgc2hq",
      "uri": "https://secure.phabricator.com/harbormaster/build/16823/",
      "name": "Run Core Tests",
      "buildablePHID": "PHID-HMBB-s6ykzm2jzxz4ymduztq3",
      "buildPlanPHID": "PHID-HMCP-pcfxcgyoif67l3buc4zt",
      "buildStatus": "passed",
      "buildStatusName": "Passed"
    }
  ],
  "cursor": {
    "limit": 100,
    "after": "16823",
    "before": null
  }
}
```

New output structure:

```lang=json
{
  "data": [
    {
      "id": 1,
      "type": "HMBD",
      "phid": "PHID-HMBD-qpgcmv67tzaauzayzit5",
      "uri": "http://ec2-54-165-244-168.compute-1.amazonaws.com/harbormaster/build/1/",
      "name": "arc lint + arc unit",
      "buildStatusName": "Passed",
      "buildablePHID": "PHID-HMBB-qdefith5uakkepqpjr2g",
      "buildPlanPHID": "PHID-HMCP-zswbhazb7ipmaf4plygg",
      "buildStatus": "passed",
      "initiatorPHID": "PHID-USER-rihx4366f3aczsvc2wtb",
      "dateCreated": 1450295643,
      "dateModified": 1450295644,
      "policy": {
        "view": "users",
        "edit": "users"
      }
    }
  ],
  "maps": {},
  "query": {
    "queryKey": null
  },
  "cursor": {
    "limit": 100,
    "after": null,
    "before": null,
    "order": null
  }
}
```

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D16356
2016-07-31 21:44:22 +00:00
Mike Riley
4865dbdff1 Search builds based on who kicked them off
Summary:
It's only natural for users to be interested their own builds. We are also building in support for other sources of builds, the only formally supported way to run a build right now is via Herald.

In our third party codebase, we designate an application as the "thing" that started builds which are scheduled and managed automatically by phabricator. I believe this is a common practice elsewhere in the codebase when you're at a loss for a real human identity and you need to apply some transactions.

Test Plan: Ran some builds manually and saw them show up under the list of things I've run.  Looking up builds based on those that had been started by a herald rule.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D16353
2016-07-31 20:54:44 +00:00
Mike Riley
33fca12816 Pick some preset build statuses
Summary:
We're picking three useful groups of build statuses to provide as default queries:

 - Stuff not yet building
 - Stuff building
 - Stuff which has finished building

These are reasonable buckets for builds since (unlike most objects in phabricatorland) users are generally waiting impatiently for the machine to do something for them, rather than being responsible for doing something with the machine.

Test Plan: clicked around the search engine and enjoyed my defaults

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D16349
2016-07-31 15:35:18 +00:00
Mike Riley
42b81a8090 Move build statuses to a constants class
Summary: No functional changes here, just lifting this out to make room for activities, heeding lint warnings along the way.

Test Plan:
before:
```lang=bash
$ grep -Rn "HarbormasterBuild::" *
src/applications/harbormaster/storage/HarbormasterBuildable.php:169:    $build = HarbormasterBuild::initializeNewBuild($viewer)
src/applications/harbormaster/storage/HarbormasterBuildable.php:173:      ->setBuildStatus(HarbormasterBuild::STATUS_PENDING);
src/applications/harbormaster/controller/HarbormasterStepEditController.php:242:    $variables = HarbormasterBuild::getAvailableBuildVariables();
src/applications/harbormaster/controller/HarbormasterBuildableViewController.php:200:        'fa-dot-circle-o '.HarbormasterBuild::getBuildStatusColor($status),
src/applications/harbormaster/controller/HarbormasterBuildableViewController.php:201:        HarbormasterBuild::getBuildStatusName($status));
src/applications/harbormaster/controller/HarbormasterBuildableViewController.php:203:      $item->addAttribute(HarbormasterBuild::getBuildStatusName($status));
src/applications/harbormaster/controller/HarbormasterBuildViewController.php:584:        HarbormasterBuild::getBuildStatusName($status);
src/applications/harbormaster/controller/HarbormasterBuildViewController.php:585:      $icon = HarbormasterBuild::getBuildStatusIcon($status);
src/applications/harbormaster/controller/HarbormasterBuildViewController.php:586:      $color = HarbormasterBuild::getBuildStatusColor($status);
src/applications/harbormaster/event/HarbormasterUIEventListener.php:135:      $status_name = HarbormasterBuild::getBuildStatusName($status);
src/applications/harbormaster/event/HarbormasterUIEventListener.php:136:      $icon = HarbormasterBuild::getBuildStatusIcon($status);
src/applications/harbormaster/event/HarbormasterUIEventListener.php:137:      $color = HarbormasterBuild::getBuildStatusColor($status);
src/applications/harbormaster/conduit/HarbormasterQueryBuildsConduitAPIMethod.php:78:        'buildStatusName' => HarbormasterBuild::getBuildStatusName($status),
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:66:      $build->setBuildStatus(HarbormasterBuild::STATUS_ERROR);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:109:      $build->setBuildStatus(HarbormasterBuild::STATUS_ABORTED);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:113:    if (($build->getBuildStatus() == HarbormasterBuild::STATUS_PENDING) ||
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:116:      $build->setBuildStatus(HarbormasterBuild::STATUS_BUILDING);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:121:      $build->setBuildStatus(HarbormasterBuild::STATUS_BUILDING);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:126:      $build->setBuildStatus(HarbormasterBuild::STATUS_PAUSED);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:132:    if ($build->getBuildStatus() == HarbormasterBuild::STATUS_BUILDING) {
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:246:      $build->setBuildStatus(HarbormasterBuild::STATUS_FAILED);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:254:      $build->setBuildStatus(HarbormasterBuild::STATUS_PASSED);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:290:      $build->setBuildStatus(HarbormasterBuild::STATUS_DEADLOCKED);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:446:      if ($build->getBuildStatus() != HarbormasterBuild::STATUS_PASSED) {
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:449:      if ($build->getBuildStatus() == HarbormasterBuild::STATUS_FAILED ||
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:450:          $build->getBuildStatus() == HarbormasterBuild::STATUS_ERROR ||
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:451:          $build->getBuildStatus() == HarbormasterBuild::STATUS_DEADLOCKED) {
```

after:

```lang=bash
$ grep -Rn "HarbormasterBuild::" *
src/applications/harbormaster/storage/HarbormasterBuildable.php:169:    $build = HarbormasterBuild::initializeNewBuild($viewer)
src/applications/harbormaster/controller/HarbormasterStepEditController.php:242:    $variables = HarbormasterBuild::getAvailableBuildVariables();
```

ran a manual build as a sanity check

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D16348
2016-07-31 14:56:31 +00:00
Mike Riley
2c55a4ad72 Provide a basic search engine for builds
Summary:
This supports a few basic use cases that aren't served by the buildable search engine:

 - I'm trying to discover when the last time that this particular build plan failed was.
 - I want to know if any builds have deadlocked.
 - At a glance, I'm more interested in what build plans are running, not which buildables are being built. This is more often than not the case.

Test Plan: {F1744003}

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D16347
2016-07-31 13:35:31 +00:00
epriestley
96c51028e5 In Harbormaster, release artifacts as soon as no waiting/running build steps will use them
Summary:
Ref T11153. If you have a build plan like this:

  - Lease machine A.
  - Lease machine B.
  - Run client-tests on machine A.
  - Run server-tests on machine B.

...and we get machine A quickly, then finish the tests, we currently do not release machine A until the whole plan finishes.

In the best case, this wastes resources (something else could be using that machine for a while).

In a worse case, this wastes a lot of resources (if machine B is slow to acquire, or the server tests are much slower than the client tests, machine A will get tied up for a really long time).

In the absolute worst case, this might deadlock things.

Instead, release artifacts as soon as no waiting/running steps take them as inputs. In this case, we'd release machine A as soon as we finished running the client tests.

In the case where machines A and B are resources of the same type, this should prevent deadlocks. In all cases, this should improve build throughput at least somewhat.

Test Plan:
I wrote this build plan which runs a "fast" step (10 seconds) and a "slow" step (120 seconds):

{F1691190}

Before the patch, running this build plan held the lease on the "fast" machine for the full 120 seconds, then released both leases at the same time at the very end.

After this patch, I ran this plan and observed the "fast" lease get released after 10 seconds, while the "slow" lease was held for the full 120.

(Also added some `var_dump()` into things to sanity check the logic; it appeared correct.)

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11153

Differential Revision: https://secure.phabricator.com/D16145
2016-06-17 16:13:56 -07:00
epriestley
fc0dc02bb9 Allow Drydock blueprints to be tagged and searched, and give types some little icons
Summary:
Ref T10457.

  - Let blueprints be tagged so you can search and annotate them a little more easily.
  - Give each blueprint type an optional icon to make things a little easier to parse visually.

Test Plan:
  - Tagged blueprints.
  - Searched by tags.
  - Looked at nice little icons.

{F1139712}

Reviewers: chad

Reviewed By: chad

Subscribers: yelirekim

Maniphest Tasks: T10457

Differential Revision: https://secure.phabricator.com/D15392
2016-03-03 15:21:58 -08:00
epriestley
5512e9724f Allow Harbormaster build plans to be tagged with projects and searched by tag
Summary: Ref T10457. This is mostly just for consitency, but I imagine it will make managing large/complex build processes easier, and if we support Herald rules it would eventually let you write "Build plan's tags include [whatever]" to apply behavior to a group of plans.

Test Plan: {F1133107}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10457

Differential Revision: https://secure.phabricator.com/D15360
2016-02-29 05:22:59 -08:00
epriestley
f078fd98d7 Support searching for Harbormater build plans by name substring
Summary: Ref T10457. Allow build plans to be queried by name.

Test Plan:
  - Searched for plans by name.
  - Renamed a plan, searched for new name.

{F1133085}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10457

Differential Revision: https://secure.phabricator.com/D15359
2016-02-29 05:22:24 -08:00
epriestley
c29ba039bb Update Buildable search in Harbormaster
Summary:
Fixes T10011.

  - Modernize searching for buildables.
  - Prepare for `harbormaster.buildable.search`.
  - Allow users to query by status (see T10011).
  - Collapse the four weird "commit / diff / revision / repository" fields into two slightly less weird fields with more UI hinting?

Test Plan: {F1131918}

Reviewers: chad

Reviewed By: chad

Subscribers: Luke081515.2

Maniphest Tasks: T10011

Differential Revision: https://secure.phabricator.com/D15356
2016-02-27 07:13:10 -08:00
epriestley
fdca684814 Slightly improve Buildable list in Harbormaster
Summary:
Ref T10457. This makes diffs/revisions show the revision as the buildable title, and commits show the commit as the title.

Previously, the title was "Buildable X".

Also makes icons/colors/labels more consitent.

Test Plan: {F1131885}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10457

Differential Revision: https://secure.phabricator.com/D15355
2016-02-27 07:11:30 -08:00
epriestley
220ac48801 Remove buildable handle / container handle logic form Harbormaster buildable queries
Summary:
Ref T10457. We currently have these weird, out-of-place methods on Harbormaster queries that just load handles. These were written before HandlePool, and HandlePool is now more convenient, simpler, and more efficient.

Drop this stuff in favor of using handle pools off `$viewer`.

Test Plan: Looked at buildable list, looked at buildable detail, grepped for removed methods.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10457

Differential Revision: https://secure.phabricator.com/D15354
2016-02-27 07:11:09 -08:00
epriestley
37532a0bf0 Remove various additional calls to getCallsign()
Summary: Ref T4245. These are all straightforward to remove.

Test Plan:
- Edited paths in a package.
- Ran `bin/audit delete --repositories ...` with various identifiers.
- Searched by repository for `R3`, `rAAAA` in Harbormaster.
- Did a Herald dry run on a commit.
- Browsed commits, made comments.
- Viewed a Releeph product list.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D14927
2016-01-02 11:04:22 -08:00
epriestley
74bf0d6ec6 Show external build links in applications
Summary: Fixes T8659. This isn't //explicitly// documented but I'm going to wait for a bit until the "Harbormaster" doc splits into internal/external builds to add docs for it. There's other similar stuff coming soon anyway.

Test Plan:
{F716439}

{F716440}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8659

Differential Revision: https://secure.phabricator.com/D13903
2015-08-15 07:29:26 -07:00
epriestley
57b0353034 Add harbormaster.createartifact
Summary:
Ref T8659. In the general case, this eventually allows build processes to do things like:

  - Upload build results (like a ".app" or ".exe" or other binary).
  - Pass complex results between build steps (e.g., build step A does something hard and build step B uses it to do something else).

Today, we're a long way away from having the infrastructure for that. However, it is useful to let third party build processes (like Jenkins) upload URIs that link back to the external build results.

This adds `harbormaster.createartifact` so they can do that. The only useful thing to do with this method today is have your Jenkins build do this:

  params = array(
    "uri": "https://jenkins.mycompany.com/build/23923/details/",
    "name": "View Build Results in Jenkins",
    "ui.external": true,
  );
  harbormaster.createartifact(target, 'uri', params);

Then (after the next diff) we'll show a link in Differential and a prominent link in Harbormaster. I didn't actually do the UI stuff in this diff since it's already pretty big.

This change moves a lot of code around, too:

  - Adds PHIDs to artifacts.
  - It modularizes build artifact types (currently "file", "host" and "URI").
  - It formalizes build artifact parameters and construction:
    - This lets me generate usable documentation about how to create artifacts.
    - This prevents users from doing dangerous or policy-violating things.
  - It does some other general modernization.

Test Plan:
{F715633}

{F715634}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8659

Differential Revision: https://secure.phabricator.com/D13900
2015-08-15 07:28:56 -07:00
epriestley
524aa39aa2 Merge branch 'master' into redesign-2015 2015-06-23 10:26:47 -07:00
epriestley
de30e15b7e Make Differential load lint/unit data from Harbormaster
Summary: Fixes T8095. Still needs UI/UX work (see T8096) but this has all the core features now.

Test Plan: Saw Harbormaster lint/unit data as though it was Differential lint-unit data.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T8095

Differential Revision: https://secure.phabricator.com/D13401
2015-06-23 10:23:52 -07:00
epriestley
b55f9b6120 Merge branch 'master' into redesign-2015 2015-06-22 12:26:41 -07:00
epriestley
76194a0dc1 Add "Autoplans" to Harbormaster
Summary:
Ref T8095. Two general problems:

  - I want Harbormaster to own all lint and unit test results.
  - I don't want users to have to configure anything for `arc` to keep working automatically.

These are in conflict because generic lint/unit test ownership in Harbormaster requires that build targets exist which we can attach build results to. However, we can't currently create build targets on demand: Harbormaster assumes it is responsible for creating targets, then running code or making third-party service calls to actually run the builds.

I considered two broad approaches to let `arc` push results into Harbormaster without requiring administrators to configure some kind of "arc results" build plan:

  # Add magic target PHIDs like `PHID-MAGIC-this-is-really-arc-unit`.
  # Add new code to build real targets with real PHIDs.

(1) is probably a bit less work to get off the ground, but I think it's worse overall and very likely to create more problems in the long run. I particularly worry that it will lead to a small amount of special casing in a very large number of places, which seems more fragile.

(2) is more work upfront but I think does a better job of putting all the special casing in one place that we can, e.g., more reasonably unit test, and letting the rest of the code rarely/never care about this case since it's just dealing with normal plans/steps/targets as far as it can tell.

This diff introduces "autoplans", which are source templates for plans/steps. This let us "push" these targets into Harbormaster. Hypthetically, any process "like" arc can use autoplans to upload test/lint/etc results. In practice, probably only `arc` will ever use this, but I think it's still quite a bit cleaner than the alternative despite all the generality.

Workflow is basically:

  - `arc` creates a diff.
  - `arc` calls `harbormaster.queryautotargets`, passing the diff PHID and saying "I have some lint and unit results I want to stick on this thing".
  - Harbormaster builds the plan, steps, and targets (if any of them don't already exist), and hands back the target PHIDs so `arc` has a completely standard-looking place to put results.
  - `arc` uploads the test results to the right targets, as though Harbormaster had asked it to run unit/lint in the first place.

(This doesn't actually do any of that yet, just sets things up.)

I'll maybe doc turn that ^^^^^^ into a doc for posterity since I think it's hard to guess what an "autotarget" is, but I'm going to grab some lunch first.

Test Plan:
  - Added unit tests to make sure we can build these things properly.
  - Used `harbormaster.queryautotargets` to build autotargets for a bunch of diffs.
  - Verified targets come up in "waiting for message" state.
  - Verified plans and steps are not editable.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: hach-que, epriestley

Maniphest Tasks: T8095

Differential Revision: https://secure.phabricator.com/D13345
2015-06-21 09:04:21 -07:00
epriestley
7ad4c9c056 Replace Harbormaster "BuildItem" with Lint/Unit messages
Summary:
Ref T8095.

Harbormaster has a `BuildItem` class, but it has no table and is unused. This was an earlier idea about representing lint/unit results and some other possible types of messages, but I think we want to be more specific than this.

Remove `BuildItem` and add `Lint` and `Unit` storage. These tables roughly parallel how we store lint/unit messages today, with some guesses about how where they'll go in the future.

Test Plan: Ran `bin/storage upgrade` and got a clean adjust out of it.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: hach-que, epriestley

Maniphest Tasks: T8095

Differential Revision: https://secure.phabricator.com/D13329
2015-06-21 09:00:00 -07:00
Chad Little
801607381d [Redesign] PhabricatorApplicationSearchResultView
Summary: Ref T8099. This adds a new class which all search engines return for layout. I thought about this a number of ways, and I think this is the cleanest path. Each Engine can return whatever UI bits they needs, and AppSearch or Dashboard picks and lays the bits out as needed. In the AppSearch case, interfaces like Notifications, Calendar, Legalpad all need more custom layouts. I think this also leaves a resonable path forward for NUX as well. Also, not sure I implemented the class correctly, but assume thats easy to fix?

Test Plan: Review and do a search in each application changed. Grep for all call sites.

Reviewers: btrahan, epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T8099

Differential Revision: https://secure.phabricator.com/D13332
2015-06-19 11:46:20 +01:00
epriestley
53ef057b1b Merge branch 'master' into redesign-2015 2015-06-15 08:06:23 -07:00
epriestley
0695687572 Fix an undefined property on HarbormasterBuildLogQuery
Fixes T8546.

Auditors: joshuaspence
2015-06-15 06:36:42 -07:00
epriestley
57b898af9a Merge branch 'master' into redesign-2015 2015-06-10 07:44:58 -07:00
Joshua Spence
af1b586990 Fix method visibilities
Summary: See also D13186.

Test Plan: Ran `arc unit --everything`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D13201
2015-06-09 07:17:44 +10:00
epriestley
b611642a0f Convert Harbormaster Build Plans to SearchField
Summary: Ref T8441. Ref T7715. Removes `saveQueryOrder()`.

Test Plan: Used all search features for build plans.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7715, T8441

Differential Revision: https://secure.phabricator.com/D13195
2015-06-08 12:22:28 -07:00
Chad Little
13f9f209df [Redesign] Remove remaining barColor callsites
Summary: Ref T8099, Fixes T8341. Switches all remaining callsites to setBarColor to use setStatusIcon (sans workboards).

Test Plan: Test each of the applications I changed as I could (not Releeph).

Reviewers: btrahan, epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T8341, T8099

Differential Revision: https://secure.phabricator.com/D13059
2015-05-28 15:17:34 -07:00
epriestley
f5580c7a08 Make buildWhereClause() a method of AphrontCursorPagedPolicyAwareQuery
Summary:
Ref T4100. Ref T5595.

To support a unified "Projects:" query across all applications, a future diff is going to add a set of "Edge Logic" capabilities to `PolicyAwareQuery` which write the required SELECT, JOIN, WHERE, HAVING and GROUP clauses for you.

With the addition of "Edge Logic", we'll have three systems which may need to build components of query claues: ordering/paging, customfields/applicationsearch, and edge logic.

For most clauses, queries don't currently call into the parent explicitly to get default components. I want to move more query construction logic up the class tree so it can be shared.

For most methods, this isn't a problem, but many subclasses define a `buildWhereClause()`. Make all such definitions protected and consistent.

This causes no behavioral changes.

Test Plan: Ran `arc unit --everything`, which does a pretty through job of verifying this statically.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: yelirekim, hach-que, epriestley

Maniphest Tasks: T4100, T5595

Differential Revision: https://secure.phabricator.com/D12453
2015-04-20 10:06:09 -07:00
epriestley
845466b49b Implement viewer() and members(project) typeahead functions
Summary:
Ref T4100. This is still a bit rough around the edges, but mostly does what we're after.

  - Implements viewer() and members(...) functions.
  - The new browse workflow makes these discoverable.

Test Plan: {F374201}

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: chad, epriestley

Maniphest Tasks: T4100

Differential Revision: https://secure.phabricator.com/D12444
2015-04-17 11:06:58 -07:00
epriestley
5eb496bb3e Make Harbormaster build plan datasource more browsable
Summary: Ref T5750.

Test Plan:
  - Browsed plans in Harbormaster.
  - Browsed plans in typeahead browser.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5750

Differential Revision: https://secure.phabricator.com/D12435
2015-04-17 11:06:58 -07:00
Bob Trahan
5a9df1a225 Policy - filter app engines where the user can't see the application from panel editing
Summary: Fixes T7118. This does the basic "filter the list" thing, though it ends up being a little manual since I guess this hasn't come up before? There is also potential weird behavior if the user was using an app and lost access to it - they will have nothing selected on edit - but I think this is actually correct behavior in this circumstance.

Test Plan:
used a user who couldn't get access to the "quick create" apps and noted that the dropdown list on dashboard panel create was missing the expected engines

ran `arc unit --everything` to verify abstract method implemented everywhere

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T7118

Differential Revision: https://secure.phabricator.com/D11687
2015-02-04 15:47:48 -08:00
Joshua Spence
e448386d39 Fix method visibility for PhabricatorApplicationSearchEngine methods
Summary: Ref T6822.

Test Plan: Visual inspection. These methods are only called from within the `PhabricatorApplicationSearchEngine` class.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6822

Differential Revision: https://secure.phabricator.com/D11242
2015-01-07 07:34:52 +11:00
James Rhodes
a26c6147f5 Prevent artifact key collision when builds are restarted
Summary: Ref T1049.  Because we no longer destroy artifacts when builds are restarted, we need the build generation number to be part of the artifact key, otherwise we get collisions when restarting builds that contain build steps that emit artifacts.

Test Plan: Ran it with a build plan of "Lease Host" and "Run Command", no longer got an artifact key crash.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D10336
2014-08-28 08:21:36 +10:00
James Rhodes
0e15393b46 Prevent crash when build step has been deleted on build plan
Summary: This prevents crashes when looking at builds, where the build steps have been deleted on the build plan since the build was run.  Currently the only information that's pulled from the build step is the description (because this was too large to copy to every target).

Test Plan: Tested it locally.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D10361
2014-08-28 08:20:11 +10:00
James Rhodes
efadfbbc97 Implement build generations in Harbormaster
Summary:
Ref T5932.  Ref T5936.  This implements build generations in Harbormaster, which provides the infrastructure required to both show users the previous states of restarted builds and to allow users to forcefully abort builds (and their targets).

You can view previous generations of a build by adding `?g=<n>` to the URI, but this isn't exposed in the UI anywhere yet.

Test Plan: Ran a build plan with a Sleep step in it.  Reconfigured it for various sleep times and viewed previous generations of the build after restarting it.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Maniphest Tasks: T5932, T5936

Differential Revision: https://secure.phabricator.com/D10321
2014-08-21 22:55:24 +10:00
James Rhodes
dfa9b27a94 Use tabs on build targets and allow build steps to have a description
Summary:
Ref T1049. This uses tabs on build targets to hide the configuration details and variables by default, instead promoting the target name, it's status and a description of the build step.  The description is a new field on each build step.

The primary advantage of having a description on build steps is that DevOps can configure appropriate description information (including any troubleshooting information for build failures) on build steps, and developers who have builds fail against their code review can then look at this information.

Test Plan: Viewed a build plan and saw the appropriate information.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D10093
2014-08-01 08:09:15 +10:00
James Rhodes
cad41ea294 Implement build simulation; convert Harbormaster to be purely dependency based
Summary:
Depends on D9806.  This implements the build simulator, which is used to calculate the order of build steps in the plan editor.  This includes a migration script to convert existing plans from sequential based to dependency based, and then drops the sequence column.

Because build plans are now dependency based, the grippable and re-order behaviour has been removed.

Test Plan: Tested the migration, saw the dependencies appear correctly.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9847
2014-07-31 11:39:49 +10:00
Joshua Spence
97a8700e45 Rename PHIDType classes
Summary: Ref T5655. Rename `PhabricatorPHIDType` subclasses for clarity (see discussion in D9839). I'm not too keen on some of the resulting class names, so feel free to suggest alternatives.

Test Plan: Ran unit tests.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin, hach-que

Maniphest Tasks: T5655

Differential Revision: https://secure.phabricator.com/D9986
2014-07-24 08:05:46 +10:00