Summary:
See PHI766. Ref T13164. Build log chunk processing does a `preg_split()` on slices, but this isn't terribly efficient.
We can get the same count more cheaply by just using `substr_count()` a few times.
(I also tried `preg_match_all()`, which was between the two in speed.)
Test Plan:
- Used `bin/harbormaster rebuild-log --id X --force` to rebuild logs. Verified that the linemap is identical before/after this change.
- Saw local time for the 18MB log in PHI766 drop from ~1.7s to ~900ms, and `preg_split()` drop out of the profiler (we're now spending the biggest chunk of time on `gzdeflate()`).
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13164
Differential Revision: https://secure.phabricator.com/D19545
Summary:
Fixes T13145. The list controllers properly support public access already, but some of the view/detail controllers did not.
Allow logged-out users to browse builds, buildables, plans, etc., provided they can see the corresponding objects.
Test Plan: As a logged-out user, browsed around builds, build plans, logs, etc., without hitting any login pages.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13145
Differential Revision: https://secure.phabricator.com/D19459
Summary: See PHI611 for details.
Test Plan:
Ran a Buildkite build, saw Buildkite confirm receipt of these parameters in the HTTP response:
{F5562054}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Differential Revision: https://secure.phabricator.com/D19419
Summary:
See PHI615. Ref T13130. An install is reporting that "Lease Working Copy" build steps always report "Built instantly" after completion.
I'm not 100% sure that this is the fix, but I'm like 99% sure: "Lease Working Copy" build steps yield after they ask Drydock for a lease. They will later reenter `doWork()`, see that the lease is filled, and complete.
Right now, we reset the start time every time we enter `doWork()`. Instead, set it only if it hasn't been set yet.
Test Plan: This is low-risk and a bit tricky to reproduce locally, but I'll run some production builds and see what they look like.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13130
Differential Revision: https://secure.phabricator.com/D19412
Summary:
Ref T13124. See PHI531. When a revision is updated, builds against the older diff tend to stop being relevant. Add an option to abort outstanding older builds automatically.
At least for now, I'm adding this as a build step instead of some kind of special checkbox. An alternate implementation would be some kind of "Edit Options" action on plans with a checkbox like `[X] When this build starts, abort older builds.`
I think adding it as a build step is a bit simpler, and likely to lead to greater consistency and flexibility down the road, make it easier to add options, etc., and since we don't really have any other current use cases for "a bunch of checkboxes". This might change eventually if we add a bunch of checkboxes for some other reason.
The actual step activates //before// the build queues, so it doesn't need to wait in queue before it can actually act. T13088 discusses some plans here if this sticks.
Test Plan:
- Created a "Sleep for 120 seconds" build plan and triggered it with Herald.
- Added an "Abort Older Builds" step.
- Updated a revision several times in a row, saw older builds abort.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13124
Differential Revision: https://secure.phabricator.com/D19376
Summary:
See PHI565. Ref T13120. Although this older log is on the chopping block (see T13088), there's some migration guidance and other complexity around just replacing it.
Until it gets replaced, make clicking the "number of lines" elements respect the current "Build Generation" setting. Prior to this change, clicking the links would lose the generation information and jump you to the most recent build generation.
Also fix some collateral damage from T13105 where we ended up with white text on a white background in some cases.
Test Plan:
- Restarted a build to get multiple generations.
- On each generation, clicked the various "25", "50", etc., links.
- Saw generation and log window sizes both respected by the links.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13120
Differential Revision: https://secure.phabricator.com/D19367
Summary:
Depends on D19280. Ref T13110. Although Harbormaster cares about all builds, Differential does not practically care about local lint and unit results in determining build status.
In Differential, orient publishing around "remote builds" instead of "builds".
This does not yet change any of the draft logic, it just makes the timeline story use newer logic.
Test Plan: Used `bin/harbormaster publish` (with some guard-clause removal) to publish some buildables to revisions without anything crashing.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13110
Differential Revision: https://secure.phabricator.com/D19281
Summary:
Depends on D19279. Ref T13110. This implements the existing publishing logic for buildables, but does so via ModularTransactions instead of a core transaction type.
Since each application is implementing build transactions independently, this removes the core type.
Next, Differential will get a similar treatment.
Test Plan: Used `bin/harbormaster publish` (with some commenting-out-guard-clauses) to publish a commit Buildable; saw unchanged feed behavior.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13110
Differential Revision: https://secure.phabricator.com/D19280
Summary:
Depends on D19278. Ref T13110. This moves most of the structural logic for publishing builds to BuildableEngine and provides a `bin/harbormaster publish` to make publishing easy to retry/debug.
This intentionally removes the bit which actually does anything when builds publish. Followup changes will implement application-specific versions of the publishing logic in Differential and Diffusion.
Test Plan: Ran `bin/harbormaster publish Bxxx`, saw it do nothing (but not crash).
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13110
Differential Revision: https://secure.phabricator.com/D19279
Summary:
Ref T13110. Currently, build status is published the same way for every Buildable by the BuildEngine.
I want to change this to delegate publishing to each Buildable, particularly so that Differential may use more detailed rules for handling builds and drafts.
Rather than add additional methods to the existing `BuildableInterface`, add an engine generator method instead. This is a pattern which has seen more use recently (e.g., in Ferret) and lets us pay a little more upfront to pull complex pieces of logic out of the main class and let them use inheritence more easily. If we had Traits that might cover this to some degree.
I'd expect to eventually reduce the size of `BuildableInterface` and move the `CircleCI` and `BuildKite` interfaces so that the `BuildableEngine` implements them instead of the main object.
Here, this new engine does nothing and is never instantiated. In upcoming changes, publishing logic will move into it so that Differential can handle publishing differently.
Test Plan: Ran `arc liberate`, loaded pages, grepped for `BuildableInterface`.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13110
Differential Revision: https://secure.phabricator.com/D19278
Summary:
Ref T13114. See PHI511. Ref T13072. This makes Buildables, Builds, Targets and Artifacts destructible with `bin/remove destroy`.
This might not be totally exhaustive. In particular:
- File artifacts won't destroy the file. This is sort of okay because file artifacts are currently just a file reference, but probably shouldn't be how things work in the long term.
- `BuildCommand` doesn't get cleaned up, but `BuildMessage` does on `Build`. See T13072 for more.
Test Plan: Used `bin/remove destroy` to nuke a bunch of builds, buildables, etc. Loaded stuff in the web UI and it all looked like it got nuked properly.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13114, T13072
Differential Revision: https://secure.phabricator.com/D19269
Summary:
See PHI430. Ref T13102. When the "Build Status" element raises a policy exception, we currently fatal the whole page rather than raising a normal policy error.
This is because the policy check happens very late in page construction, long after we've made the decision to show the page instead of a policy error, and gets treated as a rendering error.
In turn, this is because the rendering is event-based rather than using a more modern Engine + EngineExtension sort of construct, so some of the actual logic runs way later than it should.
Since unwinding all of this isn't trivial and the current behavior is materially bad, limit the damage here for now by just hiding the element. See T13088 for notes on handling this in a more nuanced way in the future.
Test Plan:
- Created a revision visible to "Public".
- Ran a build against it with a build plan visible to "All Users".
- Viewed revision in an incognito window.
- Before patch: Policy fatal with a red "rendering phase" error box.
- After patch: Mostly-functional page with a missing "Build Status" element.
- Viewed revision as a user with a normal session, saw the same UI before and after the change.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13102
Differential Revision: https://secure.phabricator.com/D19232
Summary:
See PHI446. Ref T13088. Currently, there's no way to access older generations of a build unless you know the secret `?g=1` URI magic.
When a build has multiple generations, show a history table and let users click to see older run information.
This is currently very basic. It would be nice to show when each generation started, who started/restarted it, and what the build status was at the time the build was restarted. There's currently no convenient source for this information so just add a bare-bones, working version of this for now.
Test Plan:
Viewed pending, single-run and multi-restart builds. Saw table on builds with more than one generation. Clicked table entries to see different build data.
{F5471160}
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19217
Summary:
Depends on D19166. Ref T13088. When the user scrolls away from a followed log, break the focus lock.
Let users stop following a live log.
Show when lines are added more clearly.
Don't refresh quite as quickly give users a better shot at clicking the stop button.
These behaviors can probably be refined but are at least more plausible and less actively user-hostile than the first version of this behavior was.
Test Plan: Used `write-log --rate` to write a large log slowly. Clicked "Follow Log", followed for a bit. Scrolled away, still got live updates but no more scroll lock. Clicked stop, no more updates.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19167
Summary:
Depends on D19165. Ref T13088. Currently, in other applications, we use Zero Width Spaces and Javascript "copy" listeners to prevent line numbers from being copied. This isn't terribly elegant.
Modern browsers support a second approach: using psuedo-elements with `content`. Try this in Harbormaster since it's conceptually cleaner, at least. One immediate drawback is that Command-F can't find this text either.
Test Plan: In Safari, Chrome and Firefox, highlighted ranges of lines and copy/pasted text. Got just text (no line numbers) in all cases.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19166
Summary: Depends on D19164. Ref T13088. Now that the JS behaviors are generic, use them on the Harbormaster standalone page.
Test Plan: Clicked lines and dragged across line ranges. Reloaded pages. Saw expected highlighting behavior in the client and on the server across reloads.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19165
Summary:
Depends on D19162. Ref T13088. When a user links to `$1234`, we need to render a default view of the log with a piece at the head, a piece at the end, and a piece in the middle.
We also need to figure out the offset for line 1234, or multiple offsets for "1234-2345".
Since the logic views/reads mostly anticipated this it isn't too much of a mess, although there are a couple of bugs this exposes with view specifications that use combinations of parameters which were previously impossible.
Test Plan: Viewed a large log with no line marker. Viewed `$1`. Viewed `$end`. Viewed `$35-40`, etc. Expanded context around logs.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19163
Summary:
Depends on D19155. Ref T13094. Ref T4340.
We can't currently implement a strict `form-action 'self'` content security policy because some file downloads rely on a `<form />` which sometimes POSTs to the CDN domain.
Broadly, stop generating these forms. We just redirect instead, and show an interstitial confirm dialog if no CDN domain is configured. This makes the UX for installs with no CDN domain a little worse and the UX for everyone else better.
Then, implement the stricter Content-Security-Policy.
This also removes extra confirm dialogs for downloading Harbormaster build logs and data exports.
Test Plan:
- Went through the plain data export, data export with bulk jobs, ssh key generation, calendar ICS download, Diffusion data, Paste data, Harbormaster log data, and normal file data download workflows with a CDN domain.
- Went through all those workflows again without a CDN domain.
- Grepped for affected symbols (`getCDNURI()`, `getDownloadURI()`).
- Added an evil form to a page, tried to submit it, was rejected.
- Went through the ReCaptcha and Stripe flows again to see if they're submitting any forms.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13094, T4340
Differential Revision: https://secure.phabricator.com/D19156
Summary: Depends on D19152. Ref T13088. This adds live log tailing. It is probably not the final version of this feature because it prevents escape once you begin tailing a log.
Test Plan: Used `bin/harbormaster write-log --rate ...` to write a log slowly. Viewed it in the web UI. Clicked "Follow Log". Followed the log until the write finished, a lifetime later.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19153
Summary: Depends on D19151. Ref T13088. While dramatically less exciting than using `lolcat` and less general than `pv`, this should do the job adequately.
Test Plan: Piped a sizable log into `bin/harbormaster write-log` with `--rate 2048`, saw a progress bar. Loaded the log in the web UI and saw it grow as the page reloaded.
Reviewers: yelirekim
Reviewed By: yelirekim
Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19152
Summary:
Depends on D19150. Ref T13088. Allow clients to retrieve information about build logs, including log data, over the API.
(To fetch log data, take the `filePHID` to `file.search`, then issue a normal GET against the URI. Use a `Content-Range` header to get part of the log.)
Test Plan: Ran `harbormaster.log.search`, got sensible-looking results.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19151
Summary: Depends on D19149. Ref T13088. Since the new log requires a bunch of log reprocessing, the cutover is going to require at least some time for installs to run migrations. Add a link in the UI to ease the transition, smooth over some behaviors a little, and fix a fetch issue where we'd request past the end of the log (since this is now enforced).
Test Plan: Viewed a traditional Harbormaster build, saw links to the new standalone log pages.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19150
Summary: Depends on D19148. Ref T13088. The new rendering always executes range requests for data it needs, and we can satisfy these requests by loading the smallest number of chunks which span that range.
Test Plan: Piped 50,000 lines of Apache log into Harbormaster, viewed it in the new UI, got sensible rendering times and a reasonable amount of data actually going over the wire.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19149
Summary: Ref T13088. This variable bled through from an earlier loop and caused us to drop some of the lines in the middle.
Test Plan: Clicked "Show More", got an equal number of header and footer lines.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19148
Summary:
Depends on D19141. Ref T13088. Some of the fundamental log behaviors like "loading the correct rows" are now a bit better behaved.
The UI is a little less garbage, too.
Test Plan: Viewed some logs and loaded more context by clicking the buttons.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19142
Summary: Depends on D19139. Ref T13088. This doesn't actually work, but is close enough that a skilled attacker might be able to briefly deceive a small child.
Test Plan:
- Viewed some very small logs under very controlled conditions, saw content.
- Larger logs vaguely do something resembling working correctly.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19141
Summary:
Depends on D19138. Ref T13088. When we want to read the last part of a logfile //and show accurate line numbers//, we need to be able to get from byte offsets to line numbers somehow.
Our fundamental unit must remain byte offsets, because a test can emit an arbitrarily long line, and we should accommodate it cleanly if a test emits 2GB of the letter "A".
To support going from byte offsets to line numbers, compute a map with periodic line markers throughout the offsets of the file. From here, we can figure out the line numbers for arbitrary positions in the file with only a constant amount of work.
Test Plan: Added unit tests; ran unit tests.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19139
Summary: Depends on D19137. Ref T13088. This allows `rebuild-log` to skip work if the chunks are already compressed. It also prepares for a future GC which is looking for "text" or "gzip" chunks to throw away in favor of archival into Files; such a GC can use this column to find collectable logs and then write "file" to it, meaning "chunks are gone, this data is only available in Files".
Test Plan: Ran migration, saw logs populate as "text". Ran `rebuild-log`, saw logs rebuild as "gzip".
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19138
Summary: Depends on D19136. Ref T13088. Since it's probably impractical to do all the migrations these changes imply during `bin/storage upgrade`, provide some support for performing them online.
Test Plan: Ran `bin/harbormaster rebuild-log` with `--all`, `--id`, and with and without `--force`.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19137
Summary: Depends on D19135. Ref T13088. Denormalize the total log size onto the log itself. This makes reasoning about the log at display time easier, and we don't need to fish around in the database as much to figure out what we're dealing with.
Test Plan: Ran `bin/harbormaster rebuild-log`, saw an existing log populate. Ran `bin/harbormaster write-log`, saw new log write with proper length information.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19136
Summary:
Depends on D19134. Ref T13088. Future changes will support API writers, so push the log lock into the Log object.
Allow open/close ("this process is writing to this log") to be separate from live/final ("this log is still generating more data").
Test Plan: Wrote logs with `bin/harbormater write-log` and updated logs with `bin/harbormaster rebuild-log`.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19135
Summary: Depends on D19133. Ref T13088. Allows build logs to be formally destroyed, cleaning up their chunks and file data.
Test Plan:
- Used `bin/remove destroy` to destroy a log, verified chunks and files were removed.
- Used `bin/harbormaster rebuild-log` to force a log to rebuild, verified files were destroyed and regenerated.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19134
Summary: Depends on D19132. Ref T13088. This implements an extremely skeletal dedicated log page with a more-or-less functional "Download Log" button.
Test Plan: Downloaded a recent log. Tried to download an old (un-finalized) log, couldn't. Used `bin/harbormaster write-log` to get a convenient standalone link to a log.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19133
Summary: Depends on D19131. Ref T13088. During log finalization, stream the log into Files to support "Download Log", archive to Files, and API access.
Test Plan: Ran `write-log` and `rebuild-log`, saw Files objects generate with log content and appropriate permissions.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19132
Summary: Depends on D19130. Ref T13088. Currently, when a build log is closed we compress it in the same process. Separate this out into a dedicated worker since the plan is to do a lot more work during finalization, none of which needs to happen inline during builds (or, particuarly, inline during a Conduit call for API writes in the future).
Test Plan: Ran `bin/harbormaster write-log --trace`, saw compression run inline.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19131
Summary: Ref T13088. This is currently minimal but the modify-execute development loop on build logs is extremely long without it.
Test Plan: Ran `echo hi | ./bin/harbormaster write-log --target 12345`, saw the log show up in the web UI.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19130
Summary: See D19117. Instead of automatically figuring this out inside `phutil_tag()`, explicitly add rel="noreferrer" at the application level to all external links.
Test Plan:
- Grepped for `_blank`, `isValidRemoteURIForLink`, checked all callsites for user-controlled data.
- Created a link menu item, verified noreferrer in markup.
- Created a link custom field, verified no referrer in markup.
- Verified noreferrer for `{nav href=...}`.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Differential Revision: https://secure.phabricator.com/D19118
Summary:
Depends on D19084. Fixes T13078. When `phabricator.silent` is enabled, immediately fail the "HTTP Request", "CircleCI" and "Buildkite" build steps.
This doesn't feel quite as clean as most of the other behavior of `phabricator.silent`, since these calls are not exactly notifications in the same way that email is, and failing to make these calls means that builds run differently (whereas failing to deliver email doesn't really do anything).
However, I suspect that this behavior is almost always reasonable/correct, and that we can probably get away with it until this grey area between "notifications" and "external service calls" is more clearly defined.
Test Plan:
- Created a build with HTTP, CircleCI, and Buildkite steps.
- Put install in `phabricator.silent` mode: all three steps failed with "declining, because silent" messages.
- Put install back in normal mode: all three steps made HTTP requests.
- Read updated documentation.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13078
Differential Revision: https://secure.phabricator.com/D19085
Summary:
Depends on D19065. Ref T13054. Instead of just updating `containerPHID` and hoping for the best, queue a proper BuildWorker to process a "your container has changed, update it" message.
We also need to remove a (superfluous) `withContainerPHIDs()` when loading active diffs for a revision.
Test Plan:
- Without daemons, created a revision and saw builds stick in "preparing" with no container PHID, but also stay in draft mode.
- With daemons, saw builds actually build and get the right container PHID.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13054
Differential Revision: https://secure.phabricator.com/D19066
Summary:
Depends on D19064. Ref T13054. See that task for additional discussion.
When buildables are created by `arc` and have lint/unit messages, they can currently pass or fail before Herald triggers actual builds. This puts them in a pre-build state where they can't complete until Herald says it's okay.
On its own, this change intentionally strands `arc diff --only` diffs in the "PREPARING" stage forever.
Test Plan:
- Ran a build with `bin/harbormaster`, saw it build normally.
- Ran a build with web UI, saw it build normally.
- Ran a build with `arc diff`, saw it build normally.
- Ran a build with `arc diff --only`, saw it hang in "PREPARING" forever.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13054
Differential Revision: https://secure.phabricator.com/D19065
Summary: Depends on D19063. Ref T13054. Prepare for the addition of a new `PREPARING` status by getting rid of the "scattered mess of switch statements" pattern of status management.
Test Plan: Searched/browsed buildables. Viewed buildables. Viewed revisions. Grepped for all affected symbols.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13054
Differential Revision: https://secure.phabricator.com/D19064
Summary: Ref T13054. Companion storage change for D19062.
Test Plan: Applied migration and adjustments. Viewed messages in Harbormaster; created them with `harbormaster.sendmessage`; processed them with `bin/phd debug task`.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13054
Differential Revision: https://secure.phabricator.com/D19063
Summary:
See T13054. This prepares for Buildables to be sent messages ("attach", "done scheduling builds") to fix races between Harbormaster and Differential.
The `buildTargetPHID` is replaced with a `recipientPHID` in the API. An additional change will fix the storage.
In the future, this table could probably also replace `HarbormasterBuildCommand` now, which is approximately the same bus, but for Builds.
Test Plan: Viewed builds with messages. Sent messages with `harbormaster.sendmessage`. Processed messages with `bin/phd debug task`.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Differential Revision: https://secure.phabricator.com/D19062
Summary:
Ref T13054. Fixes T10746. Fixes T11154. This is really a one-line fix (include `ABORTED` in `BuildEngine->updateBuildable()`) but try to structure the code a little more clearly too and reduce (at least slightly) the number of random lists of status attributes spread throughout the codebase.
Also add a header tag for buildable status.
Test Plan: Aborted a build, saw buildable fail properly.
Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13054, T11154, T10746
Differential Revision: https://secure.phabricator.com/D19055
Summary:
Ref PHI261. This moves Harbormaster build status to work more similarly to other modern status types, like Differential revision status, where we try to specify as much behavior on the server as possible so that the client and server can vary independently.
(I don't have any specific plans to make Harbormaster build status configurable on the server side, but it isn't out of the question.)
Test Plan: Ran `harbormaster.querybuilds` (saw same data as before) and `harbormaster.build.search` (saw same data as before, plus new ANSI color data).
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Differential Revision: https://secure.phabricator.com/D18838
Summary:
Ref T2543. Fixes T10109.
Currently, Herald only runs in Differential when a change updates the diff. This is partly for historical reasons, and partly because we don't want to restart builds every time someone makes a comment. However, this behavior is inconsistent with other applications (which always trigger on any change), and occasionally confusing to users (in T10109, for example) or otherwise undesirable.
A similar issue is that T2543 has introduced a "Draft" state, where revisions don't send normal mail until builds finish. This interacts poorly with "Send me an email" rules (which shouldn't do anything here) and particularly with "Send me an email + only run these actions the first time the rule matches", since that might have an effect like "do nothing when the revision is created, then never anything again since you already did nothing once".
To navigate both of these issues, let objects tell Herald that certain actions (like mail or builds) are currently forbidden. If a rule uses a field or action which is currently forbidden, the whole rule automatically fails before it executes, but doesn't count toward "only the first time" as far as Herald's tracking of rule execution is concerned.
Then, forbid mail for draft revisions, and forbid builds for revisions which didn't just get updated. Forbidding mail fixes the issues with "Send me an email" that were created by the introduction of the draft state.
Finally, make Herald run on every revision update, not just substantive updates to the diff. This resolves T10109.
Test Plan:
Created revisions via the draft -> submit workflow, saw different transcripts. Here's a mail action being forbidden for a draft:
{F5237324}
Here's a build action being forbidden for a "mundane" update:
{F5237326}
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T10109, T2543
Differential Revision: https://secure.phabricator.com/D18731
Summary: See PHI176. We issue a query with only `artifactIndex` from `BuildTarget`, but don't have an applicable key.
Test Plan: This isn't on the normal Harbormaster execution path so I'm not 100% sure I have a local repro, but will confirm with customer.
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Differential Revision: https://secure.phabricator.com/D18732
Summary:
Ref T2543. When a revision is created, we check if any builds are waiting/failed, and submit it for review immediately if we aren't waiting for anything.
In doing this, we ignore builds with only autotargets, since these are client-side and failures from local `arc lint` / `arc unit` should not count (the user has already chosen to ignore/skip them).
The way we do this has some issues:
- Herald may have started builds, but they may still be PENDING and not have any targets yet. In this case, we'll see "no non-autotargets" and ignore the build, which is wrong.
- We have to load targets but don't really care about them, which is more work than we really need to do.
- And it's kind of complex, too.
Instead, just let `BuildQuery` filter out "autobuilds" (builds generated from autoplans) with a JOIN.
Test Plan: Ran `arc diff` with builds configured, got a clean "Draft" state instead of an incorrect promotion directly to "Needs Review".
Reviewers: amckinley
Reviewed By: amckinley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18721
Summary: Noticed a couple of typos in the docs, and then things got out of hand.
Test Plan:
- Stared at the words until my eyes watered and the letters began to swim on the screen.
- Consulted a dictionary.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam
Differential Revision: https://secure.phabricator.com/D18693
Summary: Fixes T12883. The task seems correct to me and I think this is a copy/paste mistake that probably blames to me.
Test Plan: Fiddled these numbers, viewed a build in Harbormaster, saw the adjusted time.
Reviewers: chad, amckinley
Reviewed By: chad
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T12883
Differential Revision: https://secure.phabricator.com/D18177
Summary:
Ref T12173.
- If we want to fetch a tag, Buildkite needs it as a "branch" (this means more like "ref to fetch").
- The API gets upset if we pass "refs/tags/...", so just pass the tag name without the prefix, which works.
- Do a better job with commits and pass a real branch to fetch.
Test Plan:
- Built a commit with Buildkite.
- Build a revision with Buildkite.
Reviewers: chad
Reviewed By: chad
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T12173
Differential Revision: https://secure.phabricator.com/D17282
Summary: Ref T12173. This might need some additional work but the basics seem like they're in good shape.
Test Plan:
- Buildkite is "bring your own hardware", so you need to launch a host to test anything.
- Launched a host in AWS.
- Configured Buildkite to use that host to run builds.
- Added a Buildkite build step to a new Harbormaster build plan.
- Used `bin/harbormaster build ...` to run the plan.
- Saw buildkite execute builds and report status back to Harbormaster
{F2553076}
{F2553077}
Reviewers: chad
Reviewed By: chad
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T12173
Differential Revision: https://secure.phabricator.com/D17270
Summary:
Fixes T9276. Fixes T8650. The story so far:
- We once published build updates to Revisions.
- An unrelated fix (D10911) sent them to the Diffs instead of Revisions, which isn't useful, since you can't see a diff's timeline anywhere.
- This also caused a race condition, where the RevisionEditor and DiffEditor would update the diff simultaneously (T8650).
- The diff update was just disabled to avoid the race (part of D13441).
- Instead, allow the updates to go somewhere else. In this case, we send commit updates to the commit but send diff updates to the revision so you can see 'em.
- Since everything will be using the revision editor now, we should either get proper lock behavior for free or it should be easy to add if something whack is still happening.
- Overall, this should pretty much put us back in working order like we were before D10911.
This behavior is undoubtedly refinable, but this should let us move forward.
Test Plan:
Saw a build failure in timeline:
{F2304575}
Reviewers: chad
Reviewed By: chad
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T9276, T8650
Differential Revision: https://secure.phabricator.com/D17139
Summary: Fixes T11910. I spent a couple of minutes looking for the root cause without much luck, but this will all be obsoleted by an eventual upgrade to `EditEngine` anyway.
Test Plan: Set and unset "Wait for Message", which now worked.
Reviewers: chad, avivey
Reviewed By: avivey
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T11910
Differential Revision: https://secure.phabricator.com/D16919
Summary:
Ref T11887. This isn't a great fix but makes the method behave properly until I get around to a real fix.
In the longer term, I want to convert all of this pluggable Harbormaster/Drydock stuff (blueprints, artifacts, build plans) to use EditEngine + EditField instead of the weird mishmash of older/custom stuff it currently uses. However, this is a more involved project to execute and I'd like to be in that area of the codebase first so it gets adequate testing.
Until that happens, just put a reasonble-ish mechanism in place to let artifacts correct inbound types. This is the only artifact type and only parameter which needs casting.
Test Plan:
- Made a `curl` call to `harbormaster.createartifact` to create a URI artifact with `?...&ui.external=1`.
- Before patch: type error on `ui.external` not being a boolean.
- After patch: artifact created successfully.
Reviewers: chad
Reviewed By: chad
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T11887
Differential Revision: https://secure.phabricator.com/D16887
Summary: Fixes T11642. Added a 'name' field to the results from harbormaster.build.search.
Test Plan: Went to `/conduit/method/harbormaster.build.search/` and ran a search that would yield results (because otherwise there will be nothing there). Noted that there was, in fact, a name in the results.
Reviewers: yelirekim, #blessed_reviewers, epriestley
Reviewed By: yelirekim, #blessed_reviewers, epriestley
Subscribers: epriestley, yelirekim
Maniphest Tasks: T11642
Differential Revision: https://secure.phabricator.com/D16569
Summary:
Ref T10867 for original use case. This workflow provides a plausible way for administrators to stop the daemons when performing upgrades or maintenance, then bring those daemons back up without resulting in the failure of builds that were running at the time.
On our organization's phab install, builds are running 24/7. The majority of these builds last for at least several minutes, and contain build steps which fail if interrupted and then resumed, as happens when turning daemons on and off.
Instead of allowing these build steps to resume execution as normal, this workflow will instruct active builds to restart their entire build process instead of just resuming whichever step they were on.
Test Plan:
contrived a build plan which would fail if resumed partway through:
- lease a working copy
- command `touch restart_{build.id}`
- command `test -e restart_{build.id} && rm restart_{build.id} && sleep 60`
followed old procedure:
- run a few of these builds manually
- `./bin/phd stop`
- `./bin/phd start`
- saw the builds fail
followed new procedure:
- run a few of these builds manually
- `./bin/phd stop`
- `./bin/harbormaster restart --active`
- `./bin/phd start`
- saw the builds pass
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Maniphest Tasks: T10867
Differential Revision: https://secure.phabricator.com/D16485
Summary: See T10746.
Test Plan: Fail one of several builds, run `./bin/harbormaster update`, see that Build Status is Failed.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, O14 ATC Monitoring, yelirekim
Maniphest Tasks: T10746
Differential Revision: https://secure.phabricator.com/D16480
Summary: Ref T11132, significantly cleans up the Config app, new layout, icons, spacing, etc. Some minor todos around re-designing "issues", mobile support, and maybe another pass at actual Group pages.
Test Plan: Visit and test every page in the config app, set new items, resolve setup issues, etc.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam, Korvin
Maniphest Tasks: T11132
Differential Revision: https://secure.phabricator.com/D16468
Summary:
Ref T11473. When running `harbormaster.build.search` with a `buildables` constraint, the constraint doesn't get passed to the Query and so currently has no effect.
This piece of logic was just accidentally omitted from D16356. It is probably not used anywhere today and doesn't show up in the UI, so it's easy to overlook (I missed it in review, too).
Test Plan: Ran `harbormaster.build.search` with a `buildables` constraint, got expected filtering.
Reviewers: yelirekim, chad
Reviewed By: chad
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T11473
Differential Revision: https://secure.phabricator.com/D16395
Summary: Fixes T11409. This syntax isn't compatible with older PHP.
Test Plan: Ran `arc lint` on the file.
Reviewers: yelirekim, chad
Reviewed By: chad
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T11409
Differential Revision: https://secure.phabricator.com/D16358
Summary: We deprecate the existing API method used to access build information from the API, but preserve its response structure after calling through to the new method. I've cordoned off the fields I needed to define in order to meet the output structure by putting those fields in a search attachment.
Test Plan:
Used the API console and looked at the list view controller for builds.
Old output structure:
```lang=json
{
"data": [
{
"id": "16823",
"phid": "PHID-HMBD-xghrwfz6luoye5rgc2hq",
"uri": "https://secure.phabricator.com/harbormaster/build/16823/",
"name": "Run Core Tests",
"buildablePHID": "PHID-HMBB-s6ykzm2jzxz4ymduztq3",
"buildPlanPHID": "PHID-HMCP-pcfxcgyoif67l3buc4zt",
"buildStatus": "passed",
"buildStatusName": "Passed"
}
],
"cursor": {
"limit": 100,
"after": "16823",
"before": null
}
}
```
New output structure:
```lang=json
{
"data": [
{
"id": 1,
"type": "HMBD",
"phid": "PHID-HMBD-qpgcmv67tzaauzayzit5",
"uri": "http://ec2-54-165-244-168.compute-1.amazonaws.com/harbormaster/build/1/",
"name": "arc lint + arc unit",
"buildStatusName": "Passed",
"buildablePHID": "PHID-HMBB-qdefith5uakkepqpjr2g",
"buildPlanPHID": "PHID-HMCP-zswbhazb7ipmaf4plygg",
"buildStatus": "passed",
"initiatorPHID": "PHID-USER-rihx4366f3aczsvc2wtb",
"dateCreated": 1450295643,
"dateModified": 1450295644,
"policy": {
"view": "users",
"edit": "users"
}
}
],
"maps": {},
"query": {
"queryKey": null
},
"cursor": {
"limit": 100,
"after": null,
"before": null,
"order": null
}
}
```
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D16356
Summary:
It's only natural for users to be interested their own builds. We are also building in support for other sources of builds, the only formally supported way to run a build right now is via Herald.
In our third party codebase, we designate an application as the "thing" that started builds which are scheduled and managed automatically by phabricator. I believe this is a common practice elsewhere in the codebase when you're at a loss for a real human identity and you need to apply some transactions.
Test Plan: Ran some builds manually and saw them show up under the list of things I've run. Looking up builds based on those that had been started by a herald rule.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D16353
Summary:
We're picking three useful groups of build statuses to provide as default queries:
- Stuff not yet building
- Stuff building
- Stuff which has finished building
These are reasonable buckets for builds since (unlike most objects in phabricatorland) users are generally waiting impatiently for the machine to do something for them, rather than being responsible for doing something with the machine.
Test Plan: clicked around the search engine and enjoyed my defaults
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D16349
Summary:
This supports a few basic use cases that aren't served by the buildable search engine:
- I'm trying to discover when the last time that this particular build plan failed was.
- I want to know if any builds have deadlocked.
- At a glance, I'm more interested in what build plans are running, not which buildables are being built. This is more often than not the case.
Test Plan: {F1744003}
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D16347
Summary: Ref T10628. This moves everything else over. I'll clean up the cruft in the next diff.
Test Plan:
- Viewed Conduit API page, toggled tabs.
- Viewed Harbormaster build, toggled tabs.
- Viewed a Drydock lease, swapped tabs.
- Viewed a Drydock resource, swapped tabs.
- Viewed mail, swapped tabs.
- Grepped for `addPropertyList(...)`, looked for any remaining calls with a second argument.
- Also checked rSAAS for any calls, but we don't have anything there that uses tabs.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10628
Differential Revision: https://secure.phabricator.com/D16207
Summary:
Ref T11153. If you have a build plan like this:
- Lease machine A.
- Lease machine B.
- Run client-tests on machine A.
- Run server-tests on machine B.
...and we get machine A quickly, then finish the tests, we currently do not release machine A until the whole plan finishes.
In the best case, this wastes resources (something else could be using that machine for a while).
In a worse case, this wastes a lot of resources (if machine B is slow to acquire, or the server tests are much slower than the client tests, machine A will get tied up for a really long time).
In the absolute worst case, this might deadlock things.
Instead, release artifacts as soon as no waiting/running steps take them as inputs. In this case, we'd release machine A as soon as we finished running the client tests.
In the case where machines A and B are resources of the same type, this should prevent deadlocks. In all cases, this should improve build throughput at least somewhat.
Test Plan:
I wrote this build plan which runs a "fast" step (10 seconds) and a "slow" step (120 seconds):
{F1691190}
Before the patch, running this build plan held the lease on the "fast" machine for the full 120 seconds, then released both leases at the same time at the very end.
After this patch, I ran this plan and observed the "fast" lease get released after 10 seconds, while the "slow" lease was held for the full 120.
(Also added some `var_dump()` into things to sanity check the logic; it appeared correct.)
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T11153
Differential Revision: https://secure.phabricator.com/D16145
Summary:
Ref T11137. This class is removed in D16099. Depends on D16099.
`PhutilURI` now attempts to "just work" with Git-style URIs, so at least in theory we can just delete all of this code and pretend it does not exist.
(I've left "Display URI" and "Effective URI" as distinct, at least for now, because I think the distinction may be relevant in the future even though it isn't right now, and to keep this diff small, although I may go remove one after I think about this for a bit.)
Test Plan:
- Created a new Git repository with a Git URI.
- Pulled/updated it, which now works correctly and should resolve the original issue in T11137.
- Verified that daemons now align the origin to a Git-style URI with a relative path, which should resolve the original issue in T11004.
- Grepped for `PhutilGitURI`.
- Also grepped in `arcanist/`, but found no matches, so no patch for that.
- Checked display/conduit URIs.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T11137
Differential Revision: https://secure.phabricator.com/D16100
Summary: Going to render these all normal case instead of all caps, and bump up the font size. Should be more consistent. Yellow if you green anything orange.
Test Plan: grep, lint
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15645
Summary: Cleans up EditEngine, adds new layout to EditEngine and descendents
Test Plan: Test creating a new form, reordering, marking and unmarking defaults. View new forms.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15531
Summary:
Ref T10537. For Nuance, I want to introduce new sources (like "GitHub" or "GitHub via Nuance" or something) but this needs to modularize eventually.
Split ContentSource apart so applications can add new content sources.
Test Plan:
This change has huge surface area, so I'll hold it until post-release. I think it's fairly safe (and if it does break anything, the breaks should be fatals, not anything subtle or difficult to fix), there's just no reason not to hold it for a few hours.
- Viewed new module page.
- Grepped for all removed functions/constants.
- Viewed some transactions.
- Hovered over timestamps to get content source details.
- Added a comment via Conduit.
- Added a comment via web.
- Ran `bin/storage upgrade --namespace XXXXX --no-quickstart -f` to re-run all historic migrations.
- Generated some objects with `bin/lipsum`.
- Ran a bulk job on some tasks.
- Ran unit tests.
{F1190182}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10537
Differential Revision: https://secure.phabricator.com/D15521
Summary: Updates the Harbormaster UI to match the new two column everywhere else.
Test Plan: Did best I could, tested builds, plans, steps, buildables. Unable to test lint/unit locally, I need to set that up. Kick the tires for me pls. :3
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15523
Summary: Ref T9456. This makes everything work, except that CircleCI doesn't fetch tags which are not ancestors of branch heads.
Test Plan: Ran passing builds through CircleCI.
Reviewers: chad
Reviewed By: chad
Subscribers: dpaola2, JustinTulloss
Maniphest Tasks: T9456
Differential Revision: https://secure.phabricator.com/D14288
Summary: Ref T9456. Some rough edges and we can't complete the build yet since I haven't written a webhook, but this mostly seems to be working.
Test Plan:
- Ran this build on some stuff.
- Ran a normal HTTP step build to make sure I didn't break that.
{F880301}
{F880302}
{F880303}
Reviewers: chad
Reviewed By: chad
Subscribers: JustinTulloss, joshma
Maniphest Tasks: T9456
Differential Revision: https://secure.phabricator.com/D14286
Summary:
First pass at converting Differential, I likely have some buggy-poos but thought I'd toss this up now in case very bad bugs present.
To do:
- Need to put status back on Hovercards
- "Diff Detail" probably needs a better design
Test Plan: Looking at lots of diffs, admittedly I dont have harbormaster, etc, running locally. Checked Diffusion for Table of Content changes on small and large commits.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15463
Summary:
Ref T10537. More infrastructure:
- Put a `bin/nuance` in place with `bin/nuance import`. This has no useful behavior yet.
- Allow sources to be searched by substring. This supports `bin/nuance import --source whatever` so you don't have to dig up PHIDs.
Test Plan:
- Applied migrations.
- Ran `bin/nuance import --source ...` (no meaningful effect, but works fine).
- Searched for sources by substring in the UI.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10537
Differential Revision: https://secure.phabricator.com/D15436
Summary: Ref T10093. Changes must be pushed to staging before they can be landed from the web.
Test Plan:
Changes must be pushed to staging before they can be landed from the web.
{F1161909}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10093
Differential Revision: https://secure.phabricator.com/D15427
Summary:
Every caller returns `true`. This was added a long time ago for Projects, but projects are no longer subscribable.
I don't anticipate needing this in the future.
Test Plan: Grepped for this method.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D15409
Summary:
Ref T10457.
- Let blueprints be tagged so you can search and annotate them a little more easily.
- Give each blueprint type an optional icon to make things a little easier to parse visually.
Test Plan:
- Tagged blueprints.
- Searched by tags.
- Looked at nice little icons.
{F1139712}
Reviewers: chad
Reviewed By: chad
Subscribers: yelirekim
Maniphest Tasks: T10457
Differential Revision: https://secure.phabricator.com/D15392
Summary:
Ref T5822.
- After a log is closed, compress it if possible.
- Provide `bin/harbormaster archive-logs` to make it easier to change the storage format of logs.
Test Plan:
- Ran `bin/harbormaster archive-logs` on a bunch of logs, compressing and decompressing them without issues (same hashes, same decompressed size across multiple iterations).
- Ran new builds, verified logs were compressed after they closed.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T5822
Differential Revision: https://secure.phabricator.com/D15380
Summary: Ref T5822. This will make it easier to compress and archive chunks without needing to hold them in memory.
Test Plan: Ran a build, looked at some logs.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T5822
Differential Revision: https://secure.phabricator.com/D15378
Summary:
Ref T10457. Currently, this table is an ad-hoc table, but can easily be turned into a normal table.
This will make iterating over log chunks to compress and archive them easier.
Test Plan: Viewed logs, ran `bin/storage adjust` with no issues.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10457
Differential Revision: https://secure.phabricator.com/D15376
Summary:
Ref T10457. Currently, every `append()` call necessarily generates queries, and these queries are slightly inefficient if a large block of data is appended to a partial log (they do about twice as much work as they technically need to).
Use `PhutilRope` to buffer `append()` input so the logic is a little cleaner and we could add a rule like "flush logs no more than once every 500ms" later.
Test Plan:
- Ran a build, saw logs.
- Set chunk size very small, ran build, saw logs, verified small logs in database.
{F1137115}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10457
Differential Revision: https://secure.phabricator.com/D15375
Summary:
Ref T5822. This prepares for inline compression and garbage collection of build logs.
This reduces the API surface area and removes a log from the "wait" step that would just log a message every 15 seconds. (If this is actually useful, I think we find a better way to communicate it.)
Test Plan:
Ran a build, saw a log:
{F1136691}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T5822
Differential Revision: https://secure.phabricator.com/D15371
Summary:
Ref T10457. Skipped tests are almost always well-behaved (e.g., `testWindows()`, but the test is running on Linux) and not interesting, and we do not expect well-written, solid systems to necessarily have 0 skips.
Although skips //could// indicate that you have missing dependencies on a build server, and thus be a bit interesting, I think they almost always indicate that a particular test is not expected to run in the current environment.
If we wanted to tackle this problem in granular detail, we could eventually add a "Missing" status or similar which would serve as "a skip you //could// reasonably fix in this environment", but I don't think that's too interesting.
Test Plan:
Here's an example of a build result with skips: B10875
{F1136511}
I think this is clearer as "Passed", as this is the expected production state of the build.
Locally, looked at some builds.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10457
Differential Revision: https://secure.phabricator.com/D15369
Summary: Ref T10457. This gives unit test results a more first-class treatment in the Differential UI, and consolidates some rendering code.
Test Plan:
Before:
{F1135536}
After:
{F1135537}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10457
Differential Revision: https://secure.phabricator.com/D15365
Summary:
Ref T10457. When tests fail, it currently takes several clicks to figure out //why// they failed.
In this project, map rebuilds and `liberate` are fairly common failure conditions, but verifying that they were the root issue requires jumping into a build, then scrolling through a log.
Instead, display details if they're available.
Test Plan:
Before:
{F1135453}
After:
{F1135454}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9951, T10457
Differential Revision: https://secure.phabricator.com/D15363
Summary: Ref T10457. These lack color and iconography and are difficult to parse. Make them easier to read.
Test Plan:
Before:
{F1135396}
After:
{F1135399}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10457
Differential Revision: https://secure.phabricator.com/D15362
Summary: Ref T10457. This is mostly just for consitency, but I imagine it will make managing large/complex build processes easier, and if we support Herald rules it would eventually let you write "Build plan's tags include [whatever]" to apply behavior to a group of plans.
Test Plan: {F1133107}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10457
Differential Revision: https://secure.phabricator.com/D15360
Summary: Ref T10457. Allow build plans to be queried by name.
Test Plan:
- Searched for plans by name.
- Renamed a plan, searched for new name.
{F1133085}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10457
Differential Revision: https://secure.phabricator.com/D15359
Summary:
Ref T10457.
- Use EditEngine for Build Plans.
- Fix some minor issues with crumbs being inconsistent.
- Fix some minor issues with mobile menus not being consistent/available.
Test Plan:
- Created and edited build plans.
- Poked around in mobile width, verified mobile menu had the right stuff in it.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10457
Differential Revision: https://secure.phabricator.com/D15357
Summary:
Fixes T10011.
- Modernize searching for buildables.
- Prepare for `harbormaster.buildable.search`.
- Allow users to query by status (see T10011).
- Collapse the four weird "commit / diff / revision / repository" fields into two slightly less weird fields with more UI hinting?
Test Plan: {F1131918}
Reviewers: chad
Reviewed By: chad
Subscribers: Luke081515.2
Maniphest Tasks: T10011
Differential Revision: https://secure.phabricator.com/D15356
