Summary:
Ref T8095. Two general problems:
- I want Harbormaster to own all lint and unit test results.
- I don't want users to have to configure anything for `arc` to keep working automatically.
These are in conflict because generic lint/unit test ownership in Harbormaster requires that build targets exist which we can attach build results to. However, we can't currently create build targets on demand: Harbormaster assumes it is responsible for creating targets, then running code or making third-party service calls to actually run the builds.
I considered two broad approaches to let `arc` push results into Harbormaster without requiring administrators to configure some kind of "arc results" build plan:
# Add magic target PHIDs like `PHID-MAGIC-this-is-really-arc-unit`.
# Add new code to build real targets with real PHIDs.
(1) is probably a bit less work to get off the ground, but I think it's worse overall and very likely to create more problems in the long run. I particularly worry that it will lead to a small amount of special casing in a very large number of places, which seems more fragile.
(2) is more work upfront but I think does a better job of putting all the special casing in one place that we can, e.g., more reasonably unit test, and letting the rest of the code rarely/never care about this case since it's just dealing with normal plans/steps/targets as far as it can tell.
This diff introduces "autoplans", which are source templates for plans/steps. This let us "push" these targets into Harbormaster. Hypthetically, any process "like" arc can use autoplans to upload test/lint/etc results. In practice, probably only `arc` will ever use this, but I think it's still quite a bit cleaner than the alternative despite all the generality.
Workflow is basically:
- `arc` creates a diff.
- `arc` calls `harbormaster.queryautotargets`, passing the diff PHID and saying "I have some lint and unit results I want to stick on this thing".
- Harbormaster builds the plan, steps, and targets (if any of them don't already exist), and hands back the target PHIDs so `arc` has a completely standard-looking place to put results.
- `arc` uploads the test results to the right targets, as though Harbormaster had asked it to run unit/lint in the first place.
(This doesn't actually do any of that yet, just sets things up.)
I'll maybe doc turn that ^^^^^^ into a doc for posterity since I think it's hard to guess what an "autotarget" is, but I'm going to grab some lunch first.
Test Plan:
- Added unit tests to make sure we can build these things properly.
- Used `harbormaster.queryautotargets` to build autotargets for a bunch of diffs.
- Verified targets come up in "waiting for message" state.
- Verified plans and steps are not editable.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: hach-que, epriestley
Maniphest Tasks: T8095
Differential Revision: https://secure.phabricator.com/D13345
Summary:
Ref T8095. This is just general groundwork for more exciting changes:
- Use more modern conventions around controllers, UI elements, and dialogs.
- Provide real CAN_EDIT policies and policy checks (they just don't do anything yet).
Test Plan:
- Used all affected controllers.
- Faked CAN_EDIT to POLICY_NOONE and verified everything was greyed out and unselectable.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T8095
Differential Revision: https://secure.phabricator.com/D13344
Summary:
Ref T8095.
Harbormaster has a `BuildItem` class, but it has no table and is unused. This was an earlier idea about representing lint/unit results and some other possible types of messages, but I think we want to be more specific than this.
Remove `BuildItem` and add `Lint` and `Unit` storage. These tables roughly parallel how we store lint/unit messages today, with some guesses about how where they'll go in the future.
Test Plan: Ran `bin/storage upgrade` and got a clean adjust out of it.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: hach-que, epriestley
Maniphest Tasks: T8095
Differential Revision: https://secure.phabricator.com/D13329
Summary: Fixes T8626. I guess that `$this->getViewer()->renderHandle(null)` renders as "Unknown Object", which is somewhat odd to me.
Test Plan: Viewed an atom from a book without a repository linked to it.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin
Maniphest Tasks: T8626
Differential Revision: https://secure.phabricator.com/D13370
Summary: From IRC, this article was removed in D12713.
Test Plan: N/A
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin
Differential Revision: https://secure.phabricator.com/D13371
Summary: Ref T8617. Provide general documentation with tools for debugging hangs and slow pages. Update DarkConsole docs and discuss how to use Services and XHProf. Explain what Multimeter is for and how to use it. Update XHProf docs and provide some usage hints.
Test Plan: Read documentation.
Reviewers: joshuaspence, btrahan
Reviewed By: joshuaspence, btrahan
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T8617
Differential Revision: https://secure.phabricator.com/D13359
Summary: Fixes T6839. Sometimes, worker tasks go astray for whatever reason. This automates the step of `bin/repository importing | xargs | mangle mangle | bin/repostiory reparse`.
Test Plan: Ran various flavors of the command, got good looking results.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6839
Differential Revision: https://secure.phabricator.com/D13362
Summary:
Fixes T8616. The rules for contributors have come up a few times recently, so update this document to give more complete advice.
Also try to do a better job with "adding new classes" (previously: libphutil libraries blah blah no one cares).
Test Plan: Read documents.
Reviewers: btrahan, joshuaspence
Reviewed By: btrahan, joshuaspence
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T8616
Differential Revision: https://secure.phabricator.com/D13358
Summary: Ref T8096. This shows the build plan name on the Harbormaster build plan view controller. Without this, the name is not displayed anywhere on the page when you're viewing a build plan's configuration (which makes things confusing if you're updating a bunch of build plans at once).
Test Plan: Viewed a build plan, saw the build plan name on the page.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin
Projects: #harbormaster
Maniphest Tasks: T8096
Differential Revision: https://secure.phabricator.com/D13356
Ref T8618. I think this was just typoed in D13070.
Immediate issue is that we're looking for symbols with the book name as a "phid".
Auditors: joshuaspence
Summary: Closes T8032
Test Plan: Verify that when editing a calendar event's description, there is a UI bar helping with adding markup to the description. Also verify that markup is displayed correctly on the event page once the event has been updated.
Reviewers: lpriestley, #blessed_reviewers, epriestley
Reviewed By: lpriestley, #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T8032
Differential Revision: https://secure.phabricator.com/D13355
Summary: Fixes T8352. Associate Diviner books and atoms with a repository. This relationship is not really surfaced anywhere in the UI but provides metadata that contextualises search results. Depends on D13091.
Test Plan: Ran `diviner generate --repository ARC` and then went to `/diviner/book/arcanist/`.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7703, T8352
Differential Revision: https://secure.phabricator.com/D13070
Summary: This drops the Windows-specific escaping code for the creation of directories when acquiring a lease. This is basically the change from D10378 without the other, no longer necessary changes.
Test Plan: This code hasn't been run in a production environment for a while (any instances of Phabricator using Drydock / Harbormaster with Windows have had this code removed by the D10378 patch for a while).
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Projects: #drydock
Differential Revision: https://secure.phabricator.com/D13341
Summary: Ref T2015. This code is only relevant when attempting to run commands on a Windows host over SSH. Since SSH on Windows is extremely fragile and hard to maintain, and WinRM is a better long-term solution, drop this code (which will end up being unused when later diffs introduce the WinRM command interface).
Test Plan: This code won't be used when D10495 lands.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Projects: #drydock
Maniphest Tasks: T2015
Differential Revision: https://secure.phabricator.com/D13340
Summary: Ref T8320. Fixes T8427. This is still a little funky because Owners has weird name rules, but should fix the bugs (unselectable packages) in T8427.
Test Plan: Browsed Owners typaheads, used various search functions.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T8320, T8427
Differential Revision: https://secure.phabricator.com/D13349
Summary:
Fixes T8601. To reproduce the problem:
- Set your time preference to `""` (the empty string). This isn't possible from the modern UI, but can be done with "Right Click > Inspect Element", or users may have carried it forward from an older setting (this is the case with me and @hach-que on this install).
- Load Calendar with some events.
- This parses an epoch, which sets `valueTime` to `""` (since there are no format characters in the preference) and then `getEpoch()` fails because `strlen($time)` is 0.
- Since `getEpoch()` failed, `getDateTime()` also fails.
To fix this:
- Only permit the date and time preferences to have valid values.
Test Plan:
- Loaded page before patch, saw fatal.
- Applied patch.
- No more fatal.
- Viewed tooltips, dates/times, dates/times in other apps.
- Changed my preferences, saw them respected.
Reviewers: lpriestley
Reviewed By: lpriestley
Subscribers: epriestley, hach-que
Maniphest Tasks: T8601
Differential Revision: https://secure.phabricator.com/D13346
Summary:
Fixes T8597. Second issue there is that if you look at a huge file in Diffusion (like `/path/to/300MB.pdf`) we pull the whole thing over Conduit upfront, then try to shove it into file storage.
Instead, pull no more than the chunk limit (normally 4MB) and don't spend more than 10s pulling data.
If we get 4MB of data and/or time out, just fail with a message in the vein of "this is a really big file".
Eventually, we could improve this:
- We can determine the //size// of very large files correctly in at least some VCSes, this just takes a little more work. This would let us show the true filesize, at least.
- We could eventually stream the data out of the VCS, but we can't stream data over Conduit right now and this is a lot of work.
This is just "stop crashing".
Test Plan: Changed limits to 0.01 seconds and 8 bytes and saw reasonable errors. Changed them back and got normal beahvior.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T8597
Differential Revision: https://secure.phabricator.com/D13348
Summary: Fixes T8603. For automatic 'order' parameters provided by infrastructure en route to T7715, clarify that they are optional (we will use the default builtin order for the underlying Query if an order is not provided).
Test Plan: Used web UI to see "optional" hint.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T8603
Differential Revision: https://secure.phabricator.com/D13342
Summary:
Ref T8597. If a page issues a large query (like inserting a blob into file storage), we may try to utf8ize the entire thing. This is slow and pointless.
Instead, truncate tab data after 4096 bytes before sanitizing.
Test Plan: Adjusted limit to 256 bytes, saw long queries get truncated reasonably.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T8597
Differential Revision: https://secure.phabricator.com/D13347
Summary: Closes T8050, Format Calendar list objects
Test Plan: Open Calendar list, check that new formatting is true to mocks.
Reviewers: chad, #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T8050
Differential Revision: https://secure.phabricator.com/D13318
Summary: Ref T8577, More correct fix - set default value array() for `getParameter` call.
Test Plan: Make sure saved queries like Month View, Day View, and Upcoming Events, still work.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin
Maniphest Tasks: T8577
Differential Revision: https://secure.phabricator.com/D13330
Summary: Fixes T8599. I'm not sure how to reproduce the original issue, but I'm fairly confident that the issue is that the issue is that `execute()` is not called on the query object.
Test Plan: Created a Harbormaster build plan with a single "Lease Host" step. Ran `./bin/harbormaster build --plan 1 D1` from the command line and hit the exception as described in T8599. Applied patch and hit a different exception (which I think is just because I don't know how to use #drydock and #harbormaster).
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: hach-que, epriestley, Korvin
Maniphest Tasks: T8599
Differential Revision: https://secure.phabricator.com/D13335
Summary: Fixes T8589. Adds a bunch of new languages to the syntax highlighting config options so that they are supported by #paste.
Test Plan: Saw new filetypes in Paste.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T8589
Differential Revision: https://secure.phabricator.com/D13337
Summary: Fixes T8565, Quoted text remarkup should be smart enough to know when to add a '>' and when to add '> '
Test Plan: Open an object with remarkup comments, add 'quote', select that text click the quote button in the remarkup menu, text should become '> quote'. Select and click again, text should become '>> quote'.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T8565
Differential Revision: https://secure.phabricator.com/D13334
Summary:
Ref T8449. Before we show a policy hint in the header of an object, compare it to the space policy (if one exists).
If the space policy is strictly stronger (more restrictive -- for example, the Space policy is 'members of X', and the object policy is 'public'), show the space policy instead.
See discussion on T8376.
Test Plan: {F509126}
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T8449
Differential Revision: https://secure.phabricator.com/D13328
Summary:
Ref T8575. We run a big "(A) UNION (B)" query on the home page and on the main Differential page.
"A" can always be improved by using `%Ls`, so it can use the second half of the `(authorPHID, status)` key.
"B" can sometimes be improved if the fraction of open revisions is smaller than the fraction of revisions you are reviewing. This is true for me on secure.phabricator.com (I'm a reviewer, either directly or via 'Blessed Reviewers', on about 80% of revisions, but <5% are open). In these cases, a `(status, phid)` key is more efficient.
Test Plan: Tweaked queries and added keys on this server, saw less derpy query plans and performance.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T8575
Differential Revision: https://secure.phabricator.com/D13325
Summary:
Ref T8575. Although we cache spaces as a whole, we don't cache viewer spaces. This can still do a lot of work if they have complex policies.
Instead, cache them in the request cache.
Test Plan: Saw this account for 37% of a page in produciton (303ms on on 835ms).
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T8575
Differential Revision: https://secure.phabricator.com/D13324
Summary: These class properties don't seem to be used.
Test Plan: Eyeball it.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Differential Revision: https://secure.phabricator.com/D13295
Summary:
Ref T8575. Because orderable columns need to build custom fields, they are relatively expensive to build. Use the request cache.
(The request cache is technically more correct than `static`, because configuration may change between requests and we may eventually reuse interpreters.)
Test Plan: Saw home page time drop 39% (from 462ms to 283ms).
Reviewers: btrahan, avivey
Reviewed By: avivey
Subscribers: avivey, epriestley
Maniphest Tasks: T8575
Differential Revision: https://secure.phabricator.com/D13322
Summary: Ref T8575. We check if users can see applications frequently, and caching on the Query isn't especially effective. Use the new Request cache instead.
Test Plan:
- Saw `/feed/` drop 7% (from ~830ms to ~770ms) on profiles.
Reviewers: btrahan, avivey
Reviewed By: avivey
Subscribers: avivey, epriestley
Maniphest Tasks: T8575
Differential Revision: https://secure.phabricator.com/D13321
Summary:
Ref T8574. Currently, failures during mail body construction, feed publishing, or search indexing could cause us to retry the publishing task and potentially send duplicate mail.
Instead, build (but do not send) the mail first, then send all the mail at the very end.
This isn't completley perfect, but should make it enormously harder for duplicate mail to be generated.
Test Plan: Sent some mail, ran the daemons, saw it show up normally in the outbound queue.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T8574
Differential Revision: https://secure.phabricator.com/D13320
Summary:
Ref T8574. This could fail because the target is disabled (as here), or doesn't have access to the API, or the API request needs to be satisfied by a different host which isn't available.
In any of these cases, just show the failure and continue generating the mail. This field isn't important enough to block the mail, and many of these errors are permanent.
(I'll follow up on T8574 with some more permanent ideas to address this class of issue.)
Test Plan: Faked API call failure, generated mail, saw clean generation of mail with a failure message.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T8574
Differential Revision: https://secure.phabricator.com/D13319
Summary: Encountered this playing with T8402 on my test instance. I think warning the user about adding a trailing "/" is unnecessary so don't do it. I think its confusing to not call out spaces / to lump them in with special characters so call out the sapces.
Test Plan: moved a page around and verified no warning if the slug is missing a "/" as user specified and that a change to spaces is called out
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Differential Revision: https://secure.phabricator.com/D13316
Summary: Fixes T8346. Also gets rid of the air quotes on Auth and uses the "Appname Application" style we use elsewhere.
Test Plan: slapped an "|| true" in the checking logic and verified the link was rendered in a usable fashion just below the error message.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T8346
Differential Revision: https://secure.phabricator.com/D13315
Summary: Fixes T8518. Defaulting $events to null creates carnage further down the stack. Instead, default it to array() so foreach, array_select_keys, etc. don't barf.
Test Plan: logic and will ask the reporting user to verify since they can repro 100%
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T8518
Differential Revision: https://secure.phabricator.com/D13314
Summary: Ref T4558. If a Diviner atom is a ghost (i.e. the underlying source code has been removed), mark it as closed in the search index.
Test Plan: Searched for a ghost atom in global searcn and saw the results show "Closed".
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin
Maniphest Tasks: T4558
Differential Revision: https://secure.phabricator.com/D13297
Summary: Ref T4558. Give `DivinerAtomPHIDType` and `DivinerBookPHIDType` a font icon.
Test Plan: Saw icons in global search.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T4558
Differential Revision: https://secure.phabricator.com/D13304
Summary: Fixes T8524, T8550. These were both missing "subscribers" and Phriction was also the only application with no "other".
Test Plan:
- Project
- set user A to notify only for project subscriber changes
- made a project with user A
- subscribed user B to project
- verified notification sent to user A
- used ./bin/mail to make sure no mail was sent to user A because "This mail has tags which control which users receive it, and this recipient has not elected to receive mail with any of the tags on this message (Settings > Email Preferences)."
- Phriction
- set user A to notify only for phriction document subscriber changes
- made a document with user A
- subscribed user B to document
- verified notification sent to user A
- used ./bin/mail to make sure no mail was sent to user A because "This mail has tags which control which users receive it, and this recipient has not elected to receive mail with any of the tags on this message (Settings > Email Preferences)."
- observed option for "other" in email preferences
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T8524, T8550
Differential Revision: https://secure.phabricator.com/D13313
Summary:
Ref T4558. This diff modernizes the #diviner application. Basically:
- Add an edit controller, accessible at `/book/$BOOK/edit/`.
- Add edit/view policies.
- Added an action menu to the `DivinerBookController` to expose the edit interface.
- Allows projects to be associated with books.
- Implement edges and transactions.
- Implemented `PhabricatorApplicationTransactionInterface` in `DivinerLiveBook`.
Test Plan:
- Generated a Diviner book with `./bin/diviner generate`.
- Added projects to a book and ensured that they persisted.
- Changed the view policy on a book and made sure it was effective.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T4558
Differential Revision: https://secure.phabricator.com/D13091
Summary: Closes T8481.
Test Plan: Verify that in Passphrase > Create an option to create a Note credential exists and credentials of type Note are createable.
Reviewers: epriestley, #blessed_reviewers, eadler, lpriestley
Reviewed By: eadler
Subscribers: joshuaspence, epriestley, Korvin
Maniphest Tasks: T8481
Differential Revision: https://secure.phabricator.com/D13261
Summary: Closes T8460.
Test Plan: In the Calendar application, create a recursive event. After editing a recursion of the event, verify that the "Recurrence of Event" property is of the form "<event sequence number> of <parent event>".
Reviewers: lpriestley, epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin
Maniphest Tasks: T8460
Differential Revision: https://secure.phabricator.com/D13307
