1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-13 09:06:14 +01:00
Commit graph

13492 commits

Author SHA1 Message Date
epriestley
a0668df75a Remove "metamta.domain" and "metamta.placeholder-to-recipient" config options
Summary:
Ref T920. This simplifies mail configuration.

The "metamta.domain" option is only used to generate Thread-ID values, and we just need something that looks like a bit like a domain in order to make GMail happy. Just use the install domain. In most cases, this is almost certainly the configured value anyway. In some cases, this may cause a one-time threading break for existing threads; I'll call this out in the changelog.

The "metamta.placeholder-to-recipient" is used to put some null value in "To:" when a mail only has CCs. This is so that if you write a local client mail rule like "when I'm in CC, burn the message in a fire" it works even if all the "to" addresses have elected not to receive the mail. Instead: just send it to an unlikely address at our own domain.

I'll add some additional handling for the possiblity that we may receive this email ourselves in the next change, but it overlaps with T7477.

Test Plan: Grepped for these configuration values.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19942
2019-01-04 13:45:15 -08:00
epriestley
afa69eedd1 Remove an old digest in Celerity code and some obsolete configuration options
Summary:
Ref T12509. This upgrades a `weakDigest()` callsite to SHA256-HMAC and removes three config options:

  - `celerity.resource-hash`: Now hard-coded, since the use case for ever adjusting it was very weak.
  - `celerity.enable-deflate`: Intended to make cache inspection easier, but we haven't needed to inspect caches in ~forever.
  - `celerity.minify`: Intended to make debugging minification easier, but we haven't needed to debug this in ~forever.

In the latter two cases, the options were purely developer-focused, and it's easy to go add an `&& false` somewhere in the code if we need to disable these features to debug something, but the relevant parts of the code basically work properly and never need debugging. These options were excessively paranoid, based on the static resource enviroment at Facebook being far more perilous.

The first case theoretically had end-user utility for fixing stuck content caches. In modern Phabricator, it's not intuitive that you'd go adjust a Config option to fix this. I don't recall any users ever actually running into problems here, though.

(An earlier version of this change did more magic with `celerity.resource-hash`, but this ended up with a more substantial simplification.)

Test Plan: Grepped for removed configuration options.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D19941
2019-01-04 13:43:38 -08:00
epriestley
7e87d254ab Add a parameterized Future for Twilio API calls
Summary: Ref T920. We currently embed the Twilio PHP API, but can replace it with about 100 lines of code and get a future-oriented interface as a bonus. Add a Future so we can move toward a simpler calling convention for the API.

Test Plan: Used this future to send SMS messages via the Twilio API.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19937
2019-01-04 13:42:52 -08:00
epriestley
3963c86ad5 Pass timeline view data to comment previews, restoring Differential comment previews
Summary:
Ref T13222. In D19918, I refactored how timelines get "view data". Today, this is always additional data about which images/changesets/diffs are visible on the current revision/commit/mock, so we can tell if inline comments should be linked to a `#anchor` on the same page (if the inline is rendered there somewhere) or to a `/D123?id=1&vs=2` full link on a different page (if it isn't), but in general this could be any sort of state information about the current page that affects how the timeline should render.

Previously, comment previews did not use any specialized object code and always rendered a "generic" timeline story. This was actually a bug, but none of the code we have today cares about this (since it's all inline related, and inlines render separately) so it never impacted anything.

After the `TimelineEngine` change, the preview renders with Differential-specific code. This is more correct, but we were not passing the preview the "view data" so it broke.

This preview doesn't actually need the view data and we could just make it bail out if it isn't present, but pass it through for consistency and so this works like we'd expect if we do something fancier with view data in the future.

Test Plan: Viewed comment and inline comment previews in Differential, saw old behavior restored.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19943
2019-01-03 13:06:54 -08:00
Austin McKinley
65e89c239e Add status to PhabricatorProjectQuery->getPagingValueMap()
Summary: Probably fixes https://discourse.phabricator-community.org/t/unhandled-exception-for-certain-fields-in-maniphest-search/2263. Couldn't repro this locally, but this is almost certainly the correct fix.

Test Plan: doitlive

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19951
2019-01-03 11:34:24 -08:00
epriestley
9d5b933ed5 Remove all legacy configuration options for mailers
Summary:
Ref T920. About a year ago (in 2018 Week 6, see D19003) we moved from individually configured mailers to `cluster.mailers`, primarily to support fallback across multiple mail providers.

Since this has been stable for quite a while, drop support for the older options.

Test Plan: Grepped for all removed options.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19940
2019-01-03 04:09:21 -08:00
epriestley
cfcd35d8a3 Remove standalone SMS support in favor of a "Mail, SMS, and other media are mostly the same thing" approach
Summary:
Ref T920. Over time, mail has become much more complex and I think considering "mail", "sms", "postcards", "whatsapp", etc., to be mostly-the-same is now a more promising avenue than building separate stacks for each one.

Throw away all the standalone SMS code, including the Twilio config options. I have a separate diff that adds Twilio as a mail adapter and functions correctly, but it needs some more work to bring upstream.

This permanently destroys the `sms` table, which no real reachable code ever wrote to. I'll call this out in the changelog.

Test Plan:
  - Grepped for `SMS` and `Twilio`.
  - Ran storage upgrade.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19939
2019-01-03 04:05:20 -08:00
Austin McKinley
05a9474138 Raise warning when accidentally submitting Conduit parameters as a JSON-encoded body
Summary: See T12447 for discussion. It is reasonably intuitive to try and pass Conduit parameters via a JSON-encoded HTTP body, but if you do so, you'll get an unhelpful messsage about how method so-and-so does not accept a parameter named "your_entire_json_body". Instead, detect this mistake and advise developers to use form-encoded parameters.

Test Plan:
Got a better error when attempting to make Conduit calls from React code. Tested the following additional invocations of Conduit and got the expected results without an error:

* From the Conduit UI
* With cURL:
```
~ $ curl http://local.phacility.com:8080/api/conpherence.querythread \
>     -d api.token=api-tvv2zb565zrtueab5ddprmpxvrwb \
>     -d ids[0]=1
```
* With `arc call-conduit`:
```
~ $ echo '{
>   "ids": [
>     1
>   ]
> }' | arc call-conduit --conduit-uri http://local.phacility.com:8080/ --conduit-token api-tvv2zb565zrtueab5ddprmpxvrwb conpherence.querythread
```

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19944
2019-01-02 17:31:16 -08:00
epriestley
ea8be11add Fix a qsprintf() issue in mail queries
Summary: Ref T920. Bumped into this while looking at SMS support.

Test Plan: Loaded `/mail/`, no more `qsprintf()` warning.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19936
2019-01-02 14:39:31 -08:00
epriestley
106e90dcf0 Remove the "willApplyTransactions()" hook from ApplicationTransactionEditor
Summary:
Depends on D19908. Ref T13222. In D19897, I reordered some transaction code and affected the call order of `willApplyTransactions()`.

It turns out that we use this call for only one thing, and that thing is pretty silly: naming the raw paste data file when editing paste content.

This is only user-visible in the URL when you click "View Raw Paste" and seems exceptionally low-value, so remove the hook and pick a consistent name for the paste datafiles. (We could retain the name behavior in other ways, but it hardly seems worthwhile.)

Test Plan:
  - Created and edited a paste.
  - Grepped for `willApplyTransactions()`.

Note that `EditEngine` (vs `ApplicationTransacitonEditor`) still has a `willApplyTransactions()`, which has one callsite in Phabricator (in Calendar) and a couple in Instances. That's untouched and still works.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19909
2018-12-28 00:19:38 -08:00
epriestley
1729e7b467 Improve UI for "wait" and "answered" MFA challenges
Summary:
Depends on D19906. Ref T13222. This isn't going to win any design awards, but make the "wait" and "answered" elements a little more clear.

Ideally, the icon parts could be animated Google Authenticator-style timers (but I think we'd need to draw them in a `<canvas />` unless there's some clever trick that I don't know) or maybe we could just have the background be like a "water level" that empties out. Not sure I'm going to actually write the JS for either of those, but the UI at least looks a little more intentional.

Test Plan:
{F6070914}

{F6070915}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19908
2018-12-28 00:18:53 -08:00
epriestley
918f4ebcd8 Fix a double-prompt for MFA when recovering a password account
Summary:
Depends on D19905. Ref T13222. In D19843, I refactored this stuff but `$jump_into_hisec` was dropped.

This is a hint to keep the upgraded session in hisec mode, which we need to do a password reset when using a recovery link. Without it, we double prompt you for MFA: first to upgrade to a full session, then to change your password.

Pass this into the engine properly to avoid the double-prompt.

Test Plan:
  - Used `bin/auth recover` to get a partial session with MFA enabled and a password provider.
  - Before: double MFA prompt.
  - After: session stays upgraded when it becomes full, no second prompt.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19906
2018-12-28 00:17:47 -08:00
epriestley
ca39be6091 Make partial sessions expire after 30 minutes, and do not extend them
Summary:
Depends on D19904. Ref T13226. Ref T13222. Currently, partial sessions (where you've provided a primary auth factor like a password, but not yet provided MFA) work like normal sessions: they're good for 30 days and extend indefinitely under regular use.

This behavior is convenient for full sessions, but normal users don't ever spend 30 minutes answering MFA, so there's no real reason to do it for partial sessions. If we add login alerts in the future, limiting partial sessions to a short lifetime will make them more useful, since an attacker can't get one partial session and keep extending it forever while waiting for an opportunity to get past your MFA.

Test Plan:
  - Did a partial login (to the MFA prompt), checked database, saw a ~29 minute partial session.
  - Did a full login, saw session extend to ~30 days.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13226, T13222

Differential Revision: https://secure.phabricator.com/D19905
2018-12-28 00:17:01 -08:00
epriestley
38c48ae7d0 Remove support for the "TYPE_AUTH_WILLLOGIN" event
Summary:
Depends on D19903. Ref T13222. This was a Facebook-specific thing from D6202 that I believe no other install ever used, and I'm generally trying to move away from the old "event" system (the more modern modular/engine patterns generally replace it).

Just drop support for this. Since the constant is being removed, anything that's actually using it should break in an obvious way, and I'll note this in the changelog.

There's no explicit replacement but I don't think this hook is useful for anything except "being Facebook in 2013".

Test Plan:
  - Grepped for `TYPE_AUTH_WILLLOGIN`.
  - Logged in.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19904
2018-12-28 00:16:22 -08:00
epriestley
ff49d1ef77 Allow "bin/auth recover" to generate a link which forces a full login session
Summary:
Depends on D19902. Ref T13222. This is mostly a "while I'm in here..." change since MFA is getting touched so much anyway.

Doing cluster support, I sometimes need to log into user accounts on instances that have MFA. I currently accomplish this by doing `bin/auth recover`, getting a parital session, and then forcing it into a full session in the database. This is inconvenient and somewhat dangerous.

Instead, allow `bin/auth recover` to generate a link that skips the "partial session" stage: adding required MFA, providing MFA, and signing legalpad documents.

Anyone who can run `bin/auth recover` can do this anyway, this just reduces the chance I accidentally bypass MFA on the wrong session when doing support stuff.

Test Plan:
  - Logged in with `bin/auth recover`, was prompted for MFA.
  - Logged in with `bin/auth recover --force-full-session`, was not prompted for MFA.
  - Did a password reset, followed reset link, was prompted for MFA.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19903
2018-12-28 00:15:36 -08:00
epriestley
6a6db0ac8e Allow tokens to be awarded to MFA-required objects
Summary:
Depends on D19901. Ref T13222. See PHI873. Currently, the MFA code and the (older, not-really-transactional) token code don't play nicely.

In particular, if the Editor throws we tend to get half an effect applied.

For now, just make this work. Some day it could become more modern so that the transaction actually applies the write.

Test Plan: Awarded and rescinded tokens from an MFA-required object.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19902
2018-12-28 00:14:48 -08:00
epriestley
efb01bf34f Allow "MFA Required" objects to be edited without MFA if the edit is only creating inverse edges
Summary:
Depends on D19900. Ref T13222. See PHI873. When an object requires MFA, we currently require MFA for every transaction.

This includes some ambiguous cases like "unsubscribe", but also includes "mention", which seems like clearly bad behavior.

Allow an "MFA" object to be the target of mentions, "edit child tasks", etc.

Test Plan:
  - Mentioned an MFA object elsewhere (no MFA prompt).
  - Made an MFA object a subtask of a non-MFA object (no MFA prompt).
  - Tried to edit an MFA object normally (still got an MFA prompt).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19901
2018-12-28 00:12:49 -08:00
epriestley
1c89b3175f Improve UI messaging around "one-shot" vs "session upgrade" MFA
Summary:
Depends on D19899. Ref T13222. When we prompt you for one-shot MFA, we currently give you a lot of misleading text about your session staying in "high security mode".

Differentiate between one-shot and session upgrade MFA, and give the user appropriate cues and explanatory text.

Test Plan:
  - Hit one-shot MFA on an "mfa" task in Maniphest.
  - Hit session upgrade MFA in Settings > Multi-Factor.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19900
2018-12-28 00:11:36 -08:00
epriestley
d3c325c4fc Allow objects to be put in an "MFA required for all interactions" mode, and support "MFA required" statuses in Maniphest
Summary:
Depends on D19898. Ref T13222. See PHI873. Allow objects to opt into an "MFA is required for all edits" mode.

Put tasks in this mode if they're in a status that specifies it is an `mfa` status.

This is still a little rough for now:

  - There's no UI hint that you'll have to MFA. I'll likely add some hinting in a followup.
  - All edits currently require MFA, even subscribe/unsubscribe. We could maybe relax this if it's an issue.

Test Plan:
  - Edited an MFA-required object via comments, edit forms, and most/all of the extensions. These prompted for MFA, then worked correctly.
  - Tried to edit via Conduit, failed with a reasonably comprehensible error.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19899
2018-12-28 00:10:54 -08:00
epriestley
3da9844564 Tighten some MFA/TOTP parameters to improve resistance to brute force attacks
Summary:
Depends on D19897. Ref T13222. See some discussion in D19890.

  - Only rate limit users if they're actually answering a challenge, not if they're just clicking "Wait Patiently".
  - Reduce the number of allowed attempts per hour from 100 back to 10.
  - Reduce the TOTP window from +/- 2 timesteps (allowing ~60 seconds of skew) to +/- 1 timestep (allowing ~30 seconds of skew).
  - Change the window where a TOTP response remains valid to a flat 60 seconds instead of a calculation based on windows and timesteps.

Test Plan:
  - Hit an MFA prompt.
  - Without typing in any codes, mashed "submit" as much as I wanted (>>10 times / hour).
  - Answered prompt correctly.
  - Mashed "Wait Patiently" as much as I wanted (>>10 times / hour).
  - Guessed random numbers, was rate limited after 10 attempts.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19898
2018-12-28 00:10:13 -08:00
epriestley
543f2b6bf1 Allow any transaction group to be signed with a one-shot "Sign With MFA" action
Summary:
Depends on D19896. Ref T13222. See PHI873. Add a core "Sign With MFA" transaction type which prompts you for MFA and marks your transactions as MFA'd.

This is a one-shot gate and does not keep you in MFA.

Test Plan:
  - Used "Sign with MFA", got prompted for MFA, answered MFA, saw transactions apply with MFA metadata and markers.
  - Tried to sign alone, got appropriate errors.
  - Tried to sign no-op changes, got appropriate errors.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19897
2018-12-28 00:09:30 -08:00
epriestley
10db27833b Remove old Phrequent propery rendering code and show "Time Spent" in higher precision
Summary:
See <https://discourse.phabricator-community.org/t/how-to-get-total-time-spent-on-a-task-in-minutes-or-hours/2241>.

Phrequent has two nearly-identical copies of its rendering code: one for old "property event" objects and one for newer "curtain" objects. In the upstream, both trackable object types (tasks and revisions) use curtains, so throw away the old code since it isn't reachable. Third-party trackable objects can update to the curtain UI, but it's unlikely they exist.

Render the remaining curtain UI with more precision, so we show "Time Spent: 2d, 11h, 49m" instead of "Time Spent: 2d".

Test Plan: {F6074404}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19927
2018-12-28 00:07:25 -08:00
epriestley
15df57f1c8 In Webhooks, give errors human-readable labels and show reminder text for "Silent Mode"
Summary:
Depends on D19928. See <https://discourse.phabricator-community.org/t/firehose-webhook-not-working-with-self-hosted-requestbin-instance/2240/>.

Currently, we report "hook" and "silent", which are raw internal codes.

Instead, report human-readable labels so the user gets a better hint about what's going on ("In Silent Mode").

Also, render a "hey, you're in silent mode so none of this will work" reminder banner in this UI.

Test Plan:
{F6074421}

Note:

  - New warning banner.
  - Table has more human-readable text ("In Silent Mode").

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19929
2018-12-28 00:05:46 -08:00
epriestley
3c65601285 Implement "@{config:...}" as a real Remarkup rule
Summary:
See <https://discourse.phabricator-community.org/t/firehose-webhook-not-working-with-self-hosted-requestbin-instance/2240/>. I want to make it easier to link to configuration from system text.

We currently use this weird hack with `{{...}}` that only works within Config itself. Instead, use `@{config:...}`, which is already used by Diviner for `@{class:...}`, etc., so it shouldn't conflict with anything.

Test Plan: Viewed config options, clicked links to other config options.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19928
2018-12-28 00:05:09 -08:00
epriestley
e98ee73602 Fix the last remaining (?) continue inside switch
Summary: See <https://discourse.phabricator-community.org/t/error-on-project-creation-or-edition-with-php7-3/2236/>. This is the only remaining case that the linter rule in D19931 detected in libphutil, arcanist, or Phabricator.

Test Plan: Ran `arc lint --everything ...` in all three repositories, only hit this one.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19932
2018-12-28 00:04:25 -08:00
epriestley
b3faa2874c Restore a Mock key to Pholio Images
Summary: Ref T11351. We only query for images by PHID or by Mock, so the only key we need for now is `<mockPHID>`.

Test Plan: Ran `bin/storage upgrade`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19934
2018-12-28 00:03:48 -08:00
epriestley
e469f8594e Implement Pholio file add/remove transactions without "applyInitialEffects"
Summary:
Depends on D19924. Ref T11351. Like in D19924, apply these transactions by accepting PHIDs instead of objects so we don't need to juggle the `Image` objects down to PHIDs in `applyInitialEffects`.

(Validation is a little light here for now, but only first-party code can reach this, and you can't violate policies or do anything truly bad even if you could pick values to feed in here.)

Test Plan: Created and edited Mocks; added, removed, and reordered images in a Pholio Mock.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19926
2018-12-28 00:02:55 -08:00
epriestley
741fb747db Implement "replace" transactions in Pholio without "applyInitialEffects"
Summary:
Depends on D19923. Ref T11351. Currently, this transaction takes an `Image` as the `newValue` and uses some magic to reduce it into PHIDs by the time we're done.

This creates some problems today where I'd like to get rid of `applyInitialEffects` for MFA code. In the future, it creates a problem becuase there's no way to pass an entire `Image` object over the API.

Instead, create the `Image` first, then just provide the PHID. This is generally simpler, will work well with the API in the future, and stops us from needing any `applyInitialEffects` stuff.

Test Plan: Replaced images in a Pholio mock.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19924
2018-12-28 00:02:21 -08:00
epriestley
b1e7e3a10e Reduce the amount of weird "static" and "cache" behavior in Pholio query classes
Summary: Depends on D19922. Ref T11351. These query classes have some slightly weird behavior, including `public static function loadImages(...)`. Convert all this stuff into more standard query patterns.

Test Plan: Grepped for callsites, browsed around in Pholio.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19923
2018-12-20 15:32:13 -08:00
epriestley
1e2bc7775b Remove the onboard "mailKey" from Pholio Mocks
Summary: Depends on D19921. Ref T11351. Ref T13065. Update Pholio to use the shared mail infrastructure. See D19670 for a previous change in this vein.

Test Plan: Ran upgrade, spot-checked that everything made it into the new table alive.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13065, T11351

Differential Revision: https://secure.phabricator.com/D19922
2018-12-20 15:30:02 -08:00
epriestley
28989ac231 Make the Pholio Mock "getImages" / "getAllImages" API more clear
Summary:
Depends on D19920. Ref T11351. Currently, "images" and "all images" are attached to Mocks separately, and `getImages()` gets you only some images.

Clean this up slightly:

  - One attach method; attach everything.
  - Two getters, one for "images" (returns all images); one for "active images" (returns active images).

Test Plan: Browsed around Pholio without any apparent behavioral changes.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19921
2018-12-20 15:29:04 -08:00
epriestley
11cf8f05b1 Remove "getApplicationTransactionObject()" from ApplicationTransactionInterface
Summary:
Depends on D19919. Ref T11351. This method appeared in D8802 (note that "get...Object" was renamed to "get...Transaction" there, so this method was actually "new" even though a method of the same name had existed before).

The goal at the time was to let Harbormaster post build results to Diffs and have them end up on Revisions, but this eventually got a better implementation (see below) where the Harbormaster-specific code can just specify a "publishable object" where build results should go.

The new `get...Object` semantics ultimately broke some stuff, and the actual implementation in Differential was removed in D10911, so this method hasn't really served a purpose since December 2014. I think that broke the Harbormaster thing by accident and we just lived with it for a bit, then Harbormaster got some more work and D17139 introduced "publishable" objects which was a better approach. This was later refined by D19281.

So: the original problem (sending build results to the right place) has a good solution now, this method hasn't done anything for 4 years, and it was probably a bad idea in the first place since it's pretty weird/surprising/fragile.

Note that `Comment` objects still have an unrelated method with the same name. In that case, the method ties the `Comment` storage object to the related `Transaction` storage object.

Test Plan: Grepped for `getApplicationTransactionObject`, verified that all remaining callsites are related to `Comment` objects.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19920
2018-12-20 15:16:19 -08:00
epriestley
937e88c399 Remove obsolete, no-op implementations of "willRenderTimeline()"
Summary:
Depends on D19918. Ref T11351. In D19918, I removed all calls to this method. Now, remove all implementations.

All of these implementations just `return $timeline`, only the three sites in D19918 did anything interesting.

Test Plan: Used `grep willRenderTimeline` to find callsites, found none.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19919
2018-12-20 15:04:49 -08:00
epriestley
6c43d1d52c Remove "willRenderTimeline()" from ApplicationTransactionInterface
Summary:
Depends on D19914. Ref T11351. Some of the Phoilo rabbit holes go very deep.

`PhabricatorApplicationTransactionInterface` currently requires you to implement `willRenderTimeline()`. Almost every object just implements this as `return $timeline`; only Pholio, Diffusion, and Differential specialize it. In all cases, they are specializing it mostly to render inline comments.

The actual implementations are a bit of a weird mess and the way the data is threaded through the call stack is weird and not very modern.

Try to clean this up:

  - Stop requiring `willRenderTimeline()` to be implemented.
  - Stop requiring `getApplicationTransactionViewObject()` to be implemented (only the three above, plus Legalpad, implement this, and Legalpad's implementation is a no-op). These two methods are inherently pretty coupled for almost any reasonable thing you might want to do with the timeline.
  - Simplify the handling of "renderdata" and call it "View Data". This is additional information about the current view of the transaction timeline that is required to render it correctly. This is only used in Differential, to decide if we can link an inline comment to an anchor on the same page or should link it to another page. We could perhaps do this on the client instead, but having this data doesn't seem inherently bad to me.
  - If objects want to customize timeline rendering, they now implement `PhabricatorTimelineInterface` and provide a `TimelineEngine` which gets a nice formal stack.

This leaves a lot of empty `willRenderTimeline()` implementations hanging around. I'll remove these in the next change, it's just going to be deleting a couple dozen copies of an identical empty method implementation.

Test Plan:
  - Viewed audits, revisions, and mocks with inline comments.
  - Used "Show Older" to page a revision back in history (this is relevant for "View Data").
  - Grepped for symbols: willRenderTimeline, getApplicationTransactionViewObject, Legalpad classes.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19918
2018-12-20 14:55:07 -08:00
epriestley
21f07bf6f7 Make Images in Pholio refer to mocks by PHID instead of ID
Summary:
Ref T11351. In Pholio, we currently use a `mockID`, but a `mockPHID` is generally preferable / more modern / more flexible. In particular, we need PHIDs to load handles and prefer PHIDs when exposing information to the API, and using PHIDs internally makes a bunch of things easier/better/faster and ~nothing harder/worse/slower.

I'll add some inlines about a few things.

Test Plan: Ran migrations, spot-checked database for sanity. Loaded Pholio, saw data unchanged. Created and edited images.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19914
2018-12-20 14:54:25 -08:00
epriestley
961fd7e849 In Legalpad, prompt for MFA at the end of the workflow instead of the beginning
Summary: Depends on D19895. Ref T13222. This is a simple behavioral improvement for the current MFA implementation in Legalpad: don't MFA the user and //then// realize that they forgot to actually check the box.

Test Plan:
  - Submitted form without the box checked, got an error saying "check the box" instead of MFA.
  - Submitted the form with the box checked, got an MFA prompt.
  - Passed the MFA gate, got a signed form.
  - Tried to sign another form, hit MFA timed lockout.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19896
2018-12-20 14:51:25 -08:00
epriestley
b63783c067 Carry MFA responses which have been "answered" but not "completed" through the MFA workflow
Summary:
Depends on D19894. Ref T13222. See PHI873. When you provide a correct response to an MFA challenge, we mark it as "answered".

Currently, we never let you reuse an "answered" token. That's usually fine, but if you have 2+ factors on your account and get one or more (but fewer than all of them) right when you submit the form, you need to answer them all again, possibly after waiting for a lockout period. This is needless.

When you answer a challenge correctly, add a hidden input with a code proving you got it right so you don't need to provide another answer for a little while.

Why not just put your response in a form input, e.g. `<input type="hidden" name="totp-response" value="123456" />`?

  - We may allow the "answered" response to be valid for a different amount of time than the actual answer. For TOTP, we currently allow a response to remain valid for 60 seconds, but the actual code you entered might expire sooner.
  - In some cases, there's no response we can provide (with push + approve MFA, you don't enter a code, you just tap "yes, allow this" on your phone). Conceivably, we may not be able to re-verify a push+approve code if the remote implements one-shot answers.
  - The "responseToken" stuff may end up embedded in normal forms in some cases in the future, and this approach just generally reduces the amount of plaintext MFA we have floating around.

Test Plan:
  - Added 2 MFA tokens to my account.
  - Hit the MFA prompt.
  - Provided one good response and one bad response.
  - Submitted the form.
  - Old behavior: good response gets locked out for ~120 seconds.
  - New behavior: good response is marked "answered", fixing the other response lets me submit the form.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19895
2018-12-20 14:46:45 -08:00
epriestley
ce953ea447 Explicitly mark MFA challenges as "answered" and "completed"
Summary:
Depends on D19893. Ref T13222. See PHI873. A challenge is "answered" if you provide a valid response. A challenge is "completed" if we let you through the MFA check and do whatever actual action the check is protecting.

If you only have one MFA factor, challenges will be "completed" immediately after they are "answered". However, if you have two or more factors, it's possible to "answer" one or more prompts, but fewer than all of the prompts, and end up with "answered" challenges that are not "completed".

In the future, it may also be possible to answer all the challenges but then have an error occur before they are marked "completed" (for example, a unique key collision in the transaction code). For now, nothing interesting happens between "answered" and "completed". This would take the form of the caller explicitly providing flags like "wait to mark the challenges as completed until I do something" and "okay, mark the challenges as completed now".

This change prevents all token reuse, even on the same workflow. Future changes will let the answered challenges "stick" to the client form so you don't have to re-answer challenges for a short period of time if you hit a unique key collision.

Test Plan:
  - Used a token to get through an MFA gate.
  - Tried to go through another gate, was told to wait for a long time for the next challenge window.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19894
2018-12-20 14:45:22 -08:00
epriestley
657f3c3806 When accepting a TOTP response, require it respond explicitly to a specific challenge
Summary:
Depends on D19890. Ref T13222. See PHI873. Currently, we only validate TOTP responses against the current (realtime) timestep. Instead, also validate them against a specific challenge.

This mostly just moves us toward more specifically preventing responses from being reused, and supporting flows which must look more like this (SMS/push).

One rough edge here is that during the T+3 and T+4 windows (you request a prompt, then wait 60-120 seconds to respond) only past responses actually work (the current code on your device won't). For example:

  - At T+0, you request MFA. We issue a T+0 challenge that accepts codes T-2, T-1, T+0, T+1, and T+2. The challenge locks out T+3 and T+4 to prevent the window from overlapping with the next challenge we may issue (see D19890).
  - If you wait 60 seconds until T+3 to actually submit a code, the realtime valid responses are T+1, T+2, T+3, T+4, T+5. The challenge valid responses are T-2, T-1, T+0, T+1, and T+2. Only T+1 and T+2 are in the intersection. Your device is showing T+3 if the clock is right, so if you type in what's shown on your device it won't be accepted.
  - This //may// get refined in future changes, but, in the worst case, it's probably fine if it doesn't. Beyond 120s you'll get a new challenge and a full [-2, ..., +2] window to respond, so this lockout is temporary even if you manage to hit it.
  - If this //doesn't// get refined, I'll change the UI to say "This factor recently issued a challenge which has expired, wait N seconds." to smooth this over a bit.

Test Plan:
  - Went through MFA.
  - Added a new TOTP factor.
  - Hit some error cases on purpose.
  - Tried to use an old code a moment after it expired, got rejected.
  - Waited 60+ seconds, tried to use the current displayed factor, got rejected (this isn't great, but currently expected).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19893
2018-12-20 14:44:35 -08:00
epriestley
0673e79d6d Simplify and correct some challenge TTL lockout code
Summary:
Depends on D19889. Ref T13222. Some of this logic is either not-quite-right or a little more complicated than it needs to be.

Currently, we TTL TOTP challenges after three timesteps -- once the current code could no longer be used. But we actually have to TTL it after five timesteps -- once the most-future acceptable code could no longer be used. Otherwise, you can enter the most-future code now (perhaps the attacker compromises NTP and skews the server clock back by 75 seconds) and then an attacker can re-use it in three timesteps.

Generally, simplify things a bit and trust TTLs more. This also makes the "wait" dialog friendlier since we can give users an exact number of seconds.

The overall behavior here is still a little odd because we don't actually require you to respond to the challenge you were issued (right now, we check that the response is valid whenever you submit it, not that it's a valid response to the challenge we issued), but that will change in a future diff. This is just moving us generally in the right direction, and doesn't yet lock everything down properly.

Test Plan:
  - Added a little snippet to the control caption to list all the valid codes to make this easier:

```
    $key = new PhutilOpaqueEnvelope($config->getFactorSecret());
    $valid = array();
    foreach ($this->getAllowedTimesteps() as $step) {
      $valid[] = self::getTOTPCode($key, $step);
    }

    $control->setCaption(
      pht(
        'Valid Codes: '.implode(', ', $valid)));
```

  - Used the most-future code to sign `L3`.
  - Verified that `L4` did not unlock until the code for `L3` left the activation window.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19890
2018-12-20 14:44:07 -08:00
Austin McKinley
2de632d4fe Update continue/break for php 7.3
Summary:
Fixes https://discourse.phabricator-community.org/t/error-on-project-creation-or-edition-with-php7-3/2236

I didn't actually repro this because I don't have php 7.3 installed. I'm also not sure if the `break; break` was intentional or not, since I'm not sure you could ever reach two consecutive break statements.

Test Plan: Created some projects. Didn't actually try to hit the code that fires if you're making a project both a subproject and a milestone.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19925
2018-12-20 14:12:35 -08:00
Austin McKinley
c72d29f401 Cleanup some clustering rough edges
Summary: Suppress an unhelpful Almanac transaction and document the location of the secret clustering management capability. I thought maybe implementing `shouldHide` and checking for `isCreate` would work, but the binding apparently gets created before an interface is bound to it.

Test Plan: Looked at a fresh binding and didn't see "Unknown Object(??)", ran bin/diviner and saw expected output.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19917
2018-12-20 11:19:19 -08:00
Austin McKinley
979187132d Update accountadmin to use new admin empowerment code
Summary: Fixes https://discourse.phabricator-community.org/t/admin-account-creation-fails-call-to-undefined-method-phabricatorusereditor-makeadminuser/2227. This callsite got skipped when updating the EmpowerController to use the new transactional admin approval code.

Test Plan: Invoked `accountadmin` to promote a user, no longer got an exception.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19915
2018-12-19 12:00:53 -08:00
epriestley
aa3b2ec5dc Give Pholio Images an authorPHID and use ExtendedPolicies to implement policy behavior
Summary:
Depends on D19912. Ref T11351. Images currently use `getMock()->getPolicy()` stuff to define policies. This causes bugs with object policies like "Subscribers", since the policy engine tries to evaluate the subscribers //for the image// when the intent is to evaluate the subscribers for the mock.

Move this to ExtendedPolicies to fix the behavior, and give Images sensible policy behavior when they aren't attached to a mock (specifically: only the user who created the image can see it).

Test Plan: Applied migrations, created and edited mocks and images without anything blowing up. Set mock visibility to "Subscribers", everything worked great.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19913
2018-12-19 10:50:52 -08:00
epriestley
c4c5d8a210 Un-implement MarkupInterface from Mocks and Images in Pholio
Summary: Depends on D19911. Ref T11351. `MarkupInterface` has mostly been replaced with `PHUIRemarkupView`, and isn't really doing anything for us here. Get rid of it to simplify the code.

Test Plan: Viewed various mocks with descriptions and image descriptions, saw remarkup presented properly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19912
2018-12-19 10:47:27 -08:00
epriestley
1d84b5b86b Give Pholio images a more modern initializer method
Summary: Depends on D19910. Ref T11351. Minor changes to make this behave in a more modern way.

Test Plan:
  - Destroyed a mock.
  - Lipsum'd a mock.
  - Poked around, edited/viewed mocks.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19911
2018-12-19 10:47:02 -08:00
epriestley
38083f6f9e Slightly modernize PholioImageQuery
Summary:
Ref T11351. My end goal is to remove `applyInitialEffects()` from Pholio to clear the way for D19897.

Start with some query modernization.

Test Plan: Browsed Pholio, nothing appeared to have changed.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19910
2018-12-19 10:46:10 -08:00
Austin McKinley
95ea4f11b9 Fix sorting bug in ProjectDatasource
Summary:
See https://discourse.phabricator-community.org/t/typeahead-returning-only-archived-results/2220. Ref T12538.

If a user has more than 100 disabled projects matching their search term, only disabled projects will be returned in the typeahead search results.

Test Plan: Harcoded hard limit in `PhabricatorTypeaheadModularDatasourceController` to force truncation of search results, observed active project on top of results as expected.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12538

Differential Revision: https://secure.phabricator.com/D19907
2018-12-18 12:17:52 -08:00
epriestley
46052878b1 Bind MFA challenges to particular workflows, like signing a specific Legalpad document
Summary:
Depends on D19888. Ref T13222. When we issue an MFA challenge, prevent the user from responding to it in the context of a different workflow: if you ask for MFA to do something minor (award a token) you can't use the same challenge to do something more serious (launch nukes).

This defuses highly-hypothetical attacks where the attacker:

  - already controls the user's session (since the challenge is already bound to the session); and
  - can observe MFA codes.

One version of this attack is the "spill coffee on the victim when the code is shown on their phone, then grab their phone" attack. This whole vector really strains the bounds of plausibility, but it's easy to lock challenges to a workflow and it's possible that there's some more clever version of the "spill coffee" attack available to more sophisticated social engineers or with future MFA factors which we don't yet support.

The "spill coffee" attack, in detail, is:

  - Go over to the victim's desk.
  - Ask them to do something safe and nonsuspicious that requires MFA (sign `L123 Best Friendship Agreement`).
  - When they unlock their phone, spill coffee all over them.
  - Urge them to go to the bathroom to clean up immediately, leaving their phone and computer in your custody.
  - Type the MFA code shown on the phone into a dangerous MFA prompt (sign `L345 Eternal Declaration of War`).
  - When they return, they may not suspect anything (it would be normal for the MFA token to have expired), or you can spill more coffee on their computer now to destroy it, and blame it on the earlier spill.

Test Plan:
  - Triggered signatures for two different documents.
  - Got prompted in one, got a "wait" in the other.
  - Backed out of the good prompt, returned, still prompted.
  - Answered the good prompt.
  - Waited for the bad prompt to expire.
  - Went through the bad prompt again, got an actual prompt this time.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19889
2018-12-18 12:06:16 -08:00
Austin McKinley
b6999c7ef4 Move admin promotions to modular transactions
Summary: Continue clean up of super-old code. I am pretty proud of "defrocked", but would also consider "dethroned", "ousted", "unseated", "unmade", or "disenfranchised". I feel like there's a word for being kicked out of Hogwarts and having your wizarding powers revoked, but it is not leaping to mind.

Test Plan: Promoted/demoted users to/from admin, attempted to demote myself and observed preserved witty text, checked user timelines, checked feed, checked DB for sanity, including `user_logs`. I didn't test exposing this via Conduit to attempt promoting a user without having admin access.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19891
2018-12-18 10:15:47 -08:00
epriestley
5e94343c7d Add a garbage collector for MFA challenges
Summary:
Depends on D19886. Ref T13222. Clean up MFA challenges after they expire.

(There's maybe some argument to keeping these around for a little while for debugging/forensics, but I suspect it would never actually be valuable and figure we can cross that bridge if we come to it.)

Test Plan:
  - Ran `bin/garbage collect --collector ...` and saw old MFA challenges collected.
  - Triggered a new challenge, GC'd again, saw it survive GC while still active.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19888
2018-12-17 07:00:55 -08:00
epriestley
b8cbfda07c Track MFA "challenges" so we can bind challenges to sessions and support SMS and other push MFA
Summary:
Ref T13222. See PHI873. Ref T9770.

Currently, we support only TOTP MFA. For some MFA (SMS and "push-to-app"-style MFA) we may need to keep track of MFA details (e.g., the code we SMS'd you). There isn't much support for that yet.

We also currently allow free reuse of TOTP responses across sessions and workflows. This hypothetically enables some "spyglass" attacks where you look at someone's phone and type the code in before they do. T9770 discusses this in more detail, but is focused on an attack window starting when the user submits the form. I claim the attack window opens when the TOTP code is shown on their phone, and the window between the code being shown and being submitted is //much// more interesting than the window after it is submitted.

To address both of these cases, start tracking MFA "Challenges". These are basically a record that we asked you to give us MFA credentials.

For TOTP, the challenge binds a particular timestep to a given session, so an attacker can't look at your phone and type the code into their browser before (or after) you do -- they have a different session. For now, this means that codes are reusable in the same session, but that will be refined in the future.

For SMS / push, the "Challenge" would store the code we sent you so we could validate it.

This is mostly a step on the way toward one-shot MFA, ad-hoc MFA in comment action stacks, and figuring out what's going on with Duo.

Test Plan:
  - Passed MFA normally.
  - Passed MFA normally, simultaneously, as two different users.
  - With two different sessions for the same user:
    - Opened MFA in A, opened MFA in B. B got a "wait".
    - Submitted MFA in A.
    - Clicked "Wait" a bunch in B.
    - Submitted MFA in B when prompted.
  - Passed MFA normally, then passed MFA normally again with the same code in the same session. (This change does not prevent code reuse.)

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222, T9770

Differential Revision: https://secure.phabricator.com/D19886
2018-12-17 07:00:21 -08:00
epriestley
c731508d74 Require MFA implementations to return a formal result object when validating factors
Summary:
Ref T13222. See PHI873. Currently, MFA implementations return this weird sort of ad-hoc dictionary from validation, which is later used to render form/control stuff.

I want to make this more formal to handle token reuse / session binding cases, and let MFA factors share more code around challenges. Formalize this into a proper object instead of an ad-hoc bundle of properties.

Test Plan:
  - Answered a TOTP MFA prompt wrong (nothing, bad value).
  - Answered a TOTP MFA prompt properly.
  - Added new TOTP MFA, survived enrollment.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19885
2018-12-17 06:59:46 -08:00
epriestley
54b952df5d Fix weird gap/spacing on user "Manage" page
Summary: I added this recently for debugging test notifications, but goofed up the markup, thought it was just some weird layout issue, and never got back to it.

Test Plan: {F6063455}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19892
2018-12-14 15:40:26 -08:00
Austin McKinley
d23cc4b862 Move user renames to modular transactions
Summary: Cleaning up more super-old code from `PhabricatorUserEditor`. Also fix user logging in approve transactions. I'm not sure how it worked at all previously.

Test Plan: Created new users, renamed them, checked DB for sanity. Entered invalid names, duplicate names, and empty names, got appropriate error messages.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19887
2018-12-13 16:47:54 -08:00
epriestley
080fb1985f Upgrade an old "weakDigest()" inside TOTP synchronization code
Summary:
Ref T13222. Ref T12509. When you add a new MFA TOTP authenticator, we generate a temporary token to make sure you're actually adding the key we generated and not picking your own key.

That is, if we just put inputs in the form like `key=123, response=456`, users could pick their own keys by changing the value of `key` and then generating the correct `response`. That's probably fine, but maybe attackers could somehow force users to pick known keys in combination with other unknown vulnerabilities that might exist in the future. Instead, we generate a random key and keep track of it to make sure nothing funny is afoot.

As an additional barrier, we do the standard "store the digest, not the real key" sort of thing so you can't force a known value even if you can read the database (although this is mostly pointless since you can just read TOTP secrets directly if you can read the database). But it's pretty standard and doesn't hurt anything.

Update this from SHA1 to SHA256. This will break any TOTP factors which someone was in the middle of adding during a Phabricator upgrade, but that seems reasonable. They'll get a sensible failure mode.

Test Plan: Added a new TOTP factor.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T12509

Differential Revision: https://secure.phabricator.com/D19884
2018-12-13 16:16:13 -08:00
epriestley
1d34238dc9 Upgrade sessions digests to HMAC256, retaining compatibility with old digests
Summary:
Ref T13222. Ref T13225. We store a digest of the session key in the session table (not the session key itself) so that users with access to this table can't easily steal sessions by just setting their cookies to values from the table.

Users with access to the database can //probably// do plenty of other bad stuff (e.g., T13134 mentions digesting Conduit tokens) but there's very little cost to storing digests instead of live tokens.

We currently digest session keys with HMAC-SHA1. This is fine, but HMAC-SHA256 is better. Upgrade:

  - Always write new digests.
  - We still match sessions with either digest.
  - When we read a session with an old digest, upgrade it to a new digest.

In a few months we can throw away the old code. When we do, installs that skip upgrades for a long time may suffer a one-time logout, but I'll note this in the changelog.

We could avoid this by storing `hmac256(hmac1(key))` instead and re-hashing in a migration, but I think the cost of a one-time logout for some tiny subset of users is very low, and worth keeping things simpler in the long run.

Test Plan:
  - Hit a page with an old session, got a session upgrade.
  - Reviewed sessions in Settings.
  - Reviewed user logs.
  - Logged out.
  - Logged in.
  - Terminated other sessions individually.
  - Terminated all other sessions.
  - Spot checked session table for general sanity.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13225, T13222

Differential Revision: https://secure.phabricator.com/D19883
2018-12-13 16:15:38 -08:00
epriestley
c58506aeaa Give sessions real PHIDs and slightly modernize session queries
Summary:
Ref T13222. See PHI873. I'm preparing to introduce a new MFA "Challenge" table which stores state about challenges we've issued (to bind challenges to sessions and prevent most challenge reuse).

This table will reference sessions (since each challenge will be bound to a particular session) but sessions currently don't have PHIDs. Give them PHIDs and slightly modernize some related code.

Test Plan:
  - Ran migrations.
  - Verified table got PHIDs.
  - Used `var_dump()` to dump an organic user session.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19881
2018-12-13 16:14:41 -08:00
epriestley
ecae936d97 Fix another qsprintf() straggler in "Has Open Subtasks"
Summary: See <https://discourse.phabricator-community.org/t/error-message-is-not-being-logged-when-unable-to-connect-to-the-database/2201/>.

Test Plan: Queried for "With Open Subtasks" and "With No Open Subtasks".

Reviewers: amckinley, joshuaspence

Reviewed By: joshuaspence

Subscribers: joshuaspence

Differential Revision: https://secure.phabricator.com/D19880
2018-12-13 05:17:02 -08:00
epriestley
02933acbd5 Remove all application callers to "putInSet()"
Summary: Ref T13218. This is the last public-facing API call for `loadRelatives/loadOneRelative`. This just "primed" objects to make the other calls work and had no direct effects.

Test Plan:
- Ran `bin/fact analyze`.
- Used `bin/storage upgrade -f --apply` to apply `20181031.board.01.queryreset.php`, which uses `LiskMigrationIterator`.
- Browsed user list.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim

Maniphest Tasks: T13218

Differential Revision: https://secure.phabricator.com/D19878
2018-12-12 16:41:12 -08:00
epriestley
793f185d29 Remove application callsites to "LiskDAO->loadOneRelative()"
Summary: Ref T13218. This is like `loadOneWhere(...)` but with more dark magic. Get rid of it.

Test Plan:
- Forced `20130219.commitsummarymig.php` to hit this code and ran it with `bin/storage upgrade --force --apply ...`.
- Ran `20130409.commitdrev.php` with `bin/storage upgrade --force --apply ...`.
- Called `user.search` to indirectly get primary email information.
- Did not test Releeph at all.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13218

Differential Revision: https://secure.phabricator.com/D19876
2018-12-12 16:39:44 -08:00
epriestley
5c99163b7c Remove application callers to "LiskDAO->loadRelatives()"
Summary: Ref T13218. See that task for some discussion. `loadRelatives()` is like `loadAllWhere(...)` except that it does enormous amounts of weird magic which we've moved away from.

Test Plan: Did not test whatsoever since these changes are in Releeph.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13218

Differential Revision: https://secure.phabricator.com/D19874
2018-12-12 16:33:39 -08:00
Austin McKinley
aba9945923 Move user approval to modular transactions
Summary: See https://discourse.phabricator-community.org/t/how-to-approve-user-via-conduit-api/2189. This particular use case doesn't seem very compelling, but moving this logic out of `PhabricatorUserEditor` is a win anyway.

Test Plan: Registered a new user, approved/unapproved them conduit, approved from the UI.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19877
2018-12-12 16:12:23 -08:00
Austin McKinley
5cb462d511 Show more of UTC offset when user's TZ is not an integer number of hours offset
Summary: See https://discourse.phabricator-community.org/t/personal-timezone-setting-mismatch-cleared-and-more-specific-cases/1680. The code has always worked correctly, but the resulting timezone mismatch warning messsage wasn't specific enough when the mismatch is by a non-integer number of hours.

Test Plan: Set timezone locally to Asia/Vladivostok and in Phabricator to Australia/Adelaide (which as of today's date are 30 minutes apart) and observed a more precise error message: F6061330

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19873
2018-12-12 14:02:30 -08:00
epriestley
2814d34036 Fix a stray qsprintf() in the Herald rules engine when recording rule application to objects
Summary: Ref T13217. See PHI1006.

Test Plan: Touched an object with associated Herald rules.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19872
2018-12-12 11:31:36 -08:00
epriestley
66ff6d4a2c Fix an issue with creating tasks directly into workboard columns
Summary:
See <https://discourse.phabricator-community.org/t/tasks-created-via-workboard-column-menu-are-moved-to-wrong-column/2200>. The recent `setIsConduitOnly()` / `setIsFormField()` change (in D19842) disrupted creating tasks directly into a column from the workboard UI.

This field //is// a form field, it just doesn't render a visible control.

Test Plan:
  - Created a task directly into a workboard column. Before: column selection ignored. After: appeared in correct column.
  - Used "move on workboard" comment action.
  - Edited tasks; edited forms for tasks. Didn't observe any collateral damage (weird "Column" fields being present).

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19870
2018-12-12 09:21:39 -08:00
epriestley
d8e2bb9f0f Fix some straggling qsprintf() warnings in repository import
Summary:
Ref T13217. See <https://discourse.phabricator-community.org/t/unsafe-raw-string-warnings-while-importing-git-commits/2191>.

Hunt down and fix two more `qsprintf()` things.

I just converted the "performance optimization" into a normal, safe call since we're dealing with far less SVN stuff nowadays and the actual issue has been lost in the mists of time. If it resurfaces, we can take another look.

Test Plan: Imported some commits, no longer saw these warnings in the daemon logs.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19869
2018-12-12 09:21:12 -08:00
epriestley
0e067213fb Make viewing a user's profile page clear notifications about that user
Summary: Ref T13222. See PHI996. This is a general correctness improvement, but also allows you to clear test notifications by clicking on them (since their default destination is the recipient's profile page).

Test Plan: Clicked a test notification, got taken to my profile page, saw notification marked as read.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19867
2018-12-10 16:26:25 -08:00
epriestley
05900a4cc9 Add a CLI workflow for testing that notifications are being delivered
Summary: Depends on D19865. Ref T13222. See PHI996. Provide a `bin/aphlict notify --user ... --message ...` workflow for sending test notifications from the CLI.

Test Plan: {F6058287}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19866
2018-12-10 16:05:53 -08:00
epriestley
e43f9124f8 Remove obsolete "NotifyTest" feed story
Summary: Depends on D19864. Ref T13222. See PHI996. This is no longer used by anything, so get rid of it.

Test Plan: Grepped; viewed a feed with these stories in it to make sure nothing crashed/exploded.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19865
2018-12-10 16:03:42 -08:00
epriestley
773b4eaa9e Separate "feed" and "notifications" better, allow stories to appear in notifications only
Summary:
Depends on D19861. Ref T13222. See PHI996. Fixes T10743. Currently, notifications only work if a story also has a feed rendering.

Separate "visible in feed" and "visible in notifications", and make notifications query only notifications and vice versa.

Then, set the test notification stories to be visible in notifications only, not feed.

This could be refined a bit (there's no way to have the two views render different values today, for example) but since the only actual use case we have right now is test notifications I don't want to go //too// crazy future-proofing it. I could imagine doing some more of this kind of stuff in Conpherence eventually, though, perhaps.

Test Plan: Sent myself test notifications, saw them appear on my profile timeline and in the JS popup, and in my notifications menu, but not in feed.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T10743

Differential Revision: https://secure.phabricator.com/D19864
2018-12-10 16:02:43 -08:00
epriestley
ba83380565 Update the "Notification Test" workflow to use more modern mechanisms
Summary:
Depends on D19860. Ref T13222. Ref T10743. See PHI996.

Long ago, there were different types of feed stories. Over time, there was less and less need for this, and nowadays basically everything is a "transaction" feed story. Each story renders differently, but they're fundamentally all about transactions.

The Notification test controller still uses a custom type of feed story to send notifications. Move away from this, and apply a transaction against the user instead. This has the same ultimate effect, but involves less weird custom code from ages long forgotten.

This doesn't fix the actual problem with these things showing up in feed. Currently, stories always use the same rendering for feed and notifications, and there need to be some additional changes to fix this. So no behavioral change yet, just slightly more reasonable code.

Test Plan: Clicked the button and got some test notifications, with Aphlict running.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T10743

Differential Revision: https://secure.phabricator.com/D19861
2018-12-10 16:02:11 -08:00
epriestley
508df60a62 When users mark their own inline comments as "Done", suppress the timeline/mail stories
Summary:
Depends on D19858. Ref T13222. See PHI995. In D19635 and related revisions, inline behavior changed to allow you to pre-mark your own inlines as done (as a reviewer) and to pre-mark your inlines for you (as an author).

These actions generate low-value stories in the timeline, like "alice marked 3 comments done." when an author adds some notes to their own revision. These aren't helpful and can be a little misleading.

Instead, just don't count it when someone marks their own inlines as "done". If we throw away all the marks after throwing away the self-marks, hide the whole story.

This happens in three cases:

  # You comment on your own revision, and don't uncheck the "Done" checkbox.
  # You comment on someone else's revision, and check the "Done" checkbox before submitting.
  # You leave a not-"Done" inline on your own revision, then "Done" it later.

Cases (1) and (2) seem unambiguously good/clear. Case (3) is a little more questionable, but I think this still isn't very useful for reviewers.

If there's still a clarity issue around case (3), we could change the story text to "alice marked 3 inline comments by other users as done.", but I think this is probably needlessly verbose and that no one will be confused by the behavior as written here.

(Also note that this story is never shown in feed.)

Test Plan: Created and marked a bunch of inlines as "Done" in Differential and Diffusion, as the author and reviewer/auditor. My own marks didn't generate timeline stories; marking others' comments still does.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19859
2018-12-10 15:37:18 -08:00
epriestley
46feccdfcf Share more inline "Done" code between Differential and Diffusion
Summary:
Ref T13222. See PHI995. Before making a change to inline rendering, consolidate this code for generating the "alice added inlines comments." and "alice marked X inlines as done." transactions.

Both Differential and Diffusion have four very similar chunks of code. Merge them into shared methods and reduce code duplication across the methods.

(In the next change, I plan to hide the "done" story when the mark affects your own inline, since users marking their own inlines as "done" is generally not very interesting or useful.)

Test Plan: As author and reviewer/auditor, added inlines in Differential and Diffusion. As author, marked own and others inlines as done and undone. Got sensible transaction rendering and persistence of "Done".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19858
2018-12-10 15:36:52 -08:00
epriestley
68b1dee139 Replace the "Choose Subtype" radio buttons dialog with a simpler "big stuff you click" sort of UI
Summary:
Ref T13222. Fixes T12588. See PHI683. In several cases, we present the user with a choice between multiple major options: Alamnac service types, Drydock blueprint types, Repository VCS types, Herald rule types, etc.

Today, we generally do this with radio buttons and a "Submit" button. This isn't terrible, but often it means users have to click twice (once on the radio; once on submit) when a single click would be sufficient. The radio click target can also be small.

In other cases, we have a container with a link and we'd like to link the entire container: notifications, the `/drydock/` console, etc. We'd like to just link the entire container, but this causes some problems:

  - It's not legal to link block eleements like `<a><div> ... </div></a>` and some browsers actually get upset about it.
  - We can `<a><span> ... </span></a>` instead, then turn the `<span>` into a block element with CSS -- and this sometimes works, but also has some drawbacks:
    - It's not great to do that for screenreaders, since the readable text in the link isn't necessarily very meaningful.
    - We can't have any other links inside the element (e.g., details or documentation).
  - We can `<form><button> ... </button></form>` instead, but this has its own set of problems:
    - You can't right-click to interact with a button in the same way you can with a link.
    - Also not great for screenreaders.

Instead, try adding a `linked-container` behavior which just means "when users click this element, pretend they clicked the first link inside it".

This gives us natural HTML (real, legal HTML with actual `<a>` tags) and good screenreader behavior, but allows the effective link target to be visually larger than just the link.

If no issues crop up with this, I'd plan to eventually use this technique in more places (Repositories, Herald, Almanac, Drydock, Notifications menu, etc).

Test Plan:
{F6053035}

  - Left-clicked and command-left-clicked the new JS fanciness, got sensible behaviors.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222, T12588

Differential Revision: https://secure.phabricator.com/D19855
2018-12-10 14:59:18 -08:00
epriestley
a6632f8c18 Allow "maniphest.subtypes" to configure which options are presented by "Create Subtask"
Summary:
Ref T13222. Ref T12588. See PHI683. After D19853, "Create Subtask" may pop a dialog to let you choose between multiple forms.

Allow users to configure which forms are available by using `maniphest.subtypes` to choose available children for each subtype. Users may either specify particular subtypes or specific forms.

Test Plan: Configured "Quest" tasks to have "Objective" children, got appropriate prompt behavior. Used "subtypes" and "forms" to select forms; used "forms" to reorder forms.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T12588

Differential Revision: https://secure.phabricator.com/D19854
2018-12-10 14:58:28 -08:00
epriestley
d1bcdaeda4 Allow the "Create Subtask" workflow to prompt for a subtype selection, and prepare for customizable options
Summary:
Ref T13222. Ref T12588. See PHI683. Currently, "Create Subtask" always uses the first edit form that the user has access to for the same task subtype. (For example, if you "Create Subtask" from a "Bug", you get the first edit form for "Bugs".)

I didn't want to go too crazy with the initial subtype implementation, but it seems like we're generally on firm ground and it's working fairly well: user requests are for more flexibility in using the system as implemented, not changes to the system or confusion/difficulty with any of the tradeoffs. Thus, I'm generally comfortable continuing to build it out in the same direction. To improve flexibility, I want to make the options from "Create Subtask" more flexible/configurable.

I plan to let you specify that a given subtype (say, "Quest") prompts you with creation options for a set of other subtypes (say, "Objective"), or prompts you with a particular set of forms.

If we end up with a single option, we just go into the current flow (directly to the edit form). If we end up with more than one option, we prompt the user to choose between them.

This change is a first step toward this:

  - When building "Create Subtask", query for multiple forms.
  - The default behavior is now "prompt user to choose among create forms of the same subtype". Previously, it was "use the first edit form of the same subtype". This is a behavioral change.
  - The next change will make the selected forms configurable.
  - (I also plan to make the dialog itself less rough.)

Test Plan: {F6051067}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T12588

Differential Revision: https://secure.phabricator.com/D19853
2018-12-10 14:44:26 -08:00
Austin McKinley
da4341cf8b Make it less confusing to create root-level Phriction doc
Summary: Without an existing root document, Phriction shows a nice little "fake" document as the landing page, which has its own nice "Edit this document" button. When showing that page, don't also render the standard "New Document" breadcrumb in the top right. That button always prompts first for a slug name, which is silly when the root document doesn't exist (because the slug name is required to be '').

Test Plan: Loaded Phriction with and without a root document.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19863
2018-12-10 14:10:18 -08:00
Austin McKinley
00a7071e2d Fix handling of Phriction conduit edits
Summary: See https://discourse.phabricator-community.org/t/conduit-method-phriction-edit-requires-title-while-the-docs-say-its-optional/2176. Make code consistent with documentation by not requiring either `content` or `title`.

Test Plan: Hit the method via the UI and no longer got an error on missing `content` or `title` fields.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19862
2018-12-10 13:38:13 -08:00
epriestley
bf6c534b56 Give "Track Only" repository detail proper getters/setters
Summary: Depends on D19856. Ref T13222. See D19829. Make access to "Track Only" slightly cleaner and more consistent..

Test Plan: Set, edited, and removed "Track Only" settings for a repository. Saw sensible persistence and display behaviors.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19857
2018-12-10 10:22:37 -08:00
epriestley
c3206476a3 Give "Autoclose Only" repository detail proper getters/setters
Summary:
Ref T13222. See D19829. We're inconsistent about using `getDetail()/setDetail()` to do some ad-hoc reads. Put this stuff in proper accessor methods.

Also a couple of text fixes from D19850.

Test Plan: Set, edited, and removed autoclose branches from a repository. Got sensible persistence and rendering behavior.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19856
2018-12-10 10:22:06 -08:00
epriestley
1a6a0181a8 Allow "bin/repository thaw --demote" to demote an entire service, not just a single device
Summary: Ref T13222. See PHI992. If you lose an entire cluster, you may want to aggressively demote it out of existence. You currently need to `xargs` your way through this. Allow `--demote <service>`, which demotes all devices in a service.

Test Plan: Demoted with `--demote <device>` and `--demote <service>`. Hit the `--promote service` error.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19850
2018-12-09 16:46:17 -08:00
epriestley
bba4186005 Allow "bin/repository thaw" to accept "--all-repositories" instead of a list of repositories
Summary:
Ref T13222. See PHI992. If you've lost an entire cluster (or have lost a device and are willing to make broad assumptions about the state the device was in) you currently have to `xargs` to thaw everything or do something else creative.

Since this workflow is broadly reasonable, provide an easier way to accomplish the goal.

Test Plan:
  - Ran with `--all-repositories`, a list of repositories, both (error) and neither (error).
  - Saw a helpful new list of affected repositories.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19849
2018-12-09 16:41:57 -08:00
epriestley
1e4bdc39a1 Add an "availaiblity" attachment for user.search
Summary: Ref T13222. See PHI990. The older `user.query` supports availability information, but it isn't currently available in a modern way. Make it available.

Test Plan: {F6048126}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19851
2018-12-09 16:41:12 -08:00
epriestley
1029081b28 Correct two straggling "%Q" + "implode(...)" callsites in Revision updates
Summary:
See <https://discourse.phabricator-community.org/t/error-seems-to-be-related-with-da40f8074-and-php-7-3/2102/12>. When creating or updating revisions, we do some manual query construction to update the affected path table.

Update these queries to modern `qsprintf()`.

Test Plan: Created and updated revisions affecting paths, no more logs in the webserver log.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19846
2018-12-09 16:39:59 -08:00
epriestley
b88a87c43a Address a transaction issue with some audit actions not applying correctly
Summary:
See <https://discourse.phabricator-community.org/t/cannot-accept-commits-in-audit/2166/>.

In D19842, I changed `PhabricatorEditField->shouldGenerateTransactionsFromComment()`.

  - Previously, it bailed on `getIsConduitOnly()`.
  - After the patch, it bails on a missing `getCommentActionLabel()`.

The old code was actually wrong, and it was previously possible to apply possibly-invalid actions in some cases (or, at least, sneak them through this layer: they would only actually apply if not validated properly).

In practice, it let a different bug through: we sometimes loaded commits without loading their audit authority, so testing whether the viewer could "Accept" the commit or not (or take some other actions like "Raise Concern") would always fail and throw an exception: "Trying to access data not attached to this object..."

Fixing the insufficiently-strict transaction generation code exposed the "authority not attached" bug, which caused some actions to fail to generate transactions.

This appeared in the UI as either an unhelpful error ("You can't post an empty comment") or an action with no effect. The unhelpful error was because we show that error if you aren't taking any //other// actions, and we wouldn't generate an "Accept" action because of the interaction of these bugs, so the code thought you were just posting an empty comment.

Test Plan: Without leaving comments, accepted and rejected commits. No more error messages, and actions took effect.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: stephan.senkbeil, hskiba

Differential Revision: https://secure.phabricator.com/D19845
2018-12-09 16:39:21 -08:00
epriestley
f0eefdd0b5 Replace the informal "array" subtype map with a more formal "SubtypeMap" object
Summary: Ref T13222. Ref T12588. See PHI683. To make "Create Subtask..." fancier, we need slightly more logic around subtype maps. Upgrade the plain old array into a proper object so it can have relevant methods, notably "get a list of valid child subtypes for some parent subtype".

Test Plan: Created and edited tasks, changed task subtypes. Grepped for affected symbols (`newEditEngineSubtypeMap`, `newSubtypeMap`).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T12588

Differential Revision: https://secure.phabricator.com/D19852
2018-12-09 16:37:35 -08:00
epriestley
5d54f26dac Support reading and querying Almanac service PHIDs via "diffusion.repository.search"
Summary:
Ref T13222. See PHI992. If you lose all hosts in a service cluster, you may need to get a list of affected repositories to figure out which backups to pull.

Support doing this via the API.

Test Plan: Queried by service PHID and saw service PHIDs in the call results.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19848
2018-12-06 07:46:36 -08:00
epriestley
70bf63bc3a Fix a transaction editor "continue;" inside "switch()" for PHP 7.3
Summary: See <https://discourse.phabricator-community.org/t/new-git-commit-processing-fails-on-php-7-3/>. This "continue" should be a "break".

Test Plan:
{F6045490}

  - Tried to assign a task to myself while I was already the owner, got an appropriate error.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19844
2018-12-05 11:25:53 -08:00
epriestley
9bfe558587 Add a "touched paths" limit to repositories, limiting the maximum number of paths any commit may touch
Summary:
Depends on D19831. Ref T13216. See PHI908. Allegedly, a user copied a large repository into itself and then pushed it. Great backup strategy, but it can create headaches for administrators.

Allow a "maximum paths you can touch with one commit" limit to be configured, to make it harder for users to make this push this kind of commit by accident.

If you actually intended to do this, you can work around this by breaking your commit into pieces (or temporarily removing the limit). This isn't a security/policy sort of option, it's just a guard against silly mistakes.

Test Plan: Set limit to 2, tried to push 3 files, got rejected. Raised limit, pushed changes successfully.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19839
2018-11-28 14:37:36 -08:00
epriestley
c86c5749ba Make the repository "Filesize Limit" and "Clone/Fetch Timeout" configurable in the UI
Summary: Depends on D19830. Ref T13216. See PHI908. See PHI750. See PHI885. Allow users to configure a filesize limit, and allow them to adjust the clone/fetch timeout.

Test Plan:
{F6021356}

  - Configured a filesize limit and pushed, hit it. Made the limit larger and pushed, change went through.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19831
2018-11-28 14:34:00 -08:00
epriestley
c6fc05ee09 Pull Git filesize logic into a separate LowLevel query and use more Iterators
Summary:
Depends on D19829. Ref T13216. See PHI908. The current implementation is kind of a lot to live in `CommitHookEngine` and will likely fail if `git diff-tree` produces more than 2GB of output.

Pull it out and make it slightly more robust against enormous commits. It's probably limited by this, now:

```
implode("\n", $every_path)
```

We could replace that with some `PhutilReverseRopeSource` primitive or something but since we don't have one of those and it seems unlikely that we'll hit this case in practice, I left it here for now with just the easy stuff converted to be stream-oriented.

Test Plan:
Used this script to test the query against various commits, got good results:

```
<?php

require_once 'scripts/init/init-script.php';

$viewer = PhabricatorUser::getOmnipotentUser();

$repository = id(new PhabricatorRepositoryQuery())
  ->setViewer($viewer)
  ->withCallsigns(array('P'))
  ->executeOne();

var_dump(
  id(new DiffusionLowLevelFilesizeQuery())
    ->setRepository($repository)
    ->withIdentifier($argv[1])
    ->execute());
```

Used this to find large commits in history and pull filesizes (worked great, although our largest commit only touches a couple thousand paths):

```
for hash in `git log --format=%H`; do echo -n $hash; echo -n ' '; git diff-tree -r --no-commit-id $hash | wc -l | awk '{print $1}'; done | awk '{print $2 " " $1}' | sort -n
```

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19830
2018-11-28 14:32:59 -08:00
epriestley
fd12b37d16 Modularize Repository transactions
Summary: Depends on D19828. Ref T13216. Before adding new transactions to repositories (filesize limit, copy time limit, etc) modularize the existing transactions.

Test Plan:

- Created repository.
- Edited callsign (invalid, valid, duplicate, add, remove).
- Edited short name (invaild, valid, duplicate, add, remove).
- Edited description (add, remove).
- Edited encoding (invalid, valid, remove).
- Allowed/denied dangerous changes.
- Allowed/denied enormous chagnes.
- Activated, deactivated, reactivated.
- Changed tags.
- Changed push policy.
- Changed default branch (add, remove).
- Changed track only: add, remove, invalid function, invalid regex.
- Changed autoclose only: add, remove, invalid function, invalid regex.
- Changed publish/notify.
- Changed autoclose.
- Changed staging area (add, remove, invalid).
- Changed blueprints (add, remove).
- Changed symbols (add, remove).
- Grepped for `PhabricatorRepositoryTransaction::TYPE_`.
- Reviewed transaction history:

{F6021036}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19829
2018-11-28 14:29:18 -08:00
epriestley
c25d2a399d Separate the repository management UI into sections
Summary: Depends on D19826. Ref T13216. We have a fair number of options here; add some groups so the "Build" stuff can go in a little subcategory and such.

Test Plan: {F6020896}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19827
2018-11-28 13:53:30 -08:00
epriestley
c457d23a1d Tailor the "no reviewers on this revision" warnings to handle the case where all reviewers have resigned
Summary:
Ref T13216. See PHI985. We currently use a banner to warn you when a revision has no reviewers or only disabled users, but since the changes to track "Resign" more explicilty we'll no longer warn you if everyone has resigned.

(Previously, they'd no longer be reviewers, so you'd end up with the "no reviewers are assigned" warning if everyone resigned.)

This can still interact slightly oddly with some states (e.g., only a package or project reviewer) but I'd like to wait for T731 to tighten those cases up, and they're more advanced/unusual.

Test Plan:
{F6026832}

{F6026833}

{F6026834}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19834
2018-11-28 13:50:29 -08:00
epriestley
01c7be059d Add support for "harbormaster.target.search"
Summary: Ref T13222. See PHI986. See PHI896. Harbormaster build targets don't currently have a modern "*.search" API, but there's no reason not to provide one (even if some of the use cases are a little bit questionable).

Test Plan: {F6032423}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19841
2018-11-28 13:49:27 -08:00
epriestley
2f11001f6e Allow "Change Subtype" to be selected from the comment action stack
Summary:
Ref T13222. See PHI683. Currently, you can "Change subtype..." via Conduit and the bulk editor, but not via the comment action stack or edit forms.

In PHI683 an install is doing this often enough that they'd like it to become a first-class action. I've generally been cautious about pushing this action to become a first-class action (there are some inevitable rough edges and I don't want to add too much complexity if there isn't a use case for it) but since we have evidence that users would find it useful and nothing has exploded yet, I'm comfortable taking another step forward.

Currently, `EditEngine` has this sort of weird `setIsConduitOnly()` method. This actually means more like "this doesn't show up on forms". Make it better align with that. In particular, a "conduit only" field can already show up in the bulk editor, which is goofy. Change this to `setIsFormField()` and convert/simplify existing callsites.

Test Plan:
There are a lot of ways to reach EditEngine so this probably isn't entirely exhaustive, but I think I got pretty much anything which is likely to break:

- Searched for `setIsConduitOnly()` and `getIsConduitOnly()`, converted all callsites to `setIsFormField()`.
- Searched for `setIsLockable()`, `setIsReorderable()` and `setIsDefaultable()` and aligned these calls to intent where applicable.
- Created an Almanac binding.
- Edited an Almanac binding.
- Created an Almanac service.
- Edited an Almanac service.
- Edited a binding property.
- Deleted a binding property.
- Created and edited a badge.
- Awarded and revoked a badge.
- Created and edited an event.
- Made an event recurring.
- Created and edited a Conpherence thread.
- Edited and updated the diff for a revision.
- Created and edited a repository.
- Created and disabled repository URIs.
- Created and edited a blueprint.
- Created and edited tasks.
- Created a paste, edited/archived a paste.
- Created/edited/archived a package.
- Created/edited a project.
- Made comments.
- Moved tasks on workboards via comment action stack.
- Changed task subtype via comment action stack.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19842
2018-11-28 13:40:40 -08:00
epriestley
1d0b99e1f8 Allow applications to require a High Security token without doing a session upgrade
Summary:
Ref T13222. See PHI873. Currently, when applications prompt users to enter MFA, their session upgrades as a side effect.

In some cases (like managing your email addresses) it makes sense to upgrade your session for a little while since it's common to make multiple edits in sequence (add a new address, make it primary, remove an old address). We generally want MFA to stay out of the way and not feel annoying.

In other cases, we don't expect multiple high-security actions in a row. Notably, PHI873 looks at more "one-shot" use cases where a prompt is answering a specific workflow. We already have at least one of these in the upstream: answering an MFA prompt when signing a Legalpad document.

Introduce a "token" workflow (in contrast to the existing "session") workflow that just does a one-shot prompt without upgrading your session statefully. Then, make Legalpad use this new workflow.

Note that this workflow has a significant problem: if the form submission is invalid for some other reason, we re-prompt you on resubmit. In Legalpad, this workflow looks like:

  - Forget to check the "I agree" checkbox.
  - Submit the form.
  - Get prompted for MFA.
  - Answer MFA prompt.
  - Get dumped back to the form with an error.
  - When you fix the error and submit again, you have to do another MFA check.

This isn't a fatal flaw in Legalpad, but would become a problem with wider adoption. I'll work on fixing this (so the MFA token sticks to the form) in the next set of changes.

Roughly, this is headed toward "MFA sticks to the form/workflow" instead of "MFA sticks to the user/session".

Test Plan:
  - Signed a legalpad document with MFA enabled.
  - Was prompted for MFA.
  - Session no longer upgraded (no purple "session in high security" badge).
  - Submitted form with error, answered MFA, fixed error, submitted form again.
    - Bad behavior: got re-prompted for MFA. In the future, MFA should stick to the form.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19843
2018-11-28 13:39:59 -08:00
epriestley
bb369c7b71 Convert the "Repository Management" UI to a full-width, Phortune-style UI
Summary:
Ref T13216. I want to add some new management options to repositories (e.g., filesize limit, clone timeouts). Before adding new stuff here, update the UI to a full-width, Phortune-style UI.

This partially reverts D18523. About a year ago, several UIs got converted to fixed-width (repository management, config, settings, instance management in SAAS). I didn't think these were good changes and have never really gotten used to them. The rationale wasn't clear to me and these changes just felt like "be more like GitHub". I think usability is significantly worse, e.g. actions are now hidden inside button menus instead of immediately visible.

Phortune also got converted less dramatically to a full-width-with-menu UI, which I like much better. Adjust repository management to use that UI style instead of the fixed-width style.

Test Plan:
{F6020884}

Viewed every panel, including the Subversion panel.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19826
2018-11-27 05:06:07 -08:00
epriestley
88189f723f Make a Feed query construction less clever/sneaky for new qsprintf() semantics
Summary:
Ref T13216. Ref T13217. Currently, we build this query in a weird way so we end up with `(1, 2, 3)` on both 32-bit and 64-bit systems.

I can't reproduce the string-vs-int MySQL key issue on any system I have access to, so just simplify this and format as `('1', '2', '3')` instead.

The issue this is working around is that MySQL would (I think?) sometimes appear to do something goofy and miss the key if you formatted the query with strings. I never really nailed this down and could have either been mistaken about it or it could be fixed in all modern versions of MySQL. Until we have better evidence to the contrary, assume MySQL is smart enough to handle this sensibly now.

Test Plan: Ran daemons with Feed publish workers, no longer received query warnings.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217, T13216

Differential Revision: https://secure.phabricator.com/D19837
2018-11-26 10:47:19 -08:00
epriestley
5343b1f898 Remove defunct "metamta.herald.show-hints" Config option
Summary:
Ref T13216. See PHI985. This config option once controlled adding a Herald transcript link to email. However, this was never implemented in a generic way and was removed from revisions in D8459 and from commits in D10705. No one has noticed or asked for this option for several years, so this is probably a good opportunity to simplify the software and reduce the total amount of configuration.

If we did want to pursue this in the future, I'd generally prefer to make it part of the mail detail page (`/mail/detail/12345/`) anyway.

Test Plan: Grepped for `metamta.herald.show-hints` and `addHeraldSection()`, got no hits for either.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19833
2018-11-26 10:14:25 -08:00
epriestley
9473f60a36 Allow "Abandoned" revisions to be commandeered
Summary:
Ref T13216. See PHI985. You currently can't commandeer an abandoned revision, but this workflow is perfectly fine.

The caution here is just around weird use cases where, e.g., users want to reopen a revision to add a revert to it. These workflows tend to create problems so we try to guide users away from them.

Test Plan: {F6026841}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19835
2018-11-26 10:13:52 -08:00
epriestley
bcc90d8c6b Fix an off-by-one error affecting mail rendering of inlines on the final line of a file
Summary: Depends on D19837. Ref T13216. See PHI985. There's an off-by-one error here between how inline comments store "length" and how context rendering treats "length". We need to add 1 to the length, but currently do it a little too early. Do it slightly later so that inlines on the final line of a file render properly.

Test Plan: Left an inline on the final line of a new file, saw it render properly in HTML mail.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19838
2018-11-26 10:12:09 -08:00
epriestley
97e7ef0f01 When the last rejecting reviewer resigns from a revision, return it to "Needs Review"
Summary:
Ref T13216. Fixes T12920. See PHI911. If you reject a revision and then resign from it, it stays in "Needs Revision".

There's some arguable motivation for this, but it's inconsistent with how "Accept" works (if the last accepting reviewer resigns, we kick you out of "Accepted"). Make it consistent.

Test Plan:
  - As the only reviewer: requested changes to a revision, then resigned.
  - Before: revision stays in "Needs Revision".
  - After: revision moves back to "Needs Review".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216, T12920

Differential Revision: https://secure.phabricator.com/D19840
2018-11-26 10:11:41 -08:00
John Linahan
433a7321ff Add harbormaster.buildable.search API Method
Summary:
This revision adds a Conduit search method for buildables. It exposes:
  * `objectPHID`
  * `containerPHID`
  * `buildableStatus`
  * `isManual`

Test Plan:
Use the API Console to run searches. Example:
```
{
  "data": [
    {
      "id": 2,
      "type": "HMBB",
      "phid": "PHID-HMBB-m4k5lodx6naq22576a7d",
      "fields": {
        "objectPHID": "PHID-DIFF-vzvgqqcyscpd7ta4osy2",
        "containerPHID": "PHID-DREV-vsivs5276c7vtgpmssn2",
        "buildableStatus": {
          "value": "passed"
        },
        "isManual": true,
        "dateCreated": 1542407155,
        "dateModified": 1542407156,
        "policy": {
          "view": "users",
          "edit": "users"
        }
      },
      "attachments": {}
    },
    {
      "id": 1,
      "type": "HMBB",
      "phid": "PHID-HMBB-opxfl4auoz3ey5klplrx",
      "fields": {
        "objectPHID": "PHID-DIFF-vzvgqqcyscpd7ta4osy2",
        "containerPHID": null,
        "buildableStatus": {
          "value": "passed"
        },
        "isManual": false,
        "dateCreated": 1542406968,
        "dateModified": 1542406968,
        "policy": {
          "view": "users",
          "edit": "users"
        }
      },
      "attachments": {}
    }
  ],
  "maps": {},
  "query": {
    "queryKey": null
  },
  "cursor": {
    "limit": 100,
    "after": null,
    "before": null,
    "order": null
  }
}
```

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, O14 ATC Monitoring

Differential Revision: https://secure.phabricator.com/D19818
2018-11-26 14:16:57 +00:00
epriestley
03f249baf3 Remove rendering support for very old Repository transactions
Summary:
Depends on D19827. Ref T13221. Ref T13216. To prepare Repositories for a move to ModularTransactions, throw away some very old transaction rendering code.

This will cause these very old transactions (none of which have been written since at least April 2016) to render "epriestley edited this repository." instead of "epriestley changed the SSH login for this repository from X to Y."

These edits were generally obsoleted by repository URIs, Passphrase credentials, and general modernization.

Test Plan: Grepped for all constants, got no hits.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13221, T13216

Differential Revision: https://secure.phabricator.com/D19828
2018-11-24 07:21:20 -08:00
epriestley
45e93c8f1d Expose "identifiers" as a query constraint for Commit search
Summary: Ref T13216. See PHI984. The CommitSearchEngine (and, by extension, `diffusion.commit.search`) currently do not support identifier search, but this is a reasonable capability to provide.

Test Plan:
Testing that a commit exists on `master`:

{F6020742}

Same commit is not on `stable`:

{F6020743}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19825
2018-11-24 07:20:01 -08:00
epriestley
43cf4edfb1 When waiting for long-running Harbormaster futures to resolve, close idle database connections
Summary:
Ref T13216. See PHI916. Harbormaster builds may be long-running, particularly if they effectively wrap `ssh ... ./run-huge-build.sh`. If we spend more than a few seconds waiting for futures to resolve, close idle database connections.

The general goal here is to reduce the held connection load for installs with a very large number of test runners.

Test Plan: Added debugging code to `phlog()` closures, saw connections closed while running builds.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19824
2018-11-21 07:53:40 -08:00
epriestley
a0d4b6da4b Support (but do not actually enable) a maximum file size limit for Git repositories
Summary:
Depends on D19816. Ref T13216. See PHI908. See PHI750. In a few cases, users have pushed multi-gigabyte files full of various things that probably shouldn't be version controlled. This tends to create various headaches.

Add support for limiting the maximum size of any object. Specifically, we:

  - list all the objects each commit touches;
  - check their size after the commit applies;
  - if it's over the limit, reject the commit.

This change doesn't actually hook the limit up (the limit is always "0", i.e. unlimited), and doesn't have Mercurial or SVN support. The actual parser bit would probably be better in some other `Query/Parser` class eventually, too. But it at least roughly works.

Test Plan:
Changed the hard-coded limit to other values, tried to push stuff, got sensible results:

```
$ echo pew >> magic_missile.txt && git commit -am pew && git push
[master 98d07af] pew
 1 file changed, 1 insertion(+)
# Push received by "local.phacility.net", forwarding to cluster host.
# Acquiring write lock for repository "spellbook"...
# Acquired write lock immediately.
# Acquiring read lock for repository "spellbook" on device "local.phacility.net"...
# Acquired read lock immediately.
# Device "local.phacility.net" is already a cluster leader and does not need to be synchronized.
# Ready to receive on cluster host "local.phacility.net".
Counting objects: 49, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (48/48), done.
Writing objects: 100% (49/49), 3.44 KiB | 1.72 MiB/s, done.
Total 49 (delta 30), reused 0 (delta 0)
remote: +---------------------------------------------------------------+
remote: |      * * * PUSH REJECTED BY EVIL DRAGON BUREAUCRATS * * *     |
remote: +---------------------------------------------------------------+
remote:              \
remote:               \                    ^    /^
remote:                \                  / \  // \
remote:                 \   |\___/|      /   \//  .\
remote:                  \  /V  V  \__  /    //  | \ \           *----*
remote:                    /     /  \/_/    //   |  \  \          \   |
remote:                    @___@`    \/_   //    |   \   \         \/\ \
remote:                   0/0/|       \/_ //     |    \    \         \  \
remote:               0/0/0/0/|        \///      |     \     \       |  |
remote:            0/0/0/0/0/_|_ /   (  //       |      \     _\     |  /
remote:         0/0/0/0/0/0/`/,_ _ _/  ) ; -.    |    _ _\.-~       /   /
remote:                     ,-}        _      *-.|.-~-.           .~    ~
remote:   *     \__/         `/\      /                 ~-. _ .-~      /
remote:    \____(Oo)            *.   }            {                   /
remote:    (    (..)           .----~-.\        \-`                 .~
remote:    //___\\  \ DENIED!  ///.----..<        \             _ -~
remote:   //     \\                ///-._ _ _ _ _ _ _{^ - - - - ~
remote:
remote:
remote: OVERSIZED FILE
remote: This repository ("spellbook") is configured with a maximum individual file size limit, but you are pushing a change ("98d07af863e799509e7c3a639404d216f9fc79c7") which causes the size of a file ("magic_missile.txt") to exceed the limit. The commit makes the file 317 bytes long, but the limit for this repository is 1 bytes.
remote:
# Released cluster write lock.
To ssh://local.phacility.com/source/spellbook.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://epriestley@local.phacility.com/source/spellbook.git'
```

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: joshuaspence

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19817
2018-11-20 08:04:17 -08:00
epriestley
ab14f49ef8 On the Diffusion cluster status page, improve device sort order
Summary:
Ref T13216. See PHI943. When you have a large number of cluster bindings for a repository, the UI sorting can be a bit hard to manage.

One install that regularly cycles repository cluster devices had a couple dozen older disabled bindings, with the enabled bindings intermingled.

Sort the UI:

  - enabled devices come first;
  - in each group, sort by name.

Test Plan: Mixed disabled/enabled bindings, loaded {nav Diffusion > Repository > Storage} page with clustering configured. Before: relatively unhelpful sort order. After: more intuitive sort order.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19813
2018-11-20 08:03:31 -08:00
epriestley
4967cd6ab9 Fix some "%Q" behavior in PhortuneMerchantQuery
Summary: Ref T13217. This older query does some manual joins; update it for more modern joins.

Test Plan: Ran `instances/` unit tests and got a clean result, browsed Phortune merchants.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19820
2018-11-20 07:59:57 -08:00
epriestley
9481b9eff1 Allow "Can Configure Application" permissions to be configured
Summary:
Ref T13216. See PHI980. Currently, each application in {nav Applications > X > Configure} has a "Can Configure Application" permission which is hard-coded to "Administrators".

There's no technical reason for this, there just hasn't been a great use case for unlocking it. I think when I originally wrote it our protections against locking yourself out of things weren't that great (i.e., it was easier to set the policy to something that prevented you from editing it after the new policy took effect). Our protections are better now.

The major goal here is to let installs open up Custom Forms for given applications (mostly Maniphest) to more users, but the other options mostly go hand-in-hand with that.

Also, in developer mode, include stack traces for policy exceptions. This makes debugging weird stuff (like the indirect Config application errors here) easier.

Test Plan:
  - Granted "Can Configure Application" for Maniphest to all users.
  - Edited custom forms as a non-administrator.
  - Configured Maniphest as a non-administrator.
  - Installed/uninstalled Maniphest as a non-administrator.
  - Tried to lock myself out (got an error message).

{F6015721}

Reviewers: amckinley, joshuaspence

Reviewed By: joshuaspence

Subscribers: joshuaspence

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19822
2018-11-19 07:25:41 -08:00
epriestley
cb033673b6 Unify intracluster sync and Drydock working copy construction timeouts as a repository "copy time limit"
Summary:
Depends on D19814. Ref T13216. See PHI885. For various eldritch reasons, `git fetch` can hang. Although we'd probably like to fix this with `git fetch --require-sustained-network-transfer-rate=512KB/5s` or similar, that flag doesn't exist and we don't have a reasonable way to build it.

Short of that, move toward formalizing a repository "copy time limit": the longest amount of time anything may spend trying to make a copy of this repository.

This grows out of the existing intracluster sync limit, which is effectively the same thing. Here, apply it to `git clone` and `git fetch` in Drydock working copy construction, too. A future change may make it configurable.

Test Plan:
  - Set the limit to 0.001.
  - Tried to build and lease working copies, got sensible timeout errors (see D19815).

```
<Activation Failed> Lease activation failed: [CommandException] Command killed by timeout after running for more than 0.001 seconds.
COMMAND
ssh '-o' 'LogLevel=quiet' '-o' 'StrictHostKeyChecking=no' '-o' 'UserKnownHostsFile=/dev/null' '-o' 'BatchMode=yes' -l '********' -p '2222' -i '********' '127.0.0.1' -- '(cd '\''/var/drydock/workingcopy-163/repo/spellbook/'\'' && git clean -d --force && git fetch && git reset --hard)'
```

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19816
2018-11-16 13:08:12 -08:00
epriestley
933462b487 Continue cleaning up queries in the wake of changes to "%Q"
Summary: Depends on D19810. Ref T13217. Ref T13216. I mostly used `grep implode | grep OR` and `grep implode | grep AND` to find these -- not totally exhaustive but should be a big chunk of the callsites that are missing `%LO` / `%LA`.

Test Plan:
These are tricky to test exhaustively, but I made an attempt to hit most of them:

- Browsed Almanac interfaces.
- Created/browsed Calendar events.
- Enabled/disabled/showed the lock log.
- Browsed repositories.
- Loaded Facts UI.
- Poked at Multimeter.
- Used typeahead for users and projects.
- Browsed Phriction.
- Ran various fulltext searches.

Not sure these are reachable:

- All the lint stuff might be dead/unreachable/nonfunctional?

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim

Maniphest Tasks: T13217, T13216

Differential Revision: https://secure.phabricator.com/D19814
2018-11-16 12:49:44 -08:00
epriestley
49483bdb48 Use "%P" to protect session key hashes in SessionEngine queries from DarkConsole
Summary:
Ref T6960. Ref T13217. Ref T13216. Depends on D19811. Use the recently-introduced "%P" conversion ("Password/Secret") to load sessions in SessionEngine.

This secret isn't critical to protect (it's the //hash// of the actual secret and not useful to attackers on its own) but it shows up on every page in DarkConsole and is an obvious case where `%P` is a more appropriate conversion.

Test Plan:
Note "*********" in the middle of the output here, instead of a session key hash:

{F6012805}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217, T13216, T6960

Differential Revision: https://secure.phabricator.com/D19812
2018-11-16 12:36:35 -08:00
epriestley
b2e91d2205 Move the "container updated" message for Buildables that build Diffs outside of the transaction
Summary:
Ref T13216. See PHI970. Ref T13054. See some discussion in T13216.

When a Harbormaster Buildable object is first created for a Diff, it has no `containerPHID` since the revision has not yet been created.

We later (after creating a revision) send the Buildable a message telling it that we've added a container and it should re-link the container object.

Currently, we send this message in `applyExternalEffects()`, which runs inside the Differential transaction. If Harbormaster races quickly enough, it can read the `Diff` object before the transaction commits, and not see the container update.

Add a `didCommitTransaction()` callback after the transactions commit, then move the message code there instead.

Test Plan:
  - See T13216 for substantial evidence that this change is on the right track.
  - Before change: added `sleep(15)`, reproduced the issue reliably.
  - After change: unable to reproduce issue even with `sleep(15)` (the `containerPHID` always populates correctly).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216, T13054

Differential Revision: https://secure.phabricator.com/D19807
2018-11-16 12:34:06 -08:00
epriestley
44c32839a6 When you "Request Review" of a draft revision, change the button text from "Submit Quietly" to "Publish Revision"
Summary:
See PHI975. Ref T13216. Ref T2543. Previously, see D19204 and PHI433.

When you're acting on a draft revision, we change the button text to "Submit Quietly" as a hint that your actions don't generate notifications yet.

However, this isn't accurate when one of your actions is "Request Review", which causes the revision to publish.

Allow actions to override the submit button text, and make the "Request Review" action change the button text to "Publish Revision".

The alternative change I considered was to remove the word "Quietly" in all cases.

I'm not //thrilled// about how complex this change is to adjust one word, but the various pieces are all fairly clean individually. I'm not sure we'll ever be able to use it for anything else, but I do suspect that the word "Quietly" was the change in D19204 with the largest effect by far (see T10000).

Test Plan:
  - Created a draft revision. Saw "Submit Quietly" text.
  - Added a "Request Review" action, saw it change to "Publish Revision".
  - Reloaded page, saw stack saved and "Publish Revision".
  - Removed action, saw "Submit Quietly".
  - Repeated on a non-draft revision, button stayed put as "Submit".
  - Submitted the various actions, saw them have the desired effects.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216, T2543

Differential Revision: https://secure.phabricator.com/D19810
2018-11-15 20:50:21 -08:00
epriestley
533e4e13b3 Add a bin/herald test ... for doing test runs via the CLI
Summary: Ref T13216. See D19666. It's currently tricky to profile Herald test runs since you have to submit a form and repeating them is a bit of a mess. Provide a simple CLI wrapper so we can use `--xprofile`. This is also maybe nice-to-have if we're ever debugging anything here.

Test Plan: Ran `bin/herald test --object ... --type ...` and got a sensible looking transcript in the UI.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19806
2018-11-15 15:48:52 -08:00
epriestley
8a8123c9db Replace the primary "Disabled/Enabled" Herald Rule filter with "Active/Inactive", considering author status
Summary:
Ref T13216. See PHI947. In Herald, Personal rules do not run if their author's account is disabled.

This isn't communicated very clearly in the UI, and the way the SearchEngine/Query are set up isn't great.

Define "active" as "rule will actually run", which specifically means "rule is enabled, and has a valid (non-disabled) author if it needs one".

Change the meaning of the "Active" default filter from "rule is enabled" to "rule is enabled, and has a valid author if it needs one".

Refine the status badge on the view controller to show this "invalid author" state.

Tweak the language for "Disable/Enable" to be more consistent -- we currently call it "disabled" in some cases and "archived" in others.

Test Plan:
  - Disabled a user account and saw their personal rules behave properly with the new filters/options/view controller.
  - Disabled/enabled a rule, saw consistent text.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19805
2018-11-15 15:47:35 -08:00
epriestley
bbd292b9b3 Modernize the Herald rule search engine
Summary: Ref T13216. Update the Herald Rule SearchEngine and Query to use a more modern style.

Test Plan: Ran various rule queries in the UI, got sensible results

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19803
2018-11-15 15:37:00 -08:00
epriestley
e57bfbf421 Pull some debugging code back out of "master"
See D19778. This is a workaround for T13179 that landed by mistake.
2018-11-15 08:19:29 -08:00
epriestley
86fd204148 Fix all query warnings in "arc unit --everything"
Summary:
Ref T13216. Ref T13217. Depends on D19800. This fixes all of the remaining query warnings that pop up when you run "arc unit --everything".

There's likely still quite a bit of stuff lurking around, but hopefully this covers a big set of the most common queries.

Test Plan: Ran `arc unit --everything`. Before change: lots of query warnings. After change: no query warnings.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217, T13216

Differential Revision: https://secure.phabricator.com/D19801
2018-11-15 03:51:25 -08:00
epriestley
2f10d4adeb Continue making application fixes to Phabricator for changes to %Q semantics
Summary: Depends on D19789. Ref T13217. Continue updating things to use the new %Q-flavored conversions instead of smushing a bunch of strings together.

Test Plan: Browsed around, far fewer errors. These changes are largely mechanical in nature.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19790
2018-11-15 03:50:02 -08:00
epriestley
98690ee326 Update many Phabricator queries for new %Q query semantics
Summary: Depends on D19785. Ref T13217. This converts many of the most common clause construction pathways to the new %Q / %LQ / %LO / %LA / %LJ semantics.

Test Plan: Browsed around a bunch, saw fewer warnings and no obvious behavioral errors. The transformations here are generally mechanical (although I did them by hand).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: hach-que

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19789
2018-11-15 03:48:10 -08:00
epriestley
64b52b9952 Make SELECT construction in PolicyAwareQuery safer
Summary: Depends on D19784. Ref T13217. Reduce uses of unsafe `%Q` in SELECT construction.

Test Plan: This reduces the number of safety warnings when loading Phabricator home from ~900 to ~800.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19785
2018-11-14 15:32:09 -08:00
epriestley
e26c4bddab Replace magical "branch" behavior in "diffusion.branchquery" with an explicit "patterns"
Summary:
See PHI958. Ref T13210. Previously, see PHI720.

The use case for the magic in PHI720 involves multiple patterns, and no parameter can be passed to `branch` that will result in multiple patterns being passed to `git`.

Replace the implicit magic with an explicit `patterns` parameter.

This whole thing is a bit shaky but probably isn't hurting anything.

Test Plan:
  - Ran query with no `patterns`.
  - Ran query with invalid `patterns`, got readable error.
  - Ran query with various valid `patterns` (plain branch name, globs with "?" and "*"), got sensible results.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19771
2018-11-14 14:50:53 -08:00
epriestley
da40f80741 Update PhabricatorLiskDAO::chunkSQL() for new %Q semantics
Summary:
Ref T13217. This method is slightly tricky:

  - We can't safely return a string: return an array instead.
  - It no longer makes sense to accept glue. All callers use `', '` as glue anyway, so hard-code that.

Then convert all callsites.

Test Plan: Browsed around, saw fewer "unsafe" errors in error log.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19784
2018-11-13 08:59:18 -08:00
epriestley
315d857a8a Add a basic web UI for intracluster sync logs
Summary: Depends on D19798. Ref T13216. This puts at least a basic UI on top of sync logs.

Test Plan:
Viewed logs from the web UI and exported data. Note that these syncs are somewhat simulated since I my local cluster is somewhat-faked (i.e., not actually multiple machines).

{F5995899}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19799
2018-11-10 04:58:36 -08:00
epriestley
1d7c960531 Put push log "hookWait" to data export and add all wait values to UI
Summary:
Depends on D19797. Ref T13216.

  - Put the new `hookWait` in the export and the UI.
  - Put the existing waits in the UI, not just the export.
  - Make order consistent: host, write, read, hook (this is the order the timers start in).

Test Plan: Pushed some stuff, viewed web UI and saw sensible numbers, exported data and got the same values.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19798
2018-11-10 04:47:38 -08:00
epriestley
2a7ac8e388 Make "bin/repository thaw" workflow more clear when devices are disabled
Summary:
Ref T13216. See PHI943. If autoscale lightning strikes all your servers at once and destroys them, the path to recovery can be unclear. You're "supposed" to:

  - demote all the devices;
  - disable the bindings;
  - bind the new servers;
  - put whatever working copies you can scrape up back on disk;
  - promote one of the new servers.

However, the documentation is a bit misleading (it was sort of written with "you lost one or two devices" in mind, not "you lost every device") and demote-before-disable is unnecessary and slightly risky if servers come back online. There's also a missing guardrail before the promote step which lets you accidentally skip the demotion step and end up in a confusing state. Instead:

  - Add a guard rail: when you try to promote a new server, warn if inactive devices still have versions and tell the user to demote them.
  - Allow demotion of inactive devices: the order "disable, demote" is safer and more intuitive than "demote, disable" and there's no reason to require the unintuitive order.
  - Make the "cluster already has leaders" message more clear.
  - Make the documentation more clear.

Test Plan:
  - Bound a repository to two devices.
  - Wrote to A to make it a leader, then disabled it (simulating a lightning strike).
  - Tried to promote B. Got a new, useful error ("demote A first").
  - Demoted A (before: error about demoting inactive devices; now: works fine).
  - Promoted B. This worked.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19793
2018-11-10 04:46:46 -08:00
epriestley
c32fa06266 Use phutil_microseconds_since(...) to simplify some timing arithmetic
Summary: Depends on D19796. Simplify some timing code by using phutil_microseconds_since() instead of duplicate casting and arithmetic.

Test Plan: Grepped for `1000000` to find these. Pulled, pushed, made a conduit call. This isn't exhaustive but it should be hard for these to break in a bad way since they're all just diagnostic.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19797
2018-11-08 16:46:32 -08:00
epriestley
b12e92e6e2 Add timing information for commit hooks to push logs
Summary:
Depends on D19779. Ref T13216. The push logs currently record the "hostWait", which is roughly "locking + subprocess cost". We also record locking separately, so we can figure out "subprocess cost" alone by subtracting the lock costs.

However, the subprocess (normally `git receive-pack`) runs hooks, and we don't have an easy way to figure out how much time was spent doing actual `git` stuff vs spent doing commit hook processing. This would have been useful in diagnosing at least one recent issue.

Track at least a rough hook cost and record it in the push logs.

Test Plan: Pushed to a repository, saw a reasonable hook cost appear in the database table.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19780
2018-11-08 06:00:26 -08:00
epriestley
966db4d38e Add an intracluster synchronization log for cluster repositories
Summary:
Depends on D19778. Ref T13216. See PHI943, PHI889, et al.

We currently have a push log and a pull log, but do not separately log intracluster synchronization events. We've encountered several specific cases where having this kind of log would be helpful:

  - In PHI943, an install was accidentally aborting locks early. Having timing information in the sync log would let us identify this more quickly.
  - In PHI889, an install hit an issue with `MaxStartups` configuration in `sshd`. A log would let us identify when this is an issue.
  - In PHI889, I floated a "push the linux kernel + fetch timeout" theory. A sync log would let us see sync/fetch timeouts to confirm this being a problem in practice.
  - A sync log will help us assess, develop, test, and monitor intracluster routing sync changes (likely those in T13211) in the future.

Some of these events are present in the pull log already, but only if they make it as far as running a `git upload-pack` subprocess (not the case with `MaxStartups` problems) -- and they can't record end-to-end timing.

No UI yet, I'll add that in a future change.

Test Plan:
  - Forced all operations to synchronize by adding `|| true` to the version check.
  - Pulled, got a sync log in the database.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19779
2018-11-07 18:24:20 -08:00
epriestley
e09d29fb1a Clean up the workflow for some post-push logging code
Summary:
Ref T13216. When a repository is clustered, we run this cleanup code (to tell the repository to update, and log some timing information) on both nodes. Currently, we do slightly too much work, which is unnecessary and can be a bit confusing to human readers.

The double update message doesn't hurt anything, but there's no reason to write it twice.

Likewise, the second timing information update query doesn't do anything: there's no PushEvent object with the right identifier, so it just updates nothing. We don't need to run it, and it's confusing that we do.

Instead, only do these writes if we're actually the final node with the repository on it.

Test Plan: Added some logging, saw double writes/updates before the change and no doubles afterwards, with no other behavioral changes.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19778
2018-11-07 17:46:50 -08:00
epriestley
8a4bf38655 Use 160-bit TOTP keys rather than 80-bit TOTP keys
Summary:
See <https://hackerone.com/reports/435648>. We currently use 80-bit TOTP keys. The RFC suggests 128 as a minimum and recommends 160.

The math suggests that doing the hashing for an 80-bit key is hard (slightly beyond the reach of a highly motivated state actor, today) but there's no reason not to use 160 bits instead to put this completely out of reach.

See some additional discussion on the HackerOne report about enormous key sizes, number of required observations, etc.

Test Plan: Added a new 160-bit TOTP factor to Google Authenticator without issue.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19792
2018-11-07 15:44:02 -08:00
epriestley
1f6a4cfffe Prevent users from selecting excessively bad passwords based on their username or email address
Summary:
Ref T13216. We occasionally receive HackerOne reports concerned that you can select your username as a password. I suspect very few users actually do this and that this is mostly a compliance/checklist sort of issue, not a real security issue.

On this install, we have about 41,000 user accounts. Of these, 100 have their username as a password (account or VCS). A substantial subset of these are either explicitly intentional ("demo", "bugmenot") or obvious test accounts ("test" in name, or name is a nonsensical string of gibberish, or looks like "tryphab" or similar) or just a bunch of numbers (?), or clearly a "researcher" doing this on purpose (e.g., name includes "pentest" or "xss" or similar).

So I'm not sure real users are actually very inclined to do this, and we can't really ever stop them from picking awful passwords anyway. But we //can// stop researchers from reporting that this is an issue.

Don't allow users to select passwords which contain words in a blocklist: their username, real name, email addresses, or the install's domain name. These words also aren't allowed to contain the password (that is, neither your password nor your username may be a substring of the other one). We also do a little normalization to try to split apart email addresses, domains, and real names, so I can't have "evan1234" as my password.

Test Plan:
  - Added unit tests and made them pass.
  - Tried to set my password to a bunch of variations of my username / email / domain name / real name / etc, got rejected.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19776
2018-11-06 12:44:07 -08:00
epriestley
c206b066df When {meme ...} embed has no text, just use the raw file data unmodified
Summary:
Ref T13216. See PHI948. When you use the remarkup hint button to embed a meme with no text, you get `{meme src=X}`.

If the source is a GIF, we currently split the source apart into frame-by-frame images, process them, and stitch them back together. The end result is the same image we started with, but this process can be slow/expensive, and may timeout for sufficiently large GIFs.

Instead: when there's no text, just return the original image data.

Test Plan:
  - Used `{meme src=X}` with no text, got an image faster.
  - Used `{meme src=X, above=...}` to add text, got an attempt to add text (which didn't get very far locally since I don't have GD configured).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19777
2018-11-06 09:40:22 -08:00
epriestley
d38e768ed8 Prevent users from voting for invalid Slowvote options
Summary:
Depends on D19773. See <https://hackerone.com/reports/434116>. You can currently vote for invalid options by submitting, e.g., `vote[]=12345`.

By doing this, you can see the responses, which is sort of theoretically a security problem? This is definitely a bug, regardless.

Instead, only allow users to vote for options which are actually part of the poll.

Test Plan:
  - Tried to vote for invalid options by editing the form to `vote[]=12345` (got error).
  - Tried to vote for invalid options by editing the radio buttons on a plurality poll into checkboxes, checking multiple boxes, and submitting (got error).
  - Voted in approval and plurality polls the right way, from the main web UI and from the embed (`{V...}`) UI.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19774
2018-11-06 09:21:18 -08:00
epriestley
5e1d94f336 Remove nonfunctional AJAX embed behavior for Slowvote
Summary:
See <https://hackerone.com/reports/434116>. Slowvote has a piece of Javascript that attempts to let you vote on `{V123}` polls inline.

It does not work: nothing ever triggers it (nothing renders a control with a `slowvote-option` sigil).

At least for now, just remove it. It has a completely separate pathway in the controller and both pathways are buggy, so this makes fixing them easier.

Test Plan: Voted in plurality and approval polls via Slowvote and the embedded widget.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19773
2018-11-06 09:20:07 -08:00
epriestley
d2316e8025 Fix an errant "switch ... continue"
Summary:
See <https://discourse.phabricator-community.org/t/unhandled-exception-on-create-task/2062>.

This construction has the same behavior as "switch ... break" but is unconventional. PHP 7.3 started warning about it because it's likely a mistake.

Test Plan: Created a task, edited a task owner. The new code is functionally identical to the old code.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19772
2018-11-05 10:26:27 -08:00
epriestley
24a061f844 Correct an ambiguous regexp in DiffusionRequest
Summary:
See <https://discourse.phabricator-community.org/t/diffusionrequest-regex-error/2057/>.

The intent of `[\d-,]` is "digits, hyphen, and comma" but `[x-y]` means "character range x-y".

Specify `[\d,-]` instead to disambiguate the hyphen as "literal hyphen", not a character range marker.

Test Plan: I can't reproduce the original error as reported, but browsed around Diffusion for a bit.

Reviewers: amckinley, avivey

Reviewed By: avivey

Differential Revision: https://secure.phabricator.com/D19770
2018-11-01 20:01:39 -07:00
Tim Hirsh
9bea00c159 Add harbormaster.buildplan.search api method
Summary: This revision adds a conduit search method for build plans.  Other api methods (eg: `harbormaster.build.search`) support build plan phid's as a constraint, but they weren't exposed anywhere, so this provides a way to fetch them.

Test Plan:
Used the api console to run some searches.  Output:
```
{
  "data": [
    {
      "id": 1,
      "type": "HMCP",
      "phid": "PHID-HMCP-q2c25wvegzdkxs7gzor6",
      "fields": {
        "name": "my build plan",
        "planStatus": "active",
        "dateCreated": 1538085249,
        "dateModified": 1538085249,
        "policy": {
          "view": "users",
          "edit": "admin"
        }
      },
    {
      "id": 1,
      "type": "HMCP",
      "phid": "PHID-HMCP-q2c25wvegzdkxs7gzor6",
      "fields": {
        "name": "my build plan",
        "status": {
          "value": "active"
        },
        "dateCreated": 1538085249,
        "dateModified": 1538085249,
        "policy": {
          "view": "users",
          "edit": "admin"
        }
      },
      "attachments": {}
    },
    ...
  ],
  "maps": {},
  "query": {
    "queryKey": null
  },
  "cursor": {
    "limit": 100,
    "after": null,
    "before": null,
    "order": null
  }
}
```

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, yelirekim

Differential Revision: https://secure.phabricator.com/D19769
2018-11-02 02:57:38 +00:00
epriestley
5d4970d6b2 Fix a bug where "View as Query" could replace a saved query row by ID, causing workboard 404s
Summary:
Fixes T13208. See that task for details.

The `clone $query` line is safe if `$query` is a builtin query (like "open").

However, if it's a saved query we clone not only the query parameters but the ID, too. Then when we `save()` the query later, we overwrite the original query.

So this would happen in the database. First, you run a query and save it as the workboard default (query key "abc123"):

| 123 | abc123 | {"...xxx..."} |

Then we `clone` it and change the parameters, and `save()` it. But that causes an `UPDATE ... WHERE id = 123` and the table now looks like this:

| 123 | def456 | {"...yyy..."} |

What we want is to create a new query instead, with an `INSERT ...`:

| 123 | abc123 | {"...xxx..."} |
| 124 | def456 | {"...yyy..."} |

Test Plan:
  - Followed reproduction steps from above.
    - With just the new `save()` guard, hit the guard error.
    - With the `newCopy()`, got a new copy of the query and "View as Query" remained functional without overwriting the original query row.
  - Ran migration, saw an affected board get fixed.

Reviewers: amckinley, joshuaspence

Reviewed By: joshuaspence

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13208

Differential Revision: https://secure.phabricator.com/D19768
2018-11-01 05:44:49 -07:00
epriestley
b950f877c5 Allow Drydock Blueprints to control "supplemental allocation" behavior so all hosts in an Almanac pool get used
Summary:
Fixes T12145. Ref T13210. See PHI570. See PHI536.

Currently, when you give Drydock an Almanac host pool with more than one host, it never voluntarily builds a second host resource: there is no way to say "maximum X working copies per host" (only "maximum X global working copies") to make the first host overflow, and the allocator tries to pack resources as tightly as possible.

If you can force it to allocate the 2nd..Nth host, things will work reasonably well from there (it will spread working copies across the hosts randomly), but tricking it is very hard, especially before D19761.

To deal with this, give blueprints a new behavior around "supplemental allocations". The idea here is that a blueprint may decide that it would prefer to allocate a fresh new resource instead of allowing an otherwise valid acquisition to occur.

These supplemental allocations follow all the normal allocation rules (they can't exceed limits or actually replace existing resources), so they can only happen if there's free space in the resource pool. But a blueprint can elect for a supplemental allocation to provide a "grow the pool" hint.

The only useful policies here are probably "true" (immediately use all resources, like Almanac) or "false" (pack resources as efficiently as possible) but some other policies //might// be useful (perhaps "start growing the pool when we're getting a bit full even if we aren't at the limit yet, since our workload is bursty").

Then, give Almanac host resources a "true" policy (always allocate supplemental resources) so they use all hosts once a similar number of concurrent jobs arrive.

One aspect of this approach is that we only do supplemental resources if the normal allocation algorithm already decided that the best resource to acquire was part of the same blueprint. I started with an approach like "look at all the blueprints and see if any of them want to be greedy", but then a not-very-desirable blueprint would end up filling up its whole pool before we skipped the supplemental allocation part and ended up picking a different resource. That felt a bit silly and this feels a little cleaner and more focused.

Test Plan:
  - Without changing the Almanac blueprint policy, allocated hosts. Got A, A, A, A, ... (second host never used).
  - Changed the Almanac policy.
  - Allocated hosts, got A, B, random mix of A and B.
  - Destroyed B. Destroyed all leases on A. Allocated. Got A. This tests the "don't build a supplemental resource if there are no leases on the natural resource".

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210, T12145

Differential Revision: https://secure.phabricator.com/D19762
2018-10-31 18:06:47 -07:00
epriestley
57b4b59819 When a Drydock host based on an Almanac blueprint has its binding disabled, stop handing out leases
Summary:
Ref T13210. Ref T12145. The "Almanac Host" blueprint currently hands out new leases on a given host even if the binding has been disabled.

Although there are some more complicated cases here (e.g., involving cleanup of the existing resource and existing leases), this one seems clear cut: if the binding has been disabled, we should stop handing out new leases on it.

Test Plan:
  - Created a service with two hosts.
  - Requested a lease, got host A.
  - Requested more leases, always got host A (we never build a new host when we don't have to, and we currently never have to).
  - Disabled the binding to host A.
  - Requested a lease.
    - Before patch: got host A.
    - After patch: got host B.
  - Also disabled the other binding to host B, requested a lease, got an indefinite wait for resources (which is expected and reasonable).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210, T12145

Differential Revision: https://secure.phabricator.com/D19761
2018-10-27 07:20:30 -07:00
epriestley
65e953658a Expose Audit actions for "transaction.search" in a basic way
Summary: Ref T13210. See PHI841. This mirrors D19509 for Differential.

Test Plan: Called `transaction.search` on a commit with a bunch of audit activity, got appropriate labels in the results.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19760
2018-10-27 07:19:50 -07:00
epriestley
61ec434208 Remove unicode marks for "Accept/Raise Concern" in Audit
Summary:
Ref T13210. The comment action dropdown for audits has a heavy checkmark next to "Accept" and a heavy "X" next to "Raise Concern".

We previously removed similar marks in Differential in D19405 and that seems to have gone fine. For consistency, remove these too.

Test Plan: Viewed the comment action dropdown, no longer saw checkmark and X-mark.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19759
2018-10-27 07:19:18 -07:00
epriestley
a7c008708d Correct a mangled translation string in "bin/phd log --id X"
Summary:
Ref T13210. See PHI930. This translation is wrong: the parameter is a comma-separated list as a string, but the USEnglish translation provides alternatives. We can't select among alternatives based on a random string (it isn't a plurality value to let us select "chair" vs "chairs", and isn't a gender value to let us select "his profile" vs "her profile") so we get an error.

But the string itself is also misleading, since "bin/phd log --id A --id B --id C" will say "none of these are valid" if //any// of them are invalid.

Instead, just tell the user explicitly about the first problem.

Test Plan:
  - Ran `bin/phd log --id` with good (got logs) and bad IDs (got sensible error).
  - Ran `bin/phd log` with any logs (got logs) and (simluated) without any logs (got error).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19755
2018-10-26 06:13:18 -07:00
Mike Riley
5f3a7cb41b Expose Drydock leases via Conduit
Summary:
See T13212 for some context and discussion on this being revived.
See T11694 for original context.

Add a query constraint for lease owners and implement the conduit search method for Drydock leases.

Ref T11694. Fixes T13212.

Test Plan:
- Called the API method from conduit and browsed lease queries from the UI.
- Used the new "ownerPHIDs" constraint via API console.

{F5963044}

Reviewers: yelirekim, amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam, epriestley

Maniphest Tasks: T11694, T13212

Differential Revision: https://secure.phabricator.com/D16594
2018-10-26 06:12:38 -07:00
epriestley
f6122547d7 When a lease triggers a resource allocation for a resource which must activate, awaken the lease task after the resource activates
Summary:
Depends on D19753. Ref T13210. This is a small optimization that saves us from waiting up to 15 seconds for a yield.

When there are no Working Copy resources and a new lease comes in, we'll allocate one and yield until it activates.

If activating it (SSH'ing and running `git clone`) takes less than 15 seconds, the resource will activate (say, at T+4) but the lease won't update again for a while (say, until T+15). This leaves us with a pointless wait (in this example, we're sitting around for 9 seconds when we could move forward).

To improve this a little bit, let resources wake up the lease update tasks that triggered allocation after they activate. In the best case, that task runs ~15 seconds sooner. In the worst case, the awaken is just a no-op.

With a more-full queue, this has a smaller effect (it's likely something else will run and be able to use the resource in those 9 seconds).

With already-activated resources, this has no effect (when resources are already activated, we can lease immediately).

Test Plan:
  - Cleaned up all working copy resources.
  - Requested a new "A" working copy.
  - Before patch: got a working copy after 17-18 seconds, most of which was spent yielded.
  - After patch: got a working copy after 3-4 seconds.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19754
2018-10-26 06:11:43 -07:00
epriestley
78ab675bd8 After a Drydock lease triggers a resource to be reclaimed, stop it from triggering another reclaim until the first one completes
Summary:
Depends on D19752. Ref T13210. If resources take a long time to reclaim/destroy (normally, more than 15 seconds) a single new lease may update several times during the reclaim/destroy process and end up reclaiming multiple resources.

Instead: after a lease triggers a reclaim, prevent it from triggering another reclaim as long as the resource it is reclaiming hasn't finished its reclaim/destroy cycle. Basically, each lease only gets to destroy one resource at a time.

Test Plan:
  - Added a `sleep(120)` to `destroyResource()` to simulate a long reclaim/destroy cycle.
  - Allocated A, A, A working copies. Leased a B working copy.
  - Before patch: saw "B" lease destroy all three "A" working copies after ~0, ~15, and ~30 seconds, then build a new "B" resource after ~120 seconds (when the first reclaim/destroy finished).
  - After patch: saw "B" lease destroy one "A" working copy after ~0 seconds, then wait patiently until it finished up, then build a new "B" resource.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19753
2018-10-26 06:11:05 -07:00
epriestley
e9309fdd6a When a Drydock lease schedules a resource to be reclaimed, awaken the lease update task when the reclaim completes
Summary:
Depends on D19751. Ref T13210. When Drydock needs to reclaim an existing unused resource in order to build a new resource to satisfy a lease, the lease which triggered the reclaim currently gets thrown back into the pool with a 15-second yield.

If the queue is pretty empty and the reclaim is quick, this can mean that we spend up to 15 extra seconds just waiting for the lease update task to get another shot at building a resource (the resource reclaim may complete in a second or two, but nothing else happens until the yield ends).

Instead, when a lease triggers a reclaim, have the reclaim reawaken the lease task when it completes. In the best case, this saves us 15 seconds of waiting. In other cases (the task already completed some other way, the resource gets claimed before the lease gets to it), it's harmless.

Test Plan:
  - Allocated A, A, A working copies with limit 3. Leased a B working copy.
  - Before patch: allocation took ~32 seconds.
  - After patch: allocation takes ~17 seconds (i.e., about 15 seconds less).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19752
2018-10-26 06:09:52 -07:00
epriestley
1f6869a765 In "bin/drydock lease", take a JSON "--attributes" so we can accept complex values
Summary:
Depends on D19750. See T13210. The `bin/drydock lease` command makes it easier to request ad-hoc leases, but currently takes lease attributes in the form `--attributes x=y,a=b`.

This was okay for all leases at the time, but doesn't really work for modern WorkingCopy resources since they take a `repositories.map` which has a dictionary as a value. You can't specify that with `repositories.map=...`.

Instead, point `--attributes` at a JSON file or use `--attributes -` to read from stdin.

Test Plan: Used `--attributes` with a file and stdin to allocate working copy leases with repositories.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19751
2018-10-26 06:09:20 -07:00
epriestley
6deb09efcd When leasing a Working Copy in Drydock, just "git reset --hard" so empty repositories work
Summary:
Ref T13210. We currently "git reset --hard HEAD" during working copy leasing, mostly by convention/familiarity.

However, this command does not work in an empty repository, because there is no HEAD yet.

The command "git reset --hard" appears to have the same meaning and effect in all cases, except that it also works correctly in an empty repository.

The manual suggests that omitting HEAD should be the same as specifying HEAD, too:

> The <tree-ish>/<commit> defaults to HEAD in all forms.

Test Plan: Successfully leased a working copy for an empty repository using Drydock.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19750
2018-10-26 06:08:47 -07:00
epriestley
e2cf1e4288 Skip copied code detection for changes that are too large for it to be useful
Summary:
Ref T13210. See PHI944. When parsing certain large diffs (the case in PHI944 is an 2.5-million line JSON diff), we spend ~66% of runtime and ~80% of memory doing copy detection (the little yellow bar which shows up to give you a hint that code was moved around within a diff).

This is pretty much pointless and copy hints are almost certainly never useful on large changes. Instead, just bail if the change is larger than some arbitrary "probably too big for copy hints to ever be useful" threshold (here, 65535 lines).

Test Plan:
Roughly, ran this against a 2.5 million line JSON diff:

```
$changes = id(new ArcanistDiffParser())->parseDiff($raw_diff);
$diff = DifferentialDiff::newFromRawChanges($viewer, $changes);
```

Before the changes, it took 20s + 2.5GB RAM to parse. After the changes, it took 7s + 500MB RAM to parse.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19748
2018-10-20 03:36:34 -07:00
epriestley
e0dea4c486 Fix packages(project) to work properly and add it to "MailableFunctionDatasource"
Summary:
Ref T13210. See PHI937. This function datasource isn't quite implemented correctly: it doesn't resolve `package(project)` properly, since the logic only handles users.

This blames back to D14013, where it looks like `packages(..)` was added mostly as a general nice-to-have as part of a larger modernization change.

Test Plan: Ran a `packages(project)` query in Differential, got accurate results (previously: no results).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19747
2018-10-19 13:53:27 -07:00
epriestley
bc6c8c0e93 Explicitly shuffle nodes before selecting one for cluster sync
Summary:
Depends on D19734. Ref T13202. Ref T13109. Ref T10884. See PHI905. See PHI889. We currently rank cluster nodes in three cases:

  # when performing a write, we can go to any node (D19734 should make our ranking good);
  # when performing a read, we can go to any node (currently random, but T10884 discusses ideas to improve our ranking);
  # when performing an internal synchronization before a read or a write, we must go to an up-to-date node.

Currently, case (3) is not-exactly-deterministic but not random, and we won't spread intracluster traffic acrosss the cluster evenly if, say, half of it is up to date and half of it is still synchronizing. For a given write, I believe all nodes will tend to synchronize from whichever node first received the write today.

Instead, shuffle the list and synchronize from any up-to-date node.

(I think we could improve upon this only by knowing which nodes actually have load and selecting the least-loaded -- doable, but not trivial.)

Test Plan: Poked at it locally, will deploy to `secure`. This is hard to measure/test terribly convincingly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202, T13109, T10884

Differential Revision: https://secure.phabricator.com/D19735
2018-10-17 08:11:23 -07:00
epriestley
51073b972e Try to route cluster writes to nodes which won't need to synchronize first
Summary:
Ref T13109. Ref T13202. See PHI905. See PHI889. When we receive a write to a repository cluster, we currently send it to a random writable node.

Instead, we can prefer:

  - the node currently holding the write lock; or
  - any node which is already up to date.

These should simply be better nodes to take writes in all cases. The write lock is global for the repository, so there's no scaling benefit to spreading writes across different nodes, and these particular nodes will be able to accept the write more quickly.

Test Plan:
  - This is observable by using `fprintf(STDERR, "%s\n", ...)` in the logic, then running `git push`. I'd like to pull this routing logic out of `PhabricatorRepository` at some point, probably into a dedicated `ClusterURIQuery` sort of class, but that is a larger change.
  - Added some `fprintf(...)` stuff around which nodes were being selected.
  - Added a `sleep(10)` after grabbing the write lock.
  - In one window, pushed. Then pushed in a second window.
    - Saw the second window select the lock holder as the write target based on it currently holding the lock.
    - Without a concurrent push, saw pushes select up-to-date nodes based on their up-to-date-ness.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: joshuaspence, timhirsh

Maniphest Tasks: T13202, T13109

Differential Revision: https://secure.phabricator.com/D19734
2018-10-17 08:08:25 -07:00
epriestley
0a51bc4f05 Add a space after "View Inline" in mail to prevent double-click on the filename from selecting "Inline"
Summary:
See PHI920. Ref T13210. Since the HTML is just:

```
<a>View Inline</a><span>filename.txt</span>
```

..double-clicking "filename.txt" in email selects "Inlinefilename.txt".

Add a space to stop this. At least in Safari, a space between the tags is not sufficient (perhaps because the parent is a `<div>`?). I couldn't find an authoritative-seeming source on what the rules for this actually are and adding a space here fixes the issue without changing the visual rendering, so just put it here.

Test Plan:
  - Made an inline.
  - Used `bin/mail show-outbound --id ... --dump-html` to dump the HTML.
  - Double-clicked the filename.

{F5929186}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19742
2018-10-11 13:44:20 -07:00
epriestley
8bffc9ea0e In "bin/bulk export", require "--output <path>" by default
Summary:
Depends on D19743. Ref T13210. Since this command can easily dump a bunch of binary data (or just a huge long blob of nonsense) to stdout, default to requiring "--output <file>".

Using `--output -` will print to stdout.

Test Plan: Ran with: no `--output`, `--output file`, `--output -`, `--output - --overwrite`. Got sensible results or errors in all cases.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19744
2018-10-11 13:35:16 -07:00
epriestley
4f54d483d5 Support export of revisions to Excel/CSV/JSON/etc
Summary: Ref T13210. See PHI806. This enables basic export of revisions into flat data formats. This isn't too fancy, but just covers the basics since the driving use case isn't especially concerned about getting all the fields and details.

Test Plan: Exported some revisions into JSON, got sensible output.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19743
2018-10-11 13:34:33 -07:00
epriestley
4f557ff075 When using "bin/bulk export --overwrite", actually overwrite the file
Summary: Depends on D19738. Ref T13210. Currently, when you use "--overwrite", we just //append// the new content. Instead, actually overwrite the file.

Test Plan: Used `--overwrite`, saw an actual clean overwrite instead of an append.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19739
2018-10-11 08:13:43 -07:00
epriestley
4928c34d00 Allow "bin/bulk export" to merge multiple queries and accept more flexible flags
Summary:
Ref T13210. Minor usability improvements to "bin/bulk export":

  - Allow `--class task` to work (previously, only `--class ManiphestTaskSearchEngine` worked).
  - If you run `--query jXIlzQyOYHPU`, don't require `--class`, since the query identifies the class on its own.
  - Allow users to call `--query A --query B --query C` and get a union of all results.

Test Plan:
  - Ran `--class task`, `--query A --query B`, `--query X` (with no `--class`), got good results.
  - Ran various flavors of bad combinations (queries from different engines, invalid engines, query and class differing, ambiguous/invalid `--class` name) and got sensible errors.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19738
2018-10-10 09:14:14 -07:00
epriestley
99034efa8b Make Pholio mail render without a ton of over-escaped HTML
Summary:
Ref T13202. See PHI900. Fixes T12814. Pholio currently builds HTML comments in an older way that can dump a bunch of over-escaped HTML into mail bodies.

Update the logic to be more similar to the Differential rendering logic and stop over-escaping things.

The result isn't perfect, but is dramatically less broken.

Test Plan: {F5919860}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202, T12814

Differential Revision: https://secure.phabricator.com/D19733
2018-10-05 13:37:26 -07:00
epriestley
c6c1893dc0 Allow revisions to be filtered by created date
Summary:
Ref T13202. See PHI906. This is a reasonable capability which we support in some other applications already.

(The only real reason not to support this is that it creates some clutter in the UI, but I think we're generally in better shape now than we were in the past, and we could make this UI collapse/fold at some point.)

Test Plan: Ran queries with a minimum date, a maximum date, both, and neither. Saw appropriate results in all cases.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19732
2018-10-05 12:28:20 -07:00
epriestley
4858d43d16 Add 'autocomplete="off"' to MFA TOTP inputs
Summary:
Ref T13202. See <https://discourse.phabricator-community.org/t/2fa-input-box-isnt-hinted-as-a-password-so-browsers-suggest-auto-fills/1959>.

If browsers are autofilling this, I think browser behavior here is bad, but behavior is probably better on the balance if we hint this as `autocomplete="off"` and this is a minor concesssion.

Test Plan:
  - I couldn't immediately get any browser to try to autofill this field (perhaps I've disabled autofill, or just not enabled it aggressively?), but this change didn't break anything.
  - After the change, answered a TOTP prompt normally.
  - After the change, inspected page content and saw `autocomplete="off"` on the `<input />` node.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19722
2018-10-01 13:08:54 -07:00
epriestley
39b85c0be0 Use the changeset parse cache when rendering inline comments in mail
Summary:
Ref T13202. See PHI903 and PHI894. When a bot leaves 100 inline comments on the same file and the revision has a 30-member recipient list, we currently highlight the file 3000 times when building mail.

Instead, engage the parse cache so we highlight it once and reuse the cache 2,999 times.

Test Plan:
  - Added debugging code to stop after mail generation and show cache hits/misses.
    - Left a bunch of inlines in a file.
    - Ran the worker with `bin/worker execute --id ...`.
    - Before change: saw 4 comments x 2 recipients = 8 cache misses
    - After change: saw 8 cache hits (cache already filled from web UI rendering)
  - Removed debugging code, ran worker to completion.
  - Used `bin/mail show-outbound --id ... --dump-html` to inspect resulting mail, saw no functional differences.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19721
2018-10-01 13:08:27 -07:00
Austin McKinley
dbf2302b6c Fix self-cancelling typo
Summary: Ref D18268. This typo cancelled itself out, and I can't find any other callers.

Test Plan: arc unit

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19718
2018-09-28 17:44:33 -07:00
Austin McKinley
8065433ee8 Migrate DiffusionBlameController to use repo identities
Summary:
Now on the blame page, identities get `avatar.png` and there are little tooltips that show a few characters of the committer identity string.

Also add a default icon for repo identities.

Test Plan: Loaded some blame pages for files touched by users with and without repo identities attached.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19587
2018-09-26 14:45:58 -07:00
epriestley
021c612cb2 When we fail to acquire a repository lock, try to provide a hint about why
Summary:
Ref T13202. See PHI889. If the lock log is enabled, we can try to offer more details about lock holders.

When we fail to acquire a lock:

  - check for recent acquisitions and suggest that this is a bottleneck issue;
  - if there are no recent acquisitions, check for the last acquisition and print details about it (what process, how long ago, whether or not we believe it was released).

Test Plan:
  - Enabled the lock log.
  - Changed the lock wait time to 1 second.
  - Added a `sleep(10)` after grabbing the lock.
  - In one window, ran a Conduit call or a `git fetch`.
  - In another window, ran another operation.
  - Got useful/sensible errors for both ssh and web lock holders, for example:

> PhutilProxyException: Failed to acquire read lock after waiting 1 second(s). You may be able to retry later. (This lock was most recently acquired by a process (pid=12609, host=orbital-3.local, sapi=apache2handler, controller=PhabricatorConduitAPIController, method=diffusion.rawdiffquery) 3 second(s) ago. There is no record of this lock being released.)

> PhutilProxyException: Failed to acquire read lock after waiting 1 second(s). You may be able to retry later. (This lock was most recently acquired by a process (pid=65251, host=orbital-3.local, sapi=cli, argv=/Users/epriestley/dev/core/lib/phabricator/bin/ssh-exec --phabricator-ssh-device local.phacility.net --phabricator-ssh-key 2) 2 second(s) ago. There is no record of this lock being released.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19702
2018-09-24 15:20:07 -07:00
epriestley
7db265cd5d Parameterize the repository read and write locks
Summary:
Ref T13202. See PHI889. Update the read and write locks to the modern parameterized verison, which handles hashing/normalization and can store better logs.

This parameterized mode was added in D19173 and has been used successfully for some time, but not all locks have switched over to it yet.

Test Plan:
- Added an `fprintf(STDERR, $full_name)` to the lock code.
- Pulled a repository.
- Saw sensible lock name on stdout before "acquired read lock...".
- Additional changes in this patch series will vet this more completely.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19701
2018-09-24 15:14:28 -07:00
epriestley
3244324cb1 Fix comment box borders in timelines after Phriction commenting
Summary:
Ref T13202. In D19660, I added comments to Phriction and tweaked some CSS.

One of these tweaks was getting rid of an extra border which was rendering under the comment area. However, I took off too much and ended up removing borders from other applications.

I think we don't actually need this `setNoBorder()` stuff after all -- a later change was sufficient to stop the actual border I was trying to get rid of from rendering. So this mostly just reverts part of D19660.

This rendering still isn't perfect, but I'm fine leaving that for another day for now.

Test Plan:
  - Viewed comment areas in Phriction. Saw correct number of borders (1).
  - Viewed comment areas in Maniphest. Saw correct number of borders (1).
  - Grepped for extraneous removed classs, no hits.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19684
2018-09-19 13:56:58 -07:00
epriestley
5ba66e56fd Fix Phriction rendering for initial install and 404 pages
Summary:
Depends on D19682. Ref T13202. We currently fatal when trying to render a timeline if:

  - an install is fresh, so there are no pages yet, and you look at "/w/"; or
  - you're looking at a Phriction page which doesn't exist (yet) like "/w/aadsflknadsflnf/".

Rendering a timeline and comment area doesn't make sense in these cases, so don't render them.

Test Plan: Hit both cases described above, got "new/empty page" prompts instead of fatals.

Reviewers: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19683
2018-09-17 20:02:59 -07:00
epriestley
e5c6a5749a Fix fatal in rendering Phriction "Moved Away" stories
Summary:
Ref T13202. See PHI881. These stories have bad rendering methods, but they didn't previously render into the timeilne (since Phriction documents didn't have a timeline).

Update the rendering to work.

The rendered outcome isn't great (it isn't very clear or explicit about exactly what moved where), but I'll fix that in a followup. This is a net improvement since it doesn't fatal the page, at least.

Test Plan:
  - Moved page "X" to "Y".
  - Viewed the old page "X".
  - Before patch: bad timeline story would fatal rendering.
  - After patch: story renders, at least, just not great.

Reviewers: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19682
2018-09-17 20:02:06 -07:00
epriestley
0167f357b7 Provide a convenient way to log arbitrary text in Drydock without needing structured log classes
Summary: Depends on D19673. Ref T13197. See PHI873.

Test Plan:
Added some code like this:

```
$operation->logText('Nice convenient text logging.');
```

...then got:

{F5887712}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19674
2018-09-15 07:59:50 -07:00
epriestley
a7e060f062 Write a trivial log when starting a repository operation
Summary:
Depends on D19672. Ref T13197. See PHI873. This writes a trivial log when we begin acting on a working copy and makes it look reasonable in the UI.

This is mostly just to prove that logging works properly.

Test Plan: {F5887697}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19673
2018-09-15 07:57:11 -07:00
epriestley
92bcf85974 Add Drydock logs to the RepositoryOperation UI
Summary:
Depends on D19671. Ref T13197. See PHI873.

Expose logs in the RepositoryOperation UI. Nothing writes the logs yet, so these interfaces are currently always empty.

Test Plan:
{F5887102}

{F5887103}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19672
2018-09-15 07:56:35 -07:00
epriestley
10f219fb82 Allow Drydock logs to be associated with RepositoryOperation objects
Summary: Ref T13197. See PHI873. I want to give RepositoryOperation objects access to Drydock logging like leases, resources, and blueprints currently have. This just does the schema/query changes, no actual UI or new logging yet.

Test Plan: Ran storage upgrade, poked around the UI looking for anything broken.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19671
2018-09-15 07:55:14 -07:00
epriestley
40d5d5c984 Remove "mailKey" from "PhabricatorRepositoryCommit"
Summary: Ref T13197. Ref T13065. This continues the gradual purge of dedicated "mailKey" columns in favor of shared infrastructure.

Test Plan:
  - Ran migration.
  - Visually inspected database.
  - Grepped for `mailKey`.
  - Added some comments, saw the daemons generate corresponding mail.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13197, T13065

Differential Revision: https://secure.phabricator.com/D19670
2018-09-15 07:54:15 -07:00
epriestley
0abf29765e Make Phriction diff-of-changes view draft-aware and clarify some language
Summary:
Ref T13077. Ref T13197. See PHI840.

  - In the "History > Diff/Compare" view, the button language wasn't draft-aware.
  - Revise language to avoid the word "Revert", since this can be ambiguous.
    - "Edit this page, starting with an older version of the text" is now "Edit Older|Current|Draft Version X".
    - "Mark this older version of the page as the current published version" is now "Publish Older Version".
  - Let the user edit the current published version, too, since this is a reasonable operation if there are drafts.

Test Plan: Navigated the history diff view, saw better button and action text.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197, T13077

Differential Revision: https://secure.phabricator.com/D19668
2018-09-13 14:12:49 -07:00
epriestley
0089ef4b60 Don't show Phriction draft edit events in feed
Summary: Depends on D19663. Ref T13077. When you edit a Phriction draft, don't publish a feed story. (The eventual "Publish" event gets a story.)

Test Plan: Made draft / non-draft / publish edits, only saw feed stories for non-draft and publish edits.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19664
2018-09-12 13:56:10 -07:00
epriestley
b6ba75d991 Add a Phriction hint when a draft exists, and fix some draft editing bugs
Summary:
Depends on D19662. Ref T13077. See PHI840.

  - If you're looking at the published version of a document, but a draft version exists and you can edit it, add a hint/link.
  - Fix an issue where the "draft" transaction would complain when you created a document since the initial content is empty and no "draft" transaction is adding any content.

Test Plan: Created new documents, viewed documents with current published versions and unpublished drafts.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19663
2018-09-12 13:45:33 -07:00
epriestley
550028a882 Allow Phriction document edits to be saved as drafts
Summary:
Depends on D19661. Ref T13077. See PHI840.

When a user edits a page normally, add a "Save as Draft" button. Much of this change is around making that button render and behave properly: it needs to be an `<input type="submit" ...>` so browsers submit it and we can figure out which button the user clicked.

Then there are a few minor rules:

  - If you're editing a page which is already a draft, we only give you "Save as Draft". This makes edits to update/revise a draft more natural.
  - Highlight "Publish" if it's a likely action that you might want to take.

Internally, there are two types of edits. Both types create a new version with the new content. However:

  - A "content" edit sets the version shown on the live page to the newly-created version.
  - A "draft" edit does not update the version shown on the live page.

Test Plan: Edited a published document, edited the draft. Published documents. Reverted documents.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19662
2018-09-12 13:30:40 -07:00
epriestley
152e7713eb Remove obsolete, nonfunctional draft auto-saving in Phriction
Summary:
Depends on D19660. Ref T5811. Ref T13077.

Long ago, if you started editing a Phriction document but didn't save it, we'd save the draft in the background as part of the preview.

D11169 updated the preview to use shared infrastructure and broke this function, since we never save drafts.

Since this doesn't work right now, I want to add another thing called "draft", and the future of this feature should be more integrated with modern drafts and EditEngine (which fixed some bugs related to versioning), just get rid of this code for the moment.

Test Plan: Edited documents. This code doesn't do anything since D11169, so no behavior changed.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13077, T5811

Differential Revision: https://secure.phabricator.com/D19661
2018-09-12 13:22:16 -07:00
epriestley
e19c555913 Support (basic) commenting on Phriction documents
Summary:
Depends on D19659. Fixes T1894. Ref T13077. See PHI840.

  - Add an EditEngine, although it currently supports no fields.
  - Add (basic, top-level-only) commenting (we already had the table in the database).

This will probably create some issues. I'm most concerned about documents accumulating a ton of old, irrelevant comments over time which are hard to keep track of and no longer relevant. But I think this is probably a step forward in almost all cases, and a good thing on the balance.

This also moves us incrementally toward putting all editing on top of EditEngine.

Test Plan: {F5877347}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13077, T1894

Differential Revision: https://secure.phabricator.com/D19660
2018-09-12 13:20:52 -07:00
epriestley
f5e90a363e When a user takes actions while in a high security session, note it on the resulting transactions
Summary:
Ref T13197. See PHI873. Record when a user has MFA'd and add a little icon to the transaction, similar to the exiting "Silent" icon.

For now, this just makes this stuff more auditable. Future changes may add ways to require MFA for certain specific transactions, outside of the ones that already always require MFA (like revealing credentials).

Test Plan: {F5877960}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19665
2018-09-12 12:57:02 -07:00
epriestley
8268abcb78 Enrich "diffusion.commit.search" with identity, status, and message information
Summary:
Depends on D19657. Ref T13197. See PHI841.

This enriches the results from `diffusion.commit.search` with information similar to the information returned by the "commits" attachment from `differential.diff.search`.

Also include unreachable, imported, message, audit status, and repository PHID.

Test Plan: Called `diffusion.commit.search` and reviewed the results, which looked sensible.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19658
2018-09-12 12:45:44 -07:00
epriestley
c7e7b63f15 Rename "PhabricatorAuditCommitStatusConstants" to "DiffusionCommitAuditStatus"; remove "MODERN_"
Summary:
Depends on D19656. Ref T13197. See PHI851.

  - This class is now a real object, so get rid of the "Constants" part of the name.
  - Rename it for greater consistency with other modern objects.
  - Get rid of the `MODERN_` tag now that the old constants are gone.

Test Plan: Bunch of `grep`, browsed around.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19657
2018-09-12 12:44:43 -07:00
epriestley
aaf2269551 Remove legacy numeric Audit status constants
Summary: Depends on D19655. Ref T13197. These no longer have callers.

Test Plan: Grepped for these constants.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19656
2018-09-12 12:40:55 -07:00
epriestley
d63281cc54 Migrate remaining Audit database status constants
Summary: Depends on D19652. Ref T13197. See PHI851. This migrates the actual `auditStatus` on Commits, and older status transactions.

Test Plan:
  - Ran migrations.
  - Spot-checked the database for sanity.
  - Ran some different queries, got unchanged results from before migration.
  - Reviewed historic audit state transactions, and accepted/raised concern on new audits. All state transactions appeared to generate properly.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19655
2018-09-12 12:21:27 -07:00
epriestley
cc3b6d5790 Fix a "withHasTransactions()" typo in AuditEditor
Summary: See <https://discourse.phabricator-community.org/t/typo-in-phabricatorauditeditor-php/1910>. This is trivial and reproduces easily, I just missed it in testing.

Test Plan:
  - Left a comment on a commit which I was the author of.
  - Before change: fatal with obvious typo.
  - After change: smooth sailing.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19667
2018-09-12 12:17:58 -07:00
epriestley
2379c21fbb Give Phriction documents a normal timeline
Summary:
Ref T13077. See PHI840. Ref T1894. I'm planning to just let you comment on Phriction documents. I think this will create a few problems (e.g., around popular documents which collect long comment threads that are eventually obsolete) but nothing should be too terribly critical (e.g., we handle it gracefully when objects have very large number of comments/transactions) and for most documents this is likely just a net improvement.

"Just enable comments" is probably not the final iteration on this, but I think it's probably a step forward on the balance, not a step sideways or a slippery slope down into a dark hole or anything.

Test Plan: {F5877316}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13077, T1894

Differential Revision: https://secure.phabricator.com/D19659
2018-09-11 13:31:57 -07:00
epriestley
185e72c881 Add aural section headers for "Event Timeline", "Add Comment", and "Comment Preview"
Summary: See PHI871. Ref T13197. These sections are only divided visually and don't have textual headers. Add aural headers.

Test Plan: {F5875471}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19654
2018-09-11 13:30:10 -07:00
epriestley
8eb8e8e1d8 Make DiffusionCommitSearch accept modern (string) constants
Summary:
Depends on D19650. Ref T13197. Allow `SearchCheckboxesField` to have a "deprecated" map of older aliases, then convert them to modern values.

On the API method page, show all the values.

This technically resolves the issue in PHI841, although I still plan to migrate behind this.

Test Plan:
{F5875363}

- Queried audits, fiddled with `?status=1,audited`, etc.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19651
2018-09-10 16:25:42 -07:00
epriestley
853a816b3c Continue converting Audit constants, allowing the Query to handle either strings or integers
Summary: Ref T13197. We're almost ready to migrate: let the Query accept either older integer values or new string values. Then move some callsites to use strings.

Test Plan: Called `audit.query`, browsed audits, audited commits.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19650
2018-09-10 14:46:47 -07:00
epriestley
bae8a95114 Continue replacing Commit/Audit status checks with object-based checks
Summary: Ref T13195. See PHI851. Continuing down the path toward replacing these legacy numeric constants with more modern string constants.

Test Plan:
- Raised concern, requested verification, verified.
- Looked at commit hovercard with audit status.
- Viewed header on a commit page.
- (Didn't test the Doorkeeper stuff since it requires linking to Asana and seems unlikely to break.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19647
2018-09-10 11:20:31 -07:00
epriestley
16a6fc8341 Allow reviewers to mark their own inlines as "Done" before they submit them
Summary:
Ref T13195. Ref T8573. This allows reviewers to mark their own inline comments as "Done" before they submit them.

If you're leaving a non-actionable comment like "this is good", you can pre-check "Done" to give the author a hint that you don't expect any response.

Test Plan: On revisions and commits, added inlines as the author and a reviewer/auditor. Marked them done/not-done before submitting. As author, marked the not-done ones done after submitting. Checked preivews, toggled done/not done states.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195, T8573

Differential Revision: https://secure.phabricator.com/D19634
2018-09-07 11:17:42 -07:00
epriestley
046c1b5b82 Expose "isDraft" and "holdAsDraft" revision properties over Conduit
Summary:
Ref T13195. See PHI861. The "+ Draft" flag is not currently exposed over the API, but seems stable enough to expose.

Also expose the "hold as draft" flag, normally `arc diff --draft`.

Today, you can get "+ Draft" with some other state by:

  - abandoning a draft revision ("Abandoned + Draft"); or
  - using `arc diff --plan-changes` with an older `arc` version ("Changes Planned + Draft").

Test Plan: Called `differential.revision.search` via Conduit and got sensible-looking results for revisions in various states.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19648
2018-09-07 11:16:42 -07:00
epriestley
a1ce23b9f5 Introduce an AuditStatus object for commits and move some callsites to it
Summary:
Ref T13195. See PHI851. Add an object, analogous to the `DifferentialRevisionStatus` object, to handle audit status management.

This will primarily make it easier to swap storage over to strings later, but also cleans things up a bit.

Test Plan: Viewed audit/commit lists, saw sensible state icons. Ran `bin/audit synchronize`, got sensible output.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19646
2018-09-07 10:20:04 -07:00
epriestley
ab4d33bede Allow underscores to appear in Harbormaster variable names
Summary:
See PHI859. Ref T13195. The regexp for replacing variables currently does not include underscores.

Include underscores.

I also made a note in T13088: we should (almost certainly?) throw immediately if you try to pass a bogus variable name as a custom parameter, but this is a slightly larger change.

Test Plan:
  - Made an "http request" build plan with `?x=${initiator.phid}&y={$some_variable}`.
  - Added `some_variable` as a parameter to the parameter collection.
  - Before patch: `initiator.phid` was replaced, but `some_variable` was not.
  - After patch: both variables are replaced.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19645
2018-09-07 09:53:59 -07:00
epriestley
e8e5dc0f56 Make a language improvement ("inlines" -> "inline comments")
Summary: See D19632. Agreed that this is more clear.

Test Plan: Read carefully.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19644
2018-09-06 11:43:52 -07:00
epriestley
ef26b06ca8 Begin transitioning audits to modern (string) status constants, from legacy (integer) status constants
Summary:
Ref T13195. See PHI851. Audits currently have older integer status constants. We've moved almost all object types away from this to string constants (which are better in basically every way, and particularly way better for exposing over the API).

Commits/audits are currently accessible over the API and expose these constants via a "statuses" constraint.

Prepare to move toward modern string constants by defining a new, more modern map of status details and defining the existing methods in terms of it.

Test Plan: Browsed audits checking for icons/names/open-ness, saw no changes. This change should have no user-visible effects, as it just reorganizes code.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19642
2018-09-06 10:56:47 -07:00
epriestley
5a38b75f16 In Conduit, let checkbox constraints self-document
Summary:
Ref T13195. Ref PHI851. Currently, checkbox constraints don't tell you what values are supported on the API page.

You can figure this out with "Inspect Element" or by viewing the source code, but just render a nice table instead.

Test Plan: {F5862969}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19641
2018-09-06 08:44:16 -07:00
epriestley
a20f0674a9 Improve some documentation for "diffusion.commit.search"
Summary:
Ref T13195. See PHI851. Start by making some minor improvements here:

  - Clarify that the example of what constraints look like is just an example.
  - Add descriptions for parameters missing descriptions.

Test Plan: Looked at API method page, saw more helpful/complete instructions.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19640
2018-09-06 08:39:21 -07:00
epriestley
e6ee5ee9a1 When applying repository operations via Drydock, provide more context on OperationType
Summary: Ref T13195. See PHI845. For custom OperationTypes, provide access to the Interface and Operation via getters. This is just for convenience, since passing these around everywhere can be a bit of a pain if you have a deep-ish stack of things or love using callbacks or whatever.

Test Plan: Landed a revision via upstream Drydock operations.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19636
2018-09-06 08:15:16 -07:00
epriestley
041392988e When a transaction adds more than 100 inline comments, include only the first 100 in email
Summary:
Ref T13195. An install had a user apply a transaction which added about 1,000 inline comments. Rendering the email for this transaction took a very long time because the context section for each comment must be highlighted separately.

We can make the highlighting faster (in this case, by porting the lexer to PHP) but it's also sort of silly to include more than 100 inlines in an email. These emails are likely to be truncated by outbound size rules at some point anyway. Instead, limit inlines rendered directly into email to the first 100 per transaction group.

Test Plan:
Set limit to 2, added 4 comments, viewed text and HTML emails:

{F5859967}

{F5859968}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19632
2018-09-06 07:58:05 -07:00
epriestley
650e74933a Update some inline comment logic to use more modern "Viewer"-oriented calls/variables
Summary:
Ref T13195. Ref T8573. The inline comment controllers currently use outdated `$user = $this->getRequest()->getUser()` calls.

Instead, use `$viewer = $this->getViewer()`.

This is just a small consistency update with no behavioral changes.

Test Plan: Viewed and added inlines in Differential and Diffusion.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195, T8573

Differential Revision: https://secure.phabricator.com/D19633
2018-09-06 07:57:41 -07:00
epriestley
26d9ee456f Hide the new Phriction "Publish" operation behind the "Prototype" toggle
Summary: Ref T13077. This is currently a little too confusing to go out into the world, mostly because there's no way to edit documents without auto-publishing them. Keep it out of the spotlight for this release.

Test Plan: Viewed Phriction, saw publish operation marked as a prototype.

Reviewers: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19627
2018-08-31 10:30:20 -07:00
epriestley
18e8d9a452 Make the Phriction "History" view more aware of drafts
Summary: Ref T13077. Updates the "History" view to be slightly better organized and draft-aware.

Test Plan: Viewed page history in Phriction.

Reviewers: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19626
2018-08-30 10:36:55 -07:00
epriestley
3b1294cf45 Store Phriction max version on Document, improve editing rules for editing documents with drafts
Summary:
Ref T13077. We need to know the maximum version of a document in several cases, so denormalize it onto the Document object.

Then clean up some behaviors where we edit a document with, e.g., 7 versions but version 5 is currently published. For now, we: edit starting with version 7, save as version 8, and immediately publish the new version.

Test Plan:
  - Ran migration.
  - Edited a draft page without hitting any weird version errors.
  - Checked database for sensible `maxVersion` values.

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19625
2018-08-30 10:12:51 -07:00
epriestley
0a77b0e53e Work around an issue in MariaDB where dropping a column from a UNIQUE KEY fails
Summary:
See T13193. See T13077. If we drop a column which is part of a UNIQUE KEY, MariaDB raises an error.

This is probably a bad idea on our side anyway, but in this case it wasn't an obviously bad idea.

To get around this:

  - Drop the unique key, if it exists, before dropping the column.
  - Explicitly add the new unique key afterward.

Test Plan: Ran `bin/storage upgrade` locally without issue, but I'm on MySQL. Will follow up on T13193.

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19624
2018-08-30 06:25:39 -07:00
epriestley
75a0455152 Add "Revision test plan" as a Herald field; remove test plan from the "Revision summary" field
Summary:
See PHI844. Ref T13189.

Add "Revision test plan" as an available field for Herald. This is a little niche -- and a little odd because it sticks around even if you fully disable test plans -- but probably broadly reasonable.

The existing "Revision summary" field counterintuitively included the test plan. Separate this out since it's now a separate field and the behavior was weird historic nonsense. I'll note this in the changelog.

Test Plan: Wrote a rule using both fields, verified they generated the expected values.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19623
2018-08-29 14:17:38 -07:00
epriestley
876638e428 Add a UI element for navigating between versions of a Phriction document
Summary: Depends on D19621. Ref T13077. Fixes T4815. This adds previous/current/next/draft buttons and makes navigation between unpublished and published versions of a document more clear.

Test Plan: {F5841997}

Reviewers: amckinley

Maniphest Tasks: T13077, T4815

Differential Revision: https://secure.phabricator.com/D19622
2018-08-29 13:49:15 -07:00
epriestley
349686319e Allow the published version of a Phriction document to differ from the most recent version
Summary:
Depends on D19620. Ref T13077. This adds a "Publish" operation which points the current version at some historical version of the document -- not necessarily the most recent version. Newer versions become "drafts".

This is still quite rough and missing a lot of hinting in the UI, I'm just making it work so I can start making the UI understand it.

Test Plan: Used the "Publish" action to publish older versions of a document, saw the document revert. Many UI hints are missing and this operation is puzzling and not yet usable for normal users.

Reviewers: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19621
2018-08-29 13:47:36 -07:00
epriestley
50f4adef64 Remove on-object mailkeys from Phriction
Summary: Depends on D19619. Ref T13065. Ref T13077. Migrate Phriction mail keys to the new infrastructure and drop the column.

Test Plan: Ran migrations, spot-checked the database.

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13077, T13065

Differential Revision: https://secure.phabricator.com/D19620
2018-08-29 13:43:13 -07:00
epriestley
64cee4a902 Move Phriction internal document/content references from IDs to PHIDs
Summary:
Ref T13077. This is mostly just a small cleanup change, even though the actual change is large.

We currently reference content and document objects from one another with `contentID` and `documentID`, but this means that `contentID` must be nullable. Switching to PHIDs allows the column to be non-nullable.

This also supports reorienting some current and future transactions around PHIDs, which is preferable for the API. In particular, I'm adding a "publish version X" transaction soon, and would rather callers pass a PHID than an ID or version number, since this will make the API more consistent and powerful.

Today, `contentID` gets used as a cheaty way to order documents by (content) edit time. Since PHIDs aren't orderable and stuff is going to become actually-revertible soon, replace this with an epoch timestamp.

Test Plan:
  - Created, edited, moved, retitled, and deleted Phriction documents.
  - Grepped for `documentID` and `contentID`.
  - This probably breaks //something// but I'll be in this code for a bit and am likely to catch whatever breaks.

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19619
2018-08-29 13:41:24 -07:00
epriestley
04f8270a74 Remove some unusual UI policy hints in Phriction
Summary:
Ref T13077. We currently have these weird policy hints in Phriction that we don't use in other applications. Just remove them for consistency to make the eventual swap to EditEngine a little easier.

Also nuke some unreacahble code.

Test Plan: Loaded edit page, saw more standard UI.

Reviewers: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19618
2018-08-29 07:32:45 -07:00
epriestley
4afb6446d9 Allow DocumentView to render with a curtain, and make Phriction use a curtain
Summary:
Depends on D19616. Ref T13077. Fixes T8172. In the last round of design updates, a lot of actions got stuffed into "Actions" menus.

I never really got used to these and think they're a net usability loss, and broadly agree with the feedback in T8172. I'd generally like to move back toward a state where actions are available on the page, not hidden in a menu.

For now, just put a curtain view on these pages. This could be refined later (e.g., stick this menu to the right hand side of the screen) depending on where other Phriction changes go.

(Broadly, I'm also not satisfied with where we ended up on the fixed-width pages like Diffusion > Manage, Config, and Instances. In contrast, I //do// like where we ended up with Phortune in terms of overall design. I anticipate revisiting some of this stuff eventually.)

Test Plan:
  - Looked at Phriction pages on desktop/tablet/mobile/printable -- actions are now available on the page.
  - Looked at other DocumentView pages (like Phame blogs) -- no changes for now.

Reviewers: amckinley

Maniphest Tasks: T13077, T8172

Differential Revision: https://secure.phabricator.com/D19617
2018-08-28 14:58:05 -07:00
epriestley
fd0da4c41f Rename "PHUIDocumentViewPro" to "PHUIDocumentView"
Summary: Ref T13077. There is no "PHUIDocumentView" so toss the "Pro" suffix from this classname.

Test Plan: Grepped for `PHUIDocumentView` and `PHUIDocumentViewPro`.

Reviewers: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19616
2018-08-28 14:53:07 -07:00
epriestley
614f9ba1fb Allow unit test results to specify that their details are formatted with remarkup when reporting to "harbormaster.sendmessage"
Summary: Ref T13189. See PHI710. Ref T13088. Fixes T9951. Allow callers to `harbormaster.sendmessage` to specify that the test details are remarkup so they can use rich formatting and include links, files, etc.

Test Plan: {F5840098}

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13189, T13088, T9951

Differential Revision: https://secure.phabricator.com/D19615
2018-08-28 13:26:11 -07:00
epriestley
632cafec88 Pass commit authorship information to Buildkite
Summary:
Fixes T12251. Ref T13189. See PHI610. The difficulty here is that we don't want to disclose Phabricator account information to Buildkite. We're comfortable disclosing information from `git`, etc.

  - For commits, use the Identity to provide authorship information from Git.
  - For revisions, use the local commit information on the Diff to provide the Git/Mercurial/etc author of the HEAD commit.

Test Plan:
  - Built commits and revisions in Buildkite via Harbormaster.
  - I can't actually figure out how to see author information on the Buildkite side, but the values look sane when dumped locally.

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13189, T12251

Differential Revision: https://secure.phabricator.com/D19614
2018-08-27 12:52:11 -07:00
epriestley
2f5c6541fc Add an "Activated Epoch" and an "Acquired Epoch" to Drydock Leases
Summary: Ref T13189. See PHI690. When a lease is first acquired or activated, note the time. This supports better visibility into queue lengths. For now, this is only queryable via DB and visible in the UI, but can be more broadly exposed in the future.

Test Plan: Landed a revision, saw the leases get sensible timestamps for acquisition/activation.

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19613
2018-08-27 11:27:45 -07:00
epriestley
ee823982a4 Remove another old remarkup engine callsite in Config
Summary: Ref T13189. Summaries do not appear to be meaningfully rendered with Remarkup so just drop the engine. See D19610 for the previous change in this vein.

Test Plan: Viewed config list with option summaries.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19612
2018-08-27 11:10:14 -07:00
epriestley
b87a809b0b Make some remarkup handling in Config cleaner, fixing {{other.option} links
Summary:
Depends on D19609. Ref T13189. At some point, we switched from RemarkupEngine to RemarkupView and lost this piece of hack-magic.

Restore the hack-magic. It's still hack-magic instead of a real rule, but things are at least cleaner than they were before.

Test Plan: Viewed `auth.require-approval`, etc. Saw references to other config options linked properly.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19610
2018-08-27 09:54:19 -07:00
epriestley
0295a00229 When there are no setup issues, don't show a weird empty box
Summary: Ref T13189. When there are no setup issues, we currently double-render a weird setup issues box underneath the notice. Get rid of it.

Test Plan: Viewed page with and without setup issues, saw less awkward UI.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19609
2018-08-27 09:53:52 -07:00
epriestley
cd8b5b82c8 Stop requiring CAN_EDIT to reach the TransactionEditor via "*.edit" in EditEngine
Summary:
Depends on D19607. Ref T13189. See PHI642. Ref T13186.

Some transactions can sometimes be applied to objects you can not edit. Currently, using `*.edit` to edit an object always explicitly requires CAN_EDIT.

Now that individual transactions require CAN_EDIT by default and can reduce or replace this requirement, stop requiring CAN_EDIT to reach the editor.

The only expected effect of this change is that low-permission edits (like disabling a user, leaving a project, or leaving a thread) can now work via `*.edit`.

Test Plan:
  - Tried to perform a normal edit (changing a task title) against an object with no CAN_EDIT. Still got a permissions error.
  - As a non-admin, disabled other users while holding the "Can Disable Users" permission.
  - As a non-admin, got a permissions error while trying to disable other users while not holding the "Can Disable Users" permission.

Reviewers: amckinley

Maniphest Tasks: T13189, T13186

Differential Revision: https://secure.phabricator.com/D19608
2018-08-27 08:10:08 -07:00
epriestley
f9192d07f2 Align web UI "Disable" and "Approve/Disapprove" flows with new "Can Disable Users" permission
Summary:
Depends on D19606. Ref T13189. See PHI642.

  - Disabling/enabling users no longer requires admin. Now, you just need "Can Disable Users".
  - Update the UI to appropriately show the action in black or grey depending on what clicking the button will do.
  - For "Approve/Disapprove", fix a couple bugs, then let them go through without respect for "Can Disable Users". This is conceptually a different action, even though it ultimately sets the "Disabled" flag.

Test Plan:
  - Disabled/enabled users from the web UI as various users, including a non-administrator with "Can Disable Users".
  - Hit permissions errors from the web UI as various users, including an administrator without "Can Disable Users".
  - Saw the "Disable/Enable" action activate properly based on whether clicking the button would actually work.
  - Disapproved a user without "Can Disable Users" permission, tried to re-disapprove a user.
  - Approved a user, tried to reapprove a user.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19607
2018-08-27 08:09:42 -07:00
epriestley
058952e72e Add a "Can Disable Users" capability to the "People" application
Summary:
Depends on D19605. Ref T13189. See PHI642. This adds a separate "Can Disable Users" capability, and makes the underlying transaction use it.

This doesn't actually let you weaken the permission, since all pathways need more permissions:

  - `user.edit` needs CAN_EDIT.
  - `user.disable/enable` need admin.
  - Web UI workflow needs admin.

Upcoming changes will update these pathways.

Without additional changes, this does let you //strengthen// the permission.

This also fixes the inability to disable non-bot users via the web UI.

Test Plan:
  - Set permission to "No One", tried to disable users. Got a tailored policy error.
  - Set permission to "All Users", disabled/enabled a non-bot user.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19606
2018-08-27 08:01:27 -07:00
epriestley
8cf56913d8 Deprecate "user.enable" and "user.disable" API methods, redefine them in terms of "user.edit"
Summary:
Depends on D19604. Ref T13189. See PHI642. Deprecates these in favor of "user.edit", redefines them in terms of it, and removes the old `disableUser()` method.

I kept the "is admin" permissions check for consistency, since these methods have always said "(admin only)". This check may not be the most tailored check soon, but we can just keep executing it in addition to the real check.

For now, this change stops this method from actually disabling non-bot users (since it implicitly adds a CAN_EDIT requirement, and even administrators don't have that). An upcoming change will fix that.

Test Plan: Enabled and disabled a (bot) user via these methods. Checked API UI, saw them marked as "disabled".

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19605
2018-08-27 08:00:48 -07:00
epriestley
2f7b10c023 Replace "Disable User" web UI flow with transactions
Summary:
Ref T13189. See PHI642. Upgrades the "Disable" action in the web UI to be transaction-based.

This technically breaks things a little (you can't disable non-bot users, since they now require CAN_EDIT and you won't have it) but an upcoming change will fix the permissions issue.

Test Plan: Disabled and enabled a (bot) user from the web UI.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19604
2018-08-27 08:00:21 -07:00
epriestley
4d89afcc61 Remove requireCapabilities() from ApplicationTransactionEditor and require CAN_EDIT by default
Summary:
Depends on D19585. Ref T13164.

Almost all transactions require CAN_EDIT on the object, but they generally do not enforce this directly today. Instead, this is effectively enforced by Controllers, API methods, and EditEngine doing a `CAN_EDIT` check when loading the object to be edited.

A small number of transactions do not require CAN_EDIT, and instead require only a weaker/lesser permission. These are:

  - Joining a project which you have CAN_JOIN on.
  - Leaving a project which isn't locked.
  - Joining a Conpherence thread you can see (today, no separate CAN_JOIN permission for Conpherence).
  - Leaving a Conpherence thread.
  - Unsubscribing.
  - Using the special `!history` command from email.

Additionally, these require CAN_INTERACT, which is weaker than CAN_EDIT:

  - Adding comments.
  - Subscribing.
  - Awarding tokens.

Soon, I want to add "disabling users" to this list, so that you can disable users if you have "Can Disable User" permission, even if you can not otherwise edit users.

It's possible this list isn't exhaustive, so this change might break something by adding a policy check to a place where we previously didn't have one. If so, we can go weaken that policy check to the appropriate level.

Enforcement of these special cases is currently weird:

  - We mostly don't actually enforce CAN_EDIT in the Editor; instead, it's enforced before you get to the editor (in EditEngine/Controllers).
  - To apply a weaker requirement (like leaving comments or leaving a project), we let you get through the Controller without CAN_EDIT, then apply the weaker policy check in the Editor.
  - Some transactions apply a confusing/redundant explicit CAN_EDIT policy check. These mostly got cleaned up in previous changes.

Instead, the new world order is:

  - Every transaction has capability/policy requirements.
  - The default is CAN_EDIT, but transactions can weaken this explicitly they want.
  - So now we'll get requirements right in the Editor, even if Controllers or API endpoints make a mistake.
  - And you don't have to copy/paste a bunch of code to say "yes, every transaction should require CAN_EDIT".

Test Plan:
- Tried to add members to a Conpherence thread I could not edit (permissions error).
- Left a Conpherence thread I could not edit (worked properly).
- Joined a thread I could see but could not edit (worked properly).
- Tried to join a thread I could not see (permissions error).
- Implemented `requireCapabilites()` on ManiphestTransactionEditor and tried to edit a task (upgrade guidance error).
- Mentioned an object I can not edit on another object (works).
- Mentioned another object on an object I can not edit (works).
- Added a `{F...}` reference to an object I can not edit (works).
- Awarded tokens to an object I can not edit (works).
- Subscribed/unsubscribed from an object I can not edit (works).
- Muted/unmuted an object I can not edit (works).
- Tried to do other types of edits to an object I can not edit (correctly results in a permissions error).
- Joined and left a project I can not edit (works).
- Tried to edit and add members to a project I can not edit (permissions error).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19586
2018-08-24 17:45:56 -07:00
epriestley
b584834b19 In Differential: when the file tree is enabled, default to the "History" tab instead of "Files"
Summary:
Ref T13187. See PHI811. If the file tree is enabled and visible, set the default tab to "History".

  - This is a bit magic.
  - It won't work entirely on mobile (we can't tell that you're on mobile on the server, so we'll pick the "History" tab even though the file tree isn't actually visible on your device).
  - There's no corresponding logic in Diffusion. Diffusion doesn't have the same tab layout, but this makes things somewhat inconsistent.

So I don't love this, but we can try it and see if it's confusing or helpful on the balance.

Test Plan: With filetree on and off, reloaded revisions. Saw appropriate tab selected by default.

Reviewers: amckinley

Maniphest Tasks: T13187

Differential Revision: https://secure.phabricator.com/D19601
2018-08-24 10:29:35 -07:00
epriestley
b6fa009cf0 Enrich "priority" transactions in Maniphest for "transaction.search"
Summary:
Ref T13187. See <https://discourse.phabricator-community.org/t/task-priority-change-info-missing-in-firehose-webhook/1832/2>. We can reasonably enrich these transactions.

Since priorities don't have unique authorative string identifiers, I've mostly mimicked the `maniphest.search` structure.

Test Plan: Called `transaction.search` on tasks which were: created normally, created with a priority change, saw a priority change after creation. All the output looked useful and sensible.

Reviewers: amckinley

Maniphest Tasks: T13187

Differential Revision: https://secure.phabricator.com/D19599
2018-08-24 10:05:05 -07:00
epriestley
5e4d9dfa92 Condition "Changes Since Last Action" Differential link on "first broadcast", not "new object"
Summary: Ref T13187. Ref T13176. With drafts, we actually want to suppress this link on "first broadcast" (the first time we send mail), not on "new object" (when the revision is created as a draft).

Test Plan: Poked at this locally, will keep an eye on it in production.

Reviewers: amckinley

Maniphest Tasks: T13187, T13176

Differential Revision: https://secure.phabricator.com/D19598
2018-08-24 10:03:55 -07:00
epriestley
ca618a8679 Document that phd.taskmasters is a local setting, per daemon
Summary: Ref T13187. See PHI807. The documentation currently does not make it very clear that this is a local setting, per `phd` process. Make it more clear.

Test Plan: {F5827757}

Reviewers: amckinley

Maniphest Tasks: T13187

Differential Revision: https://secure.phabricator.com/D19597
2018-08-24 08:08:19 -07:00
epriestley
7ef2bb1b56 Support Mercurial "protocaps" wire command
Summary:
Ref T13187. See PHI834. Mercurial has somewhat-recently (changeset is from Jan 2018) introduced a new "protocaps" command, that appears in Mercurial 4.7 and possibly before then.

We must explicitly enumerate all protocol commands because you can't decode the protocol without knowing how many arguments the command expects, so enumerate it.

(Also fix an issue where the related error message had an extra apostrophe.)

Test Plan:
  - Ran `hg clone ...` with client and server on Mercurial 4.7.
  - Before: fatal on unknown "protocaps" command.
  - Midway: better typography in error message.
  - After: clean clone.

Reviewers: amckinley

Maniphest Tasks: T13187

Differential Revision: https://secure.phabricator.com/D19596
2018-08-23 15:06:25 -07:00
epriestley
75a5dd8d8c Add more accessibility labels for screen readers
Summary:
Depends on D19594. See PHI823. Ref T13164.

  - Add a label for the "X" button in comment areas, like "Remove Action: Change Subscribers".
  - Add a label for the floating header display options menu in Differential.
  - Add `role="button"` to `PHUIButtonView` objects that we render with an `<a ...>` tag.

Test Plan:
Viewed a revision with `?__aural__=true`:

  - Saw "Remove Action: ..." label.
  - Saw "Display Options" label.
  - Used inspector to verify that some `<a class="button" ...>` now have `<a class="button" role="button" ...>`. This isn't exhaustive, but at least improves things. A specific example is the "edit", "reply", etc., actions on inline comments.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19595
2018-08-17 13:31:51 -07:00
Austin McKinley
5c4c593af3 Update DiffusionLastModifiedController to use identities
Summary: Ref T12164. Updates another controller to use identities.

Test Plan:
Pretty ad-hoc, but loaded the main pages of several different repos with and without repo identities. I'm not totally convinced the `author` from this data structure is actually being used:
```
$return = array(
  'commit'    => $modified,
  'date'      => $date,
  'author'    => $author,
  'details'   => $details,
);
```

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19580
2018-08-17 12:24:21 -07:00
epriestley
438edde031 Add some missing aural button labels for accessibility
Summary:
Ref T13164. See PHI823. (See that issue for some more details and discussion.)

Add aural labels to various buttons which were missing reasonable aural labels.

The "Search" button (magnifying glass in the global search input) had an entire menu thing inside it. I moved that one level up and it doesn't look like it broke anything (?). All the other changes are pretty straightforward.

Test Plan:
{F5806497}

{F5806498}

  - Will follow up on the issue to make sure things are in better shape for the reporting user.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19594
2018-08-17 11:00:29 -07:00
epriestley
a48e6897a4 Remove obsolete setup check call to Maniphest "Can Edit <X>" field checks
Summary: Ref T13164. Missed this in D19581.

Test Plan:
  - Forced setup checks to re-run by visiting {nav Config > Setup Issues} explicitly.
  - Before patch: fatal on call to nonexistent method.
  - After patch: setup issues.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19593
2018-08-16 13:56:20 -07:00
epriestley
0ccf1410e0 Give PhabricatorAuthPassword a formal CAN_EDIT policy
Summary:
Depends on D19585. Ref T13164. This is a precursor for D19586, which causes Editors to start doing more explicit CAN_EDIT checks.

Passwords have an Editor, but don't actually define a CAN_EDIT capability. Define one (you can edit a password if you can edit the object the password is associated with).

(Today, this object is always a User -- this table just unified VCS passwords and Account passwords so they can be handled more consistently.)

Test Plan:
  - With D19586, ran unit tests and got a pass.
  - Edited my own password.
  - Tried to edit another user's password and wasn't permitted to.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19592
2018-08-16 11:53:24 -07:00
epriestley
7e29ec2e2a Move the "Can Lock Projects" check from requireCapabilities() to transaction validation
Summary: Depends on D19584. Ref T13164. This check is an //extra// check: you need EDIT //and// this capability. Thus, we can do it in validation without issues.

Test Plan:
  - This code isn't reachable today: all methods of applying this transaction do a separate check for "Can Lock" upfront.
  - Commented out the "Can Lock" check in the LockController, tried to lock as a user without permission. Was rejected with a policy exception.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19585
2018-08-16 10:56:00 -07:00
epriestley
3b92da22f4 Move the hierarchical edit policy check in Phriction from requireCapabilities() to validateTransactions()
Summary:
Depends on D19583. Ref T13164. This continues the work of getting rid of `requireCapabilities()`.

This check is valid, but can be a `validateTransactions()` check instead. This is generally more consistent with how other applications work (e.g., creating subprojects).

The UI for this isn't terribly great: you get a policy error //after// you try to create the object. But that's how it worked before, so this isn't any worse than it was. The actual policy exception is (very) slightly more clear now (raised against the right object).

Test Plan:
  - Created a child as a user with permission to do so to make sure I didn't break that.
  - Set edit permission on `a/` to just me, tried to create `a/b/` as another user, got a policy exception since they can't edit the parent.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19584
2018-08-16 10:55:11 -07:00
epriestley
24d4445845 Remove pointless requireCapabilities() method from PhabricatorRepositoryEditor
Summary: Depends on D19582. Ref T13164. It's not possible to reach the editor without passing through a CAN_EDIT check, and it shouldn't be necessarily to manually specify that edits require CAN_EDIT by default.

Test Plan: Grepped for `RepositoryEditor`, verified that all callsites pass through a CAN_EDIT check.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19583
2018-08-16 10:53:42 -07:00
epriestley
a39852ae1b Remove pointless requireCapabilities() method from PhabricatorProjectColumnTransactionEditor
Summary:
Depends on D19581. Ref T13164. This method has no effect:

  - You must always have CAN_EDIT to reach an Editor in the first place.
  - Per previous change, I'm going to restructure this so transactions explicitly check CAN_EDIT by default anyway.

Test Plan: Tried to edit and hide a project column as a user without permission, hit global permission checks long before reaching this method.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19582
2018-08-16 10:51:57 -07:00
epriestley
296bf046a8 Remove deprecated Maniphest "Can Edit <Specific Property>" capabilities
Summary:
Depends on D19579. Fixes T10003. These have been deprecated with a setup warning about their impending removal for about two and a half years.

Ref T13164. See PHI642. My overall goal here is to simplify how we handle transactions which have special policy behaviors. In particular, I'm hoping to replace `ApplicationTransactionEditor->requireCapabilities()` with a new, more clear policy check.

A problem with `requireCapabilities()` is that it doesn't actually enforce any policies in almost all cases: the default is "nothing", not CAN_EDIT. So it ends up looking like it's the right place to specialize policy checks, but it usually isn't.

For "Disable", I need to be able to weaken the check selectively (you can disable users if you have the permission, even if you can't edit them otherwise). We have a handful of other edits which work like this (notably, leaving and joining projects) but they're very rare.

Test Plan: Grepped for all removed classes. Edited a Maniphest task.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164, T10003

Differential Revision: https://secure.phabricator.com/D19581
2018-08-16 10:51:06 -07:00
epriestley
f9673a72a8 Allow "user.edit" to enable or disable users
Summary:
Depends on D19577. Ref T13164. See PHI642. This adds modern transaction-oriented enable/disable support.

Currently, this also doesn't let you disable normal users even when you're an administrator. I'll refine the policy model later in this change series, since that's also the goal here (let users set "Can Disable Users" to some more broad set of users than "Administrators").

This also leaves us with two different edit pathways: the old UserEditor one and the new UserTransactionEditor one. The next couple diffs will redefine the other pathways in terms of this pathway.

Test Plan:
  - Enabled/disabled a bot.
  - Tried to disable another non-bot user. This isn't allowed yet, since even as an administrator you don't have CAN_EDIT on them and currently need it: right now, there's no way for a particular set of transactions to say they can move forward with reduced permissions.
  - Tried to enable/disable myself. This isn't allowed since you can't enable/disable yourself.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19579
2018-08-16 10:49:35 -07:00
epriestley
65904d7c51 Add a modern "user.edit" API method for users
Summary:
Depends on D19576. Ref T13164. See PHI642. This adds an EditEngine for users and a `user.edit` modern API method.

For now, all it supports is editing real name, blurb, title, and icon (same as "Edit Profile" from the UI).

Test Plan:
  - Edited my stuff via the new API method.
  - Tried to edit another user, got rejected by policies.
  - Tried to create a user, got rejected by policies.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19577
2018-08-16 10:48:38 -07:00
epriestley
39d415e90e Move users to modular transactions
Summary:
Ref T13164. See PHI642. I'd like to provide a third-generation `user.edit` API endpoint and make `user.enable` and `user.disable` obsolete before meddling with policy details, even if it isn't full-fledged yet.

Users do already have a transactions table and a Transaction-based editor, but it's only used for editing title, real name, etc. All of these are custom fields, so their support comes in automatically through CustomField extension code.

Realign it for modular transactions so new code will be fully modern. There are no actual standalone transaction types yet so this diff is pretty thin.

Test Plan:
  - Grepped for `UserProfileEditor`.
  - Edited a user's title/real name/icon.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19576
2018-08-16 10:47:47 -07:00
Austin McKinley
cc1def6cea Remove some array typehints for passing around
Summary: See discussion at https://secure.phabricator.com/D19492#241996

Test Plan: Refreshed a few Diffusion tabs; nothing broke.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19578
2018-08-13 16:07:56 -07:00
Austin McKinley
3b05e920e0 Start changing DiffusionCommitController to use identities
Summary: Depends on D19491.

Test Plan: Viewed some commits where the identity was mapped to a user and another that wasn't; saw the header render either a link to the user or the identity object.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19492
2018-08-13 15:23:31 -07:00
epriestley
92a29f72c1 Make the Drydock repository operation page slightly richer
Summary:
Ref T13164. See PHI788. The issue requests a "created" timestamp.

Also add filtering for repository, state, and author.

Test Plan:
Used all filters.

{F5795085}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19574
2018-08-13 11:42:10 -07:00
epriestley
fb3ae72e36 When cancelling addition of an Almanac interface, return to the Device page
Summary:
Fixes T13184. In Almanac, interfaces are always added to devices. However, if you "Add New Interface" and then "Cancel", you go to the nonexistent `/interface/` page.

Instead, return to the device page.

Test Plan: From a device page, clicked "Add Interface" and then "Cancel". Ended up back where I was.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13184

Differential Revision: https://secure.phabricator.com/D19573
2018-08-13 11:39:37 -07:00
epriestley
b86dae6214 Fix an issue with error handling when no mailers are available
Summary:
Ref T13164. See PHI785. See D19546. I think I didn't test the updated error messaging here entirely properly, since I have some tasks in queue which error out here ("Missing argument 1 to newMailers(...)").

This is an error condition already, but we want to get through this call so we can raise a tailored message.

Test Plan: Tasks which errored out here now succeed. This condition is only reachable if you misconfigure things in the first place.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19572
2018-08-13 11:39:13 -07:00
epriestley
e5906f4e12 In Differential standalone views, disable some keyboard shortcuts which don't work
Summary:
Ref T13164. See PHI693. In Differential, you can {nav View Options > View Standalone} to get a standalone view of a single changeset. You can also arrive here via the big changeset list for revisions affecting a huge number of files.

We currently suggest that all the keyboard shortcuts work, but some do not. In particular, the "Next File" and "Previous File" keyboard shortcuts (and some similar shortcuts) do not work. In the main view, the next/previous files are on the same page. In the standalone view, we'd need to actually change the URI.

Ideally, we should do this (and, e.g., put prev/next links on the page). As a first step toward that, hide the nonfunctional shortcuts to stop users from being misled.

Test Plan:
  - Viewed a revision in normal and standalone views.
  - No changes in normal view, and all keys still work ("N", "P", etc).
  - In standalone view, "?" no longer shows nonfunctional key commands.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19571
2018-08-13 08:59:05 -07:00
Austin McKinley
a6951a0a5a Add migration to encourage rebuilding repository identities
Summary: Ref T12164. Defines a new manual activity that suggests rebuilding repository identities before Phabricator begins to rely on them.

Test Plan:
- Ran migration, observed expected setup issue: {F5788217}
- Ran `bin/config done identities` and observed setup issue get marked as done.
- Ran `/bin/storage upgrade --apply phabricator:20170912.ferret.01.activity.php` to make sure I didn't break the reindex migration; observed reindex setup issue appear as expected.
- Ran `./bin/config done reindex` and observed reindex issue cleared as expected.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19497
2018-08-10 13:47:03 -07:00
Austin McKinley
2951694c27 Correctly spell 'committer'
Summary: It's a funny word. h/t @joshuaspence

Test Plan: Inspection of correct spelling.

Reviewers: epriestley, joshuaspence

Reviewed By: joshuaspence

Subscribers: Korvin, joshuaspence

Differential Revision: https://secure.phabricator.com/D19570
2018-08-09 17:52:43 -07:00
epriestley
6df278bea8 In "bin/ssh-auth", cache a structure instead of a flat file because paths may change at runtime
Summary:
Fixes T12397. Ref T13164. See PHI801.

Several installs have hit various use cases where the path on disk where Phabricator lives changes at runtime. Currently, `bin/ssh-auth` caches a flat file which includes the path to `bin/ssh-exec`, so this may fall out of date if `phabricator/` moves.

These use cases have varying strengths of legitimacy, but "we're migrating to a new set of hosts and the pool is half old machines and half new machines" seems reasonably compelling and not a problem entirely of one's own making.

Test Plan:
  - Compared output on `master` to output after change, found them byte-for-byte identical.
  - Moved `phabricator/` to `phabricator2/`, ran `bin/ssh-auth`, got updated output.
  - Added a new SSH key, saw it appear in the output.
  - Grepped for `AUTHFILE_CACHEKEY` (no hits).
  - Dropped the cache, verified that the file regenerates cleanly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164, T12397

Differential Revision: https://secure.phabricator.com/D19568
2018-08-09 13:33:23 -07:00
epriestley
df31405d64 Improve compatibility of "Config > Cache Status" across APCu versions
Summary:
Ref T13164. See PHI790. Older versions of APCu reported cache keys as "key" from `apcu_cache_info()`. APC and newer APCu report it as "info".

Check both indexes for compatibility.

Test Plan:
  - Locally, with newer APCu, saw no behavioral change.
  - Will double check on `admin`, which has an older APCu with the "key" behavior.
  - (I hunted this down by dumping `apcu_cache_info()` on `admin` to see what was going on.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19569
2018-08-08 15:07:03 -07:00
epriestley
3ca3f09a1a Add an "--as" flag to "bin/conduit call ..." to improve flexibility and ease of profiling
Summary:
Ref T13164. In PHI801, an install reported a particular slow Conduit method call.

Conduit calls aren't easily profilable with normal tools (for example, `arc call-conduit --xprofile ...` gives you a profile of the //client//). They can be profiled most easily with `bin/conduit call ... --xprofile`.

However, `bin/conduit call` currently doesn't let you pick a user to execute the command on behalf of, so it's not terribly useful for profiling `*.edit`-style methods which do a write: these need a real acting user.

Test Plan:
Ran `bin/conduit call --method user.whoami --as epriestley ...` with valid, invalid, and no acting users.

```
$ echo '{}' | ./bin/conduit call --method user.whoami --as epriestley --input -
Reading input from stdin...
{
  "result": {
    "phid": "PHID-USER-icyixzkx3f4ttv67avbn",
    "userName": "epriestley",
    "realName": "Evan Priestley",
...
```

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19566
2018-08-08 09:51:21 -07:00
epriestley
91abc0f027 Stop indexing the chunk data objects for large Files stored in multiple chunks
Summary:
Ref T13164. See PHI766. Currently, when file data is stored in small chunks, we submit each chunk to the indexing engine.

However, chunks are never surfaced directly and can never be found via any search/query, so this work is pointless. Just skip it.

(It would be nice to do this a little more formally on `IndexableInterface` or similar as `isThisAnIndexableObject()`, but we'd have to add like a million empty "yes, index this always" methods to do that, and it seems unlikely that we'll end up with too many other objects like these.)

Test Plan:
  - Ran `bin/harbormaster rebuild-log --id ... --force` before and after change, saw about 200 fewer queries after the change.
  - Uploaded a uniquely named file and searched for it to make sure I didn't break that.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19563
2018-08-03 14:36:12 -07:00
epriestley
5839a54b60 Raise a tailored error when calling "transaction.search" with empty "phids" constraint
Summary:
Ref T13164. See PHI725. For real "*.search" methods, parameters get validated and you get an error if you use an empty list as a constraint.

Since "transaction.search" isn't really a normal "*.search" method, it doesn't benefit from this. Just do the check manually for now.

Test Plan: Made `transaction.search` calls with no constraints (got results); a valid costraint (got fewer results); and an invalid empty constraint (got an exception).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19562
2018-08-03 14:29:36 -07:00
epriestley
f3fa164882 Add a "Last Edited" property to Wiki pages
Summary:
Ref T13164. See PHI797. The last edit is available in the page header, but it's not precise (just says "180 days ago") and a little weird (it's unusual for us to put that kind of information in the header).

Add a precise timestamp to the footer for now. I'd imagine re-examining this the next time Phriction gets some UI work and maybe trying to integrate timeline/transactions more cleanly (see also T1894).

Test Plan: Looked at a wiki page, then edited it. Saw precise "Last Edit" timestamp adjacent to "Last Author".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19560
2018-08-03 14:29:17 -07:00
epriestley
3574a55a95 Deprecate Conduit method "diffusion.getrecentcommitsbypath"
Summary:
See D19558. This method has no callers and just wraps `diffusion.historyquery`, since D5960 (2013).

This was introduced in D315 (which didn't make it out of FB, I think) inside Facebook for unclear purposes in 2011.

Test Plan: Grepped for callers, found none.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: artms

Differential Revision: https://secure.phabricator.com/D19559
2018-08-03 09:48:58 -07:00
Arturas Moskvinas
356b2781bc Gracefully fail request if non existing callsign is passed to getrecentcommitsbypath instead of crashing
Summary:
`diffusion.getrecentcommitsbypath` fails with 500 error when non existing callsign is passed:
```
>>> UNRECOVERABLE FATAL ERROR <<<

Call to a member function getCommit() on null

```

Expected Behavior:
Return more graceful error notifying caller that such callsign/repository does not exist

Reproduction steps:
Open conduit: https://secure.phabricator.com/conduit/method/diffusion.getrecentcommitsbypath/
Enter:
callsign: "obviouslynotexisting"
path: "/random"
Click call method

Test Plan: after applying patch - call no longer fails with 500s

Reviewers: Pawka, epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19558
2018-08-02 19:49:10 +03:00
epriestley
e72296f927 Support querying Herald rules by monogram in typeahead datsources
Summary:
Depends on D19556. See PHI765. Ref T13164. Currently, if you type `H1` in this datasource, it isn't smart enough to pull up the right object.

Add support for querying by monogram. This is similar to existing support in Owners packages, etc.

Test Plan: Typed `H1` in the new push log filter, got the right object as a result in the typeahead.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19557
2018-08-01 17:52:27 -07:00
epriestley
06380e8079 Allow push events to be filtered by which Herald rule blocked the push
Summary: Depends on D19555. Ref T13164. See PHI765. An install is interested in getting a sense of the impact of a particular blocking rule, which seems reasonable. Support filtering for pushes blocked by a particular rule or set of rules.

Test Plan: {F5776385}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19556
2018-08-01 17:38:12 -07:00
epriestley
d8834377be When a Herald rule blocks a push, show which rule fired in the push log UI
Summary:
Ref T13164. See PHI765. We currently show "Rejected: Herald" in the push log UI, but don't show which rule rejected a push.

We store this data, and it's potentially useful: either for hunting down a particular issue, or for getting a general sense of how often a reject rule is triggering (maybe because you want to tune how aggressive it is).

Show this data in the web UI, and include it in the data export payload.

Test Plan:
  - Pushed to a hosted repository so that I got blocked by a Herald rule.
  - Viewed the push logs in the web UI, now saw which rule triggered things.
  - Exported logs to CSV, saw Herald rule PHIDs in the data.

{F5776211}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19555
2018-08-01 17:33:50 -07:00
epriestley
96e3c73159 Put "Subprojects" on top of "Milestones" in the Project UI
Summary:
Depends on D19550. Ref T13164. See T12144#226172, mostly. We get some requests to make milestones reorderable, but in most cases users probably wanted subprojects, not milestones.

One reason to end up here is that we put "Milestones" on top. Instead, put "Subprojects" on top, since they're the less specialized option and we aren't terribly consistent about it anyway.

Test Plan: Viewed project subprojects page, saw "Subprojects" above "Milestones".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19551
2018-08-01 13:49:42 -07:00
epriestley
45babe82f3 Add Spaces information to the project list UI
Summary: Depends on D19552. Ref T13164. We need this little `setObject(...)` hook to get the Space name into the search list UI.

Test Plan: Viewed project list, saw some Spaces listed.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19554
2018-07-31 10:24:51 -07:00
epriestley
8d8086fccf Add Spaces support to Phriction
Summary:
Ref T13164. See PHI774. Fixes T12435.

Since Phriction is hierarchical, there isn't a super strong motivation to support Spaces: you can generally set policies on a small number of documents to get the desired effective policy behavior.

However, it still improves consistency and there's no reason //not// to support Spaces. In the case where you have some moderately weird/complex policy on one or more Spaces, using Spaces to define the policy behavior can make things a bit simpler and easier to understand.

This probably doesn't actually fix whatever the root problem in T12435 was (complicated, non-hierarchical access policies?). See also a bunch of discussion in T12442. So we might end up going beyond this to address other use cases, but I think this is reasonable regardless.

Test Plan: Created and edited Phriction documents and shifted them between Spaces. Searched by Space, etc.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13164, T12435

Differential Revision: https://secure.phabricator.com/D19553
2018-07-31 10:24:28 -07:00
epriestley
d9b5b04950 Improve Space behavior for subprojects and milestones
Summary:
Depends on D19549. Ref T13164. See PHI774.

  - Make milestones inherit their parent project's space automatically, like they inherit their parent policies.
  - Make subprojects default to their parent project's space.

Test Plan: Created subprojects and milestones, got sensible default/effective Space behavior.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19550
2018-07-31 10:22:39 -07:00
epriestley
13cac5c362 Add Spaces to Projects
Summary:
See PHI774. Ref T13164. There is no reason projects //don't// support Spaces, just a vague concern that it's not hugely useful and might be a bit confusing.

However, it's at least somewhat useful (to improve consistency and reduce special casing) and doesn't necessarily seem more confusing than Projects are anyway. Support is trivial from a technical point of view, so just hook it up.

Test Plan: Created new projects, shifted projects between spaces. The support is all pretty much automatic.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19549
2018-07-31 10:15:41 -07:00
epriestley
9cf3b3bbf8 Count lines in build log slices more cheaply
Summary:
See PHI766. Ref T13164. Build log chunk processing does a `preg_split()` on slices, but this isn't terribly efficient.

We can get the same count more cheaply by just using `substr_count()` a few times.

(I also tried `preg_match_all()`, which was between the two in speed.)

Test Plan:
- Used `bin/harbormaster rebuild-log --id X --force` to rebuild logs. Verified that the linemap is identical before/after this change.
- Saw local time for the 18MB log in PHI766 drop from ~1.7s to ~900ms, and `preg_split()` drop out of the profiler (we're now spending the biggest chunk of time on `gzdeflate()`).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19545
2018-07-30 08:25:17 -07:00
epriestley
690a460c8e Allow mailers to be explicitly marked as inbound or outbound
Summary:
See PHI785. Ref T13164. In this case, an install wants to receive mail via Mailgun, but not configure it (DKIM + SPF) for outbound mail.

Allow individual mailers to be marked as not supporting inbound or outbound mail.

Test Plan:
  - Added and ran unit tests.
  - Went through some mail pathways locally, but I don't have every inbound/outbound configured so this isn't totally conclusive.
  - Hit `bin/mail send-test` with a no-outbound mailer.
  - I'll hold this until after the release cut so it can soak on `secure` for a bit.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19546
2018-07-30 08:25:06 -07:00
epriestley
9e451879d9 Add a "Changes Since Last Action" view to Differential revisions
Summary:
Ref T13151. See PHI616. Fixes T8163.

This adds `/D123/new/`, which shows the changes to the revision since the last timeline action you took.

It also adds a link to this view to diff update emails.

Test Plan:
  - Followed this link with a recent comment and no touches since update, ended up with sensible diff selections.
  - Updated revision, generated email, saw an appropriate link.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151, T8163

Differential Revision: https://secure.phabricator.com/D19541
2018-07-27 12:27:33 -07:00
epriestley
a5d3aea67c Carry the "silent" transaction flag through inverse edge edits
Summary:
See PHI751. Ref T13164. We added a "silent" flag for Editors somewhat recently (currently reachable only for bulk edits with `bin/bulk ...` command).

However, this flag doesn't carry through to the sub-editor when we make inverse edge edits. These are edits like "X is a parent of Y", which cause an implicit "Y is a child of X" edit to occur.

Pass the flag through.

Test Plan:
  - Rigged the relationships controller to make silent edits.
  - Changed the parents of a revision from the web UI. Saw no mail or feed stories.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19543
2018-07-27 12:27:16 -07:00
epriestley
cb99396c64 Make the "Is this JSON?" DocumentEngine heuristic a little tighter
Summary:
See PHI749. Ref T13164. We currently misdetect files starting with `[submodule ...` as JSON.

Make this a bit stricter:

  - If the file is short, just see if it's actually literally real JSON.
  - If the file is long, give up.

This should get the right result in pretty much all the cases people care about, I think. We could make the long-file guesser better some day.

Test Plan: Detected a `[submodule ...` file (no longer JSON) and a `{"duck": "quack"}` file (still JSON).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19544
2018-07-27 12:27:02 -07:00
epriestley
727bc2234c Capitalize "OPcache" more consistently
Summary: Fixes T13174. PHP spells this "OPcache" (lowercase "c"); we're inconsistent. Be more consistent.

Test Plan:
  - `git grep OPCache`
  - `git grep -i opcache | grep -v opcache | grep -v OPcache`

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13174

Differential Revision: https://secure.phabricator.com/D19538
2018-07-26 12:15:57 -07:00
epriestley
9e1a1577c3 Make the meme cache case-sensitive
Summary:
Fixes T13172. At one point we always capitalized all the text, and the cache uses capitalized text.

However, we stopped capitalizing the text at some point. Modern memes are more more subtle than old memes, and when we eventualy add support for things like "explodey brain" we'll certainly want to support mixed case.

Practically, this stops you from changing the capitalization of a cached meme. Get rid of the cache transform.

Test Plan:
none lul

(I don't have `gd` installed locally and buiding it requires building libjpeg and libpng or giving up and using `brew`. I'l vet this in production.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13172

Differential Revision: https://secure.phabricator.com/D19537
2018-07-26 12:15:32 -07:00
epriestley
682c3bc9ee When migrating files between storage engines with "bin/files migrate ...", skip expired temporary files
Summary:
See T7148. This just cheats us out of a weird sort of race where we:

  - Dump an instance, including some `F123` which is a temporary file which expires in 3 minutes.
  - A few minutes later, the daemons delete the data for that file.
  - A few minutes after that, we try to `bin/files migrate --copy` to copy the data from S3 into the MySQL blob store.
  - This fails since the data is already gone.

Instead, just skip these files since they're already dead to us.

Test Plan: Faked this locally, will migrate the PHI769 instance on `aux001`.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19536
2018-07-26 06:22:23 -07:00
Kenneth Endfinger
6bdd74584e
Fix file encoding migration
Summary:
See [[ https://discourse.phabricator-community.org/t/file-encryption-corruption-when-trying-to-encode-existing-files/1605 | Discourse ]]

When migrating to aes-256-cbc, integrity hashes were not updated, so data was not properly

Test Plan:
I ran [[ https://gist.github.com/kaendfinger/3e0d78350af0ebe4e74b2c8a79707bae | this test script ]] to ensure it worked.
I created some files with lipsum, ensured that after encoding them with aes-256-cbc, they were not able to be cat'd.
After applying this patch and rerunning the script, it worked successfully.

Reviewers: epriestley, amckinley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Tags: #files, #storage

Differential Revision: https://secure.phabricator.com/D19533
2018-07-23 17:41:10 -05:00
Kenneth Endfinger
6b6e1f0ba8
Fix Lipsum generators for Differential Revisions and Pastes
Summary:
When generating test data to solve a bug I have encountered, I noticed Lipsum was not working correctly for Differential Revisions and Pastes.

It seemed like they weren't updated after some refactoring. This fixes that by updating them.

Test Plan: Run Lipsum for all objects, and note that it has much less failure.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D19534
2018-07-23 15:05:51 -05:00
epriestley
77d7bb7af0 Document the Ferret "=" operator and improve related documentation
Summary:
Depends on D19529. See PHI778.

  - Document the "name" constraint as deprecated. All callers are likely better served by the "query" constraint.
  - Guide users toward the "query" constraint a little better.
  - Document the `=` syntax.

Test Plan: Read various new documentation.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19531
2018-07-23 12:44:43 -07:00
epriestley
71d4fa41c9 Support the Ferret "=" (exact match) operator in the actual query engine
Summary:
Ref PHI778. In D18492, I added support for parsing this operator, but did not actually implement it in the query engine.

Implementation is fairly straightforward. This supports querying for objects by exact title with `title:="exact title"`. This is probably a bad idea, but sometimes maybe useful anyway.

Test Plan: Queried for `title:="xxx"`, found only exact matches.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: ahoffer2

Differential Revision: https://secure.phabricator.com/D19529
2018-07-23 12:44:00 -07:00
epriestley
dee453c94d Give Config the "" (SPARKLE LIKE NEW) emoji instead of "☺" (STUPID LOOKING FACE)
Summary: Fixes T13171. Open to suggestions but that face looks real, real dumb on High Sierra.

Test Plan: Visited Config, saw a serious professional emoji in the page title.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13171

Differential Revision: https://secure.phabricator.com/D19530
2018-07-23 12:43:25 -07:00
epriestley
dce6dd5d02 Add an explicit "null" to a missed diffusion.branchquery callsite to fix Diffusion "Branches" page
Summary:
See PHI775. See D19499. Originally, see PHI720.

D19499 broke the standalone "Branches" page for commits. Normally, you reach this by taking these steps:

  - View a commit which is contained by 11 or more branches.
  - Click the "More Branches..." link in the "Branches" field.
  - You should be taken to a list of all branches which contain the commit.

The change to the 'branch' parameter was adjusted in the query that builds the "x, y, z, More Branches..." list, but not on the actual "Branches" list with the full list. Adjust it.

Test Plan:
  - Set display limit to 1, viewed a commit on "master" and "stable", clicked "More Branches".
  - Before: saw only "master".
  - After: saw both "master" and "stable".

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19532
2018-07-23 11:21:11 -07:00
epriestley
eb80a5ede1 Make the Conduit auth error for an unrecognized public key a little more useful
Summary: Ref T13168. This is just a small quality-of-life fix: we can disclose which public key we're talking about because public keys are public.

Test Plan:
  - Hit public key error (through my own bumbling / not reading or following instructions). Specifically, I haven't associated the key with a device in Almanac.
  - Before: vague error.
  - After: more specific error with enough key material that I could grep for it.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim

Maniphest Tasks: T13168

Differential Revision: https://secure.phabricator.com/D19516
2018-07-20 09:43:54 -07:00
Austin McKinley
67283c7a45 Add test plan to differential.revision.search
Summary: Ref T13151. Ref PHI622.

Test Plan: Loaded a revision in the Conduit UI; observed presence of `testPlan` field.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19518
2018-07-20 08:40:27 -07:00
epriestley
185c28f307 Update parent/child revision timeline messages to use modern language ("parent revision")
Summary:
See PHI746. See also T11833, perhaps. Ref T13151.

Long ago, parent revisions were called "dependent revisions". This was changed to "parent revisions" in the action UI to improve clarity, but not changed in the timeline stories.

Update the timeline stories to use the same language the actions in the UI use.

Test Plan:
{F5732876}

{F5732877}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19514
2018-07-13 09:02:10 -07:00
epriestley
4214b56a4f Make the dashboard panel datasource work properly with hundreds of panels
Summary:
Ref T13151. See PHI727. Update the dashboard widget/panel datasource to actually query results using what the user typed.

The current approach is blind to what the user typed when pulling results from the database, and gets limited to an artificially small number of results somewhere in the pipeline.

Test Plan:
  - Queried for panels with text queries.
  - Queried for panels with `W123` queries.
  - This is substantially similar to the Owners datasource, which received a similar update in D17142 and has worked well since then.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19511
2018-06-28 08:54:29 -07:00
epriestley
a94528ee4a Expose Differential actions for "transaction.search" in a basic way
Summary:
See PHI725. Ref T13151. These actions are somewhat unusual and I considered different ways to represent them (make them look like "status" transactions; build multiple synthetic transactions) but ultimately landed on the simplest approach of just exposing them more or less as they exist internally.

I haven't included data for any of them. Most don't really have any data, but "accept" does. I'm holding off on providing more data until after T731, which may shake up the internal format.

Test Plan: Applied most of these transactions against a revision, queried for it with `transaction.search`, got distinguishable transactions out.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19509
2018-06-28 08:51:55 -07:00
epriestley
f94cee8628 Fix querying for transactions over "transaction.search" when the object does not support comments
Summary: See PHI725. Ref T13151. We currently try to load comments unconditionally, but not all objects (like projects) have comments. Only try to load comments if an object actually has comments.

Test Plan: Queried for an object with no comments, like project `#masonry`, via `transaction.search`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19507
2018-06-26 07:59:01 -07:00
epriestley
cac3dc4983 Give "create" transactions a readable type in "transaction.search"
Summary:
Ref T13151. See PHI725. By default, "transaction.search" doesn't provide details about transactions because many have bad/weird/policy-violating internal types or fields.

The "create" transaction is simple and straightforward, so label it to allow callers to distinguish it.

Test Plan:
  - Created a new task.
  - Called `transaction.search` on it.
  - Saw the labelled "create" transaction.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: swisspol

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19505
2018-06-22 17:42:35 -07:00
epriestley
d84f866ca0 When search indexers contend for a lock, just yield
Summary:
Depends on D19503. Ref T13151. See PHI719. If you have something like a script which updates an object in a loop, we can end up queueing many search reindex tasks.

These tasks may reasonably contend for the lock, especially if the object is larger (lots of text and/or lots of comments) and indexing takes a few seconds.

This isn't concerning, and the indexers should converge to good behavior quickly once the updates stop.

Today, they'll spew a bunch of serious-looking lock exceptions into the log. Instead, just yield so it's more clear that there's (normally) no cause for concern here.

Test Plan: Ran `bin/search index Txxx --force` on a large object in multiple windows with a 0 second lock, saw an explicit yield instead of a lock exception.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19504
2018-06-22 17:41:45 -07:00
epriestley
14e911a0d8 Index only the first 1,000 comments on any object
Summary:
Depends on D19502. Ref T13151. See PHI719. An install ended up with an object with 111,000+ comments on it because someone wrote a script to treat it like a logfile.

Although we seem to do mostly okay with this (locally, it only takes about 30s to index a similar object) we'll hit a wall somewhere (since we need to hold everything in memory), and it's hard to imagine a legitimate object with more than 1,000 comments. Just ignore comments past the first thousand.

(Conpherence threads may legitimately have more than 1,000 comments, but go through a different indexer.)

Test Plan:
  - Piped some comments into `maniphest.edit` in a loop to create a task with 100K comments.
  - Ran `bin/search index Txxx --force` to reindex it, with `--trace`.
    - Before: task indexed in about 30s.
    - After: script loaded comments with LIMIT 1000 and indexed in a couple seconds.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19503
2018-06-22 17:41:05 -07:00
epriestley
cbc71e75fa When queueing search index tasks, include the "objectPHID" in the task metadata
Summary:
Ref T13151. See PHI719. One minor hiccup in debugging the issue (which ended up being "revision has 100K comments") was that the `SearchWorker` did not show which object it was indexing.

Add `'objectPHID'` to the queue call so you can see which object is affected from the web UI.

Test Plan:
  - Stopped daemons.
  - Used `bin/search index D123 --background` to queue a search task.
  - Viewed task details in web UI from `/daemon/`.
    - Before change: no indication of which object was being indexed.
    - After change: page helpfully shows that the task is indexing D123.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19502
2018-06-22 17:40:32 -07:00
epriestley
b1f4a974fe Fix two minor breadcrumb issues in Config
Summary:
Fixes T13159. Two issues here:

  - When viewing a particular config setting, there's an extra "Config" crumb.
  - On the page for a config group, the link to the parent group has an extra "/config/" in it.

Test Plan:
  - Viewed a page for a particular setting, no longer saw an extra "Config" crumb.
  - Viewed a page for a setting group, clicked parent crumb, got taken to a real page.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13159

Differential Revision: https://secure.phabricator.com/D19501
2018-06-22 17:38:50 -07:00
epriestley
8ab8c390b7 If "branch" is provided to "diffusion.branchquery", use it as the "<pattern>" argument to "git branch --contains ..."
Summary:
Ref T13151. See PHI720. If you want to test if commit X appears on specific branch Y, `git branch --contains X -- Y` is faster than (effectively) `git branch --contains X | grep Y`.

Since this call has a "branch" parameter anyway, use it as the pattern argument if provided.

Test Plan:
  - Called the API method with no parameters, got all branches.
  - Called the API method with `master`, got just master.
  - Called the API method with `maste*`, got master. This behavior is not officially supported and may change in the future.
  - Viewed a commit, still saw all branches.
    - Grepped for `diffusion.branchquery` and verified that no remaining callsites pass a default "branch" parameter.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19499
2018-06-22 17:38:19 -07:00
epriestley
6136b83275 Fix changeset construction special case for empty commits in pre-commit hooks
Summary: Fixes T13155. Ref T13151. A recent change (D19455) changed the return format here, but I missed this special case for empty commits.

Test Plan:
  - T13155 has a good set of reproduction instructions.
  - Pushed an empty commit.
    - Before: bunch of warning log spew.
    - After: clean logs.

Reviewers: amckinley, avivey

Reviewed By: avivey

Maniphest Tasks: T13155, T13151

Differential Revision: https://secure.phabricator.com/D19500
2018-06-21 16:43:20 -07:00
Austin McKinley
9db5ad3476 Allow null identities to be attached to commit objects
Summary: I landed D19491 a little aggressively, so allow this field to be null until after the migration goes out.

Test Plan: Loaded commits without identity objects; did not get any errors.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19496
2018-06-20 08:35:36 -07:00
Austin McKinley
05f333dfba Attach identities to commits and users to identities
Summary: Ref T12164. Make it easier to work with identity objects by attaching them to commits and attaching users to identities.

Test Plan: Loaded some commits with `->needIdentities(true)` and checked the resulting objects.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19491
2018-06-18 15:31:41 -07:00
Austin McKinley
787c59744b Correctly attach users to identities
Summary: This never worked.

Test Plan: Ran `bin/repository rebuild-identities` and viewed identity objects with `currentEffectiveUserID`s and no longer got errors about attempting to attach `null` objects instead of `PhabricatorUser` objects.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19495
2018-06-18 15:21:11 -07:00
epriestley
a7c681b549 Don't set mail HTML bodies if there's no actual HTML body
Summary:
See <https://discourse.phabricator-community.org/t/commit-6011085b0fcd-breaks-sending-certain-email/1571>. Some mailers get upset if we `setHTMLBody(...)` with an empty string.

There's some possible argument they should be more graceful about this, but it's reasonably pretty ambiguous.

Only try to set the HTML body if we actually have a nonempty HTML body.

Test Plan:
  - Configured an "smtp" mailer.
  - Ran `echo hi | ./bin/mail send-test --to someone@somewhere.com --subject test`.
  - Before: error about empty message body.
  - After: no more message body error.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19494
2018-06-15 14:01:40 -07:00
epriestley
1459fb3037 Make re-running rebuild-identities a bit faster and add a little progress information
Summary:
Ref T13151. Ref T12164. Two small tweaks:

  - If we aren't actually going to change anything, just skip the writes. This makes re-running/resuming a lot faster (~20x, locally).
  - Print when we touch a commit so there's some kind of visible status.

This is just a small quality-of-life tweak that I wrote anyway while investigating T13152, and will make finishing off db024, db025 and db010 manually a little easier.

Test Plan:
  - Set `authorIdentityPHID` + `committerIdentityPHID` to `NULL`.
  - Ran `rebuild-identities`, saw status information.
  - Ran `rebuild-identiites` again, saw it go faster with status information.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151, T12164

Differential Revision: https://secure.phabricator.com/D19484
2018-06-12 13:18:54 -07:00
epriestley
6011085b0f Respect "metamta.email-body-limit" when building mail HTML bodies
Summary:
Ref T13151. See T11767. See PHI686. Although we limit outbound mail text bodies, the limit doesn't currently apply to attachments, HTML bodies, or headers. T11767 discusses improving this in the general case.

In the wild, an install hit an issue (see PHI686) where edits to Phriction pages generate very large HTML bodies. Check and respect the limit when building HTML bodies.

If we don't have enough room for the HTML body, we just drop it. We have the text body to fall back to, and HTML is difficult to truncate safely.

Test Plan: Added unit tests and made them pass.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19489
2018-06-12 12:02:15 -07:00
epriestley
c5b13a6be3 Allow object subtypes to be changed via bulk editor
Summary:
Ref T13151. See PHI683. Ref T12314.

You can currently change object subtypes via Conduit (`maniphest.edit`) but not via the web UI.

Changing object subtypes is inherently a somewhat-perilous operation that likely has a lot of rough edges we'll need to smooth over eventually, mostly around changing an object from subtype X to subtype Y, where some field exists on one but not the other. This isn't a huge issue, just not entirely intuitive.

It should also, in theory, be fairly rare.

As a reasonable middle ground, provide web UI access via the bulk editor. This makes it possible, but doesn't clutter the UI up with a rarely-used option with rough edges.

Test Plan:
  - With subtypes not configured, saw a normal bulk editor with no new option.
  - With subtypes configured, swapped tasks subtypes via bulk editor.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151, T12314

Differential Revision: https://secure.phabricator.com/D19490
2018-06-12 11:58:44 -07:00
epriestley
62a402491a Allow encrypted mail to be more specific about which object is affected
Summary:
Depends on D19487. Ref T13151. See PHI647. For some objects, like revisions, we can build slightly more useful secure email without actually disclosing anything.

In the general case, the object monogram may disclose information (`#acquire-competitor`) but most do not, so applications can whitelist an acceptable nondisclosing subject and link.

Support doing this, and make Differential do it. When we don't have a whitelisted URI but do know the object the mail is about, include a generic PHID-based URI; these are always nondisclosing.

Test Plan:
  - Without the Differential changes, sent normal mail (no changes) and secure mail (new generic PHID-based link).
  - With the Differential changes, sent secure mail; got richer subject and body link.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19488
2018-06-12 11:55:18 -07:00
epriestley
94752278f4 Add a generic PHID-based object redirection controller
Summary:
Ref T13151. See PHI647. This allows us to link to any object by PHID, without disclosing information in the monogram (like `#fire-steve`).

This capability is relevant when building "secure mail", to provide a link to the object regardless of whether the monogram discloses information or not.

Test Plan: Visited `/object/D123/` (redirect), `/object/xyz/` (404), `/object/PHID-DREV-.../` (redirect).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19487
2018-06-12 11:54:59 -07:00
epriestley
cbff913432 Add a "members of all projects" (vs "...any project") custom policy rule to the upstream
Summary:
Ref T13151. See PHI702. An install is interested in a "members of all projects" (vs "members of any project", which is currently implemented) rule.

Although this is fairly niche, I think it's reasonable and doesn't have much of a maintenance cost.

This could already be implemented as an extension, but it would have to copy/paste a bunch of code.

Test Plan:
  - Ran unit tests.
  - Used the UI to select this policy for a task, with various values. Joined/left projects to satisfy/fail the rule. Behavior seemed correct.
  - Used the UI to select the existing policy rule ("any project"), joined/left projects to satisfy/fail the rule. Doesn't look like I broke anything.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19486
2018-06-12 11:51:51 -07:00
Alex Vandiver
59b95f9397 Fix typo in "button"
Test Plan: Observation.

Reviewers: #blessed_reviewers, amckinley, epriestley

Reviewed By: #blessed_reviewers, amckinley, epriestley

Subscribers: Korvin, amckinley, epriestley

Differential Revision: https://secure.phabricator.com/D19483
2018-06-08 15:09:07 -07:00
epriestley
f375427177 Use more consistent diff coloration in unified diffs
Summary:
Ref T13151. See PHI701. Unified diffs are currently missing the logic to apply the "old-full" and "new-full" classes, which results in a too-light coloration for fully added or removed lines.

Make this logic consistent with the two-up renderer so we use the same colors in both.

Test Plan: Viewed diffs and swapped between 1-up and 2-up renderers, now saw the same coloration.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19482
2018-06-08 09:39:34 -07:00
epriestley
7acda7e94e Truncate package names in diff table of contents views
Summary:
Ref T13151. See PHI654. Depends on D19477. If you have long package names, the table of contents (e.g., in Differential) can end up expanding to be gigantic.

Getting tables to behave nicely is hard (or, at least, I can't figure it out after spending a decent amount of time on it; see also `AphrontTableView::renderSingleDisplayLine()`). I tried a bunch of things and Googled for a bit but didn't make any progress on finding a CSS solution. Just truncate the package names to get reasonable behavior without falling down any kind of CSS rabbit hole.

Test Plan:
  - Created a package named "Very long package name...".
  - Created a package named "MMMMMMMMMMMMMMMMMMMMMM...".
  - Had them own a file in a Differential revision, viewed that revision.
  - Before: table is pushed out to several times the browser window width and everything is kind of a mess.
  - After: package names get truncated to something reasonable.

{F5652953}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19478
2018-06-07 13:17:01 -07:00
epriestley
2951e0c86b Include owners packages in the MailableFunction datasource
Summary:
Ref T13151. See PHI684. Currently, the `MailableFunction` datasource does not include Owners packages, but they are valid subscribers and the `Mailable` datasource includes them.

Include them in the `MailableFunction` datasource, too.

Test Plan: Searched for revisions with particular package subscribers, got expected results in the UI (tokenizer knew about packages) and response.

Reviewers: amckinley, jmeador

Reviewed By: jmeador

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19476
2018-06-07 12:02:50 -07:00
Austin McKinley
b8b2d1672d Prevent creation of empty repository identities
Summary: Fixes issue reported in https://secure.phabricator.com/rPf191a66490b194785fae28c062b71be99bb14584#43240

Test Plan: Imported an SVN repo, observed clean import instead of daemon exception.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19466
2018-06-05 16:13:59 -07:00
Aviv Eyal
dbe72df557 minor: fix translation error in exception
Test Plan: look hard at code.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D19463
2018-06-04 17:46:13 +00:00
epriestley
376ea1ddf5 Support logged-out access to more Harbormaster controllers
Summary:
Fixes T13145. The list controllers properly support public access already, but some of the view/detail controllers did not.

Allow logged-out users to browse builds, buildables, plans, etc., provided they can see the corresponding objects.

Test Plan: As a logged-out user, browsed around builds, build plans, logs, etc., without hitting any login pages.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13145

Differential Revision: https://secure.phabricator.com/D19459
2018-06-04 10:11:25 -07:00
epriestley
5bcca675e1 Add card expiration information to Phortune cart screen
Summary:
See PHI689. It can be difficult to distinguish between cards with the same number but different expiration dates (common when the bank sends you a new card).

For now, show the expiration date on the cart checkout screen.

Test Plan: Viewed a cart checkout screen with multiple cards, saw expiration dates.

Reviewers: amckinley

Differential Revision: https://secure.phabricator.com/D19462
2018-06-02 18:23:44 -07:00
Austin McKinley
2f6784ee1c Add workflow to create repository identities
Summary:
Depends on D19443. Creates a workflow for populating the new identity table by iterating over commits, either one repo at a time or all at once. Locally caches identities to avoid fetching them `inf` times. An actual migration that invokes this workflow will come in another revision that won't land until at least next week.

Performance is ~2k commits in 4.9s on my local machine.

Test Plan: Ran locally a few times with a few different states of the `repository_identity` table.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: jcox, Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19446
2018-05-31 07:29:57 -07:00
Austin McKinley
fe5fde5910 Assign RepositoryIdentity objects to commits
Summary: Depends on D19429. Depends on D19423. Ref T12164. This creates new columns `authorIdentityPHID` and `committerIdentityPHID` on commit objects and starts populating them. Also adds the ability to explicitly set an Identity's assignee to "unassigned()" to null out an incorrect auto-assign. Adds more search functionality to identities. Also creates a daemon task for handling users adding new email address and attempts to associate unclaimed identities.

Test Plan: Imported some repos, watched new columns get populated. Added a new email address for a previous commit, saw daemon job run and assign the identity to the new user. Searched for identities in various and sundry ways.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19443
2018-05-31 07:28:23 -07:00
Austin McKinley
f191a66490 Add controllers/search/edit engine functionality to RepositoryIdentity
Summary: Depends on D19423. Ref T12164. Adds controllers capable of listing and editing `PhabricatorRepositoryIdentity` objects. Starts creating those objects when commits are parsed.

Test Plan: Reparsed some revisions, observed objects getting created in the database. Altered some `Identity` objects using the controllers and observed effects in the database. No attempts made to validate behavior under "challenging" author/committer strings.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19429
2018-05-31 07:03:25 -07:00
Austin McKinley
cd84e53c44 Begin building out RepositoryIdentity indirection layer
Summary: Ref T12164. Start building initial objects for managing `RepositoryIdentity` objects. This won't land until much more of the infrastructure is in place.

Test Plan: Ran `bin/storage upgrade` and observed expected table.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19423
2018-05-31 07:01:16 -07:00
epriestley
de999af614 Improve some behaviors around memory pressure when pushing many and/or large changes
Summary:
Ref T13142. When commits are pushed, we try to handle them on one of two pathways:

  - Normal changes: we load these into memory and potentially apply Herald content rules to them.
  - "Enormous" changes: we don't load these into memory and skip content rules for them.

The goal is to degrade gracefully when users push huge changes: they should work, just not support all the features.

However, some changes can slip through the cracks right now:

  - If you push a lot of commits at once, we'll try to cache all of the changes smaller than 1GB in memory. This can require an arbitrarily large amount of RAM.
  - We calculate sizes by just looking at the `strlen()` of the diff, but a changeset takes more RAM in PHP than the raw diff does. So even if a diff is "only" 500MB, it can take much more memory than that. On systems with relatively little memory available, this may result in OOM while processing changes that are close to the "enormous" limit.

This change makes two improvements:

  - Instead of caching everything, cache only 64MB of things.
    - For most pushes, this is the same, since they have less than 64MB of diffs.
    - For pushes of single very large changes, this is a bit slower (more CPU) since we have to do some work twice.
    - For pushes of many changes, this is slower (more CPU) since we have to do some work twice, but, critically, doesn't require unlimited memory.
  - Instead of flagging changes as "enormous" at 1GB, flag them as "enormous" at 256MB.
    - This reduces how much memory is required to process the largest "non-enormous" changes.
    - This also gets us under Git's hard-coded 512MB "always binary" cutoff; see T13143.
    - This is still completely gigantic and way larger than any normal change should be.

An additional improvement would be to try to reduce the amount of memory we need to use to hold a change in process memory. I think the other changes here alone will fix the immediate issue in PHI657, but it would be nice if the "largest non-enormous change" required only a couple gigs of RAM.

Test Plan:
- Used `ini_set('memory_limit', '1G')` to artificially limit memory to 1GB.
- Pushed a series of two commits which add two 550MB text files (Temporarily, I added a `--binary` flag to trick Git into showing real diffs for these, see T13143.)
- Got a memory limit error.
- Applied the "cache only 64MB of stuff" and "consider 256MB, not 1GB, to be enormous" changes.
- Pushed again, got properly rejected as enormous.
- Added `memory_get_usage()` calls to measure how actual memory size and reported "size" estimate compare. For these changes, saw a 639MB diff require 31,479MB of memory, i.e. a factor of about 50x. This is, uh, pretty not great.
- Allowed enormous changes, pushed again, push went through.

Reviewers: amckinley

Maniphest Tasks: T13142

Differential Revision: https://secure.phabricator.com/D19455
2018-05-18 17:15:34 -07:00
epriestley
8f9b948447 When showing a diff-of-diffs, hide files which didn't get any more changes and have no inlines
Summary:
Ref T13137. See that task for discussion.

When we show a diff-of-diffs, we often render stubs for files which didn't change between the diffs. These stubs usually aren't a big deal, but for certain types of changes (like refactors) they can create a lot of clutter.

Instead, hide these stubs and show a notice that we hid them.

Test Plan:
  - Created a revision affecting 4 files.
  - Updated it with a diff that changed only 1 of the 4 files.
  - Added an inline comment to a different file.
  - Viewed the diff of diffs.
    - Before: 4 changesets with two "nothing changed" stubs.
    - After: 2 changesets with the stubs hidden.

{F5621083}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13137

Differential Revision: https://secure.phabricator.com/D19453
2018-05-16 17:18:53 -07:00
epriestley
79fdf5c127 Separate changeset analysis code from DifferentialDiff and provide a standalone rebuild-changesets workflow
Summary:
Ref T13137. The "analyze/cache data about changesets" step is becoming more involved. We recently added detection for generated code to support "Ignore generated changes" in Owners, and I now plan to hash the new file content so we can hide changes which have no effect.

Before adding this new hashing step, pull the "detect copied code" and "detect generated code" stuff out and move them to a separate `ChangesetEngine`. Then support doing a changeset rebuild directly with `bin/differential rebuild-changesets`.

This simplifies things a bit and makes testing easier since you don't need to keep creating new revisions to re-run copy/generated/hash logic.

Test Plan: Ran `bin/differential rebuild-changesets --revision Dxxx`, saw changesets rebuild. See also next change.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13137

Differential Revision: https://secure.phabricator.com/D19452
2018-05-16 17:17:28 -07:00
epriestley
3544620209 Parse unusual Subversion protocol frames which contain extra whitespace
Summary:
Fixes T13140. See PHI660.

Recent versions of Subversion can send a `(get-file true false  false )` protocol frame with extra space between "false" and "false". This is allowed by the protocol spec, but never normally happens, and we do not parse it correctly.

Instead, parse it correctly.

Test Plan:
  - Added unit tests.
  - Ran `svn proplist svn+ssh://.../diffusion/X/file.c` under SVN 1.10 before and after the change.
    - Before: indefinite hang.
    - After: completed in finite time.

Reviewers: amckinley, asherkin

Reviewed By: amckinley, asherkin

Maniphest Tasks: T13140

Differential Revision: https://secure.phabricator.com/D19451
2018-05-16 17:12:41 -07:00
epriestley
29df80b48f Fix a fatal during breadcrumb construction when viewing a dashboard you don't have permission to view
Summary: Ref PHI662. Viewing a dashboard you don't have permission to view (in the Dashboard application) currently fatals while building crumbs, since we fail to build the ` ... > Dashboard 123 > ...` crumb.

Test Plan:
  - Viewed a dashboard I didn't have permission to view in the Dashboards application.
  - Before patch, fatal when calling `getID()` on a non-object.
  - After patch, sensible policy error page.
  - Viewed a dashboard I can view, saw sensible crumbs.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19449
2018-05-14 12:06:56 -07:00
epriestley
28ee6b8080 Consistently require MFA on the actual user creation flow
Summary:
See <https://hackerone.com/reports/351361>. We currently require MFA on the screen leading into the user create flow, but not the actual create flow.

That is, `/people/create/` (which is just a "choose a type of account" page) requires MFA, but `/people/new/<type>/` does not, even though this is the actual creation page.

Requiring MFA to create users isn't especially critical: creating users isn't really a dangerous action. The major threat is probably just that an attacker can extend their access to an install by creating an account which they have credentials for.

It also isn't consistently enforced: you can invite users or approve users without an MFA check.

So there's an argument for just removing the check. However, I think the check is probably reasonable and that we'd likely prefer to add some more checks eventually (e.g., require MFA to approve or invite) since these actions are rare and could represent useful tools for an attacker even if they are not especially dangerous on their own. This is also the only way to create bot or mailing list accounts, so this check does //something// on its own, at least.

Test Plan:
  - Visited `/people/new/standard/` as an admin with MFA configured.
  - Before patch: no MFA prompt.
  - After patch: MFA prompt.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19448
2018-05-14 12:03:07 -07:00
epriestley
26d0862f4f Apply the new patch byte size limit to mail patch generation in Differential
Summary: Ref T13137. See PHI592. Depends on D19444. Apply a limit up front to stop patches which are way too big (e.g., 600MB of videos) from generating in the first place.

Test Plan:
  - Configured inline patches in git format.
  - Created a normal revision, got an inline git patch.
  - Created a revision with a 10MB video file, got no inline patch.
  - (Added a bunch of debugging stuff to make sure the internal pathway was working.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13137

Differential Revision: https://secure.phabricator.com/D19445
2018-05-14 09:10:47 -07:00
Aviv Eyal
7281300446 Allow number in generated clone uri
Summary:
See https://discourse.phabricator-community.org/t/numerical-characters-are-stripped-from-diffusion-git-repository-name-in-the-uri/

Digits are often considered reasonable characters.

Test Plan: Looked at an ascii table.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, Sam2304, epriestley

Differential Revision: https://secure.phabricator.com/D19447
2018-05-11 16:18:06 +00:00
epriestley
10a4b05ecb Fix "Any Owner" and "No Owners" searches in Maniphest
Summary:
See <https://discourse.phabricator-community.org/t/maniphest-home-page-crash-after-d19417/1445/3>. These special-token-only searches currently end up populating an empty `ownerPHIDs`, which fatals after the stricter check in D19417.

Make the fatal on `withConstraint(array())` explicit and only set the PHID constraint if we have some PHIDs left.

Test Plan: Searched for "No Owner", "Any Owner", an actual owner, "No Owner + actual user".

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19440
2018-05-09 13:24:23 -07:00
epriestley
d280b24239 Fix "arc paste" to stop creating pastes with an empty string ("") as the "language"
Summary:
See PHI652. When you `echo x | arc paste` today, you end up with a Paste object that has the empty string as its "language".

This is normally not valid. Pastes where the language should be autodetected should have the value `null`, not the empty string.

This behavior likely changed when `paste.create` got rewritten in terms of `paste.edit`. Adjust the implementation so it only adds the LANGUAGE transaction if there's an actual language.

Also, fix an issue where you can't use the "delete" key to delete tokens with the empty string as their value.

Test Plan:
  - Created a paste with `echo x | arc paste`, got a paste in autodetect mode instead of with a bogus language value.
  - Created a paste with `echo x | arc paste --lang rainbow`, got a rainbow paste.
  - Deleted an empty string token with the keyboard.
  - Deleted normal tokens with the keyboard.
  - Edited subscribers/etc normally with the keyboard and mouse to make sure I didn't ruin anything.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19437
2018-05-09 13:22:58 -07:00
epriestley
5b640a434c Support an "Ancestors Of: ..." constraint in commit queries
Summary:
Ref T13137. See PHI609. An install would like to filter audit requests on a particular branch, e.g. "master".

This is difficult in the general case because we can not apply this constraint efficiently under every conceivable data shape, but we can do a reasonable job in most practical cases.

See T13137#238822 for more detailed discussion on the approach here.

This is a bit rough, but should do the job for now.

Test Plan:
- Filtered commits by various branches, e.g. "master"; "lfs". Saw correct-seeming results.
- Stubbed out the "just list everything" path to hit the `diffusion.internal.ancestors` path, saw the same correct-seeming results.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13137

Differential Revision: https://secure.phabricator.com/D19431
2018-05-08 15:51:42 -07:00
epriestley
397645b273 Export task point values as double, not int
Summary:
See <https://discourse.phabricator-community.org/t/maniphest-non-integer-point-values-in-csv-export/1443>.

We currently export the Maniphest "points" field as an integer, but allow it to accept decimal values (e.g. "6.25").

Also fix a bug where we wouldn't roll over from "..., X, Y, Z, AA, AB, ..." correctly for Excel column names if sheet had more than 26 columns.

Test Plan:
  - Set a task point value to 6.25.
  - Exported to text, JSON, XLS.
  - Saw 6.25 represented accurately in exports.
  - Exported an excel sheet with 27+ columns.
  - Manually printed the first 200 column names to check that the algorithm looks correct.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19434
2018-05-08 15:49:40 -07:00
epriestley
304c6a4597 Improve UI and documentation for "Ignore Attributes" in Owners slightly
Summary:
See PHI251. Ref T13137.

  - Replace the perplexing text box with a checkbox that explains what it does.
  - Mention this feature in the documentation.

Test Plan:
  - Clicked/unclicked checkbox.
  - Read documentation.
  - Used an existing checkbox control in Slowvote to make sure I didn't break it.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13137

Differential Revision: https://secure.phabricator.com/D19433
2018-05-08 14:03:30 -07:00
epriestley
fddb506e98 Don't render the Maniphest edit form bottom-of-page preview panel if "Description" is locked or hidden
Summary:
See <https://discourse.phabricator-community.org/t/hidden-description-field-in-maniphest-task-breaks-form/1432>.

If you hide the "Description" field in Maniphest, we still try to render a remarkup preview for it. This causes a JS error and a nonfunctional element on the page.

Instead, hide the preview panel if the field has been locked or hidden.

Test Plan:
  - Hid the field, loaded the form, no more preview panel / JS error.
  - Used a normal form with the field visible, saw a normal preview.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19432
2018-05-08 14:01:23 -07:00
epriestley
a4a22dd2f8 Mention the "inline comments" rule in the callout for "Large" diffs
Summary:
See PHI638. When a diff is large (between 100 and 1000 files), we collapse content by default unless a change also has inline comments.

This rule isn't explicitly explained anywhere. Although it's not really a critical rule, it fits easily enough into the UI callout.

Also render the UI callout in a slightly more modern way and avoid `hsprintf()`.

Test Plan:
{F5596496}

  - Also, clicked the "Expand" link and saw everything expand properly.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19430
2018-05-07 10:38:58 -07:00
epriestley
4a98e0ff65 Allow Owners packages to be configured to ignore generated paths in Differential
Summary:
Depends on D19427. Ref T13130. See PHI251. Support configuring owners packages so they ignore generated paths.

This is still a little rough. A couple limitations:

  - It's hard to figure out how to use this control if you don't know what it's for, but we don't currently have a "CheckboxesEditField". I may add that soon.
  - The attribute ignore list doesn't apply to Diffusion, only Differential, which isn't obvious. I'll either try to make it work in Diffusion or note this somewhere.
  - No documentation yet (which could mitigate the other two issues a bit).

But the actual behavior seems to work fine.

Test Plan:
  - Set a package to ignore paths with the "generated" attribute. Saw the package stop matching generated paths in Differential.
  - Removed the attribute from the ignore list.
  - Tried to set invalid attributes, got sensible errors.
  - Queried a package with Conduit, got the ignored attribute list.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13130

Differential Revision: https://secure.phabricator.com/D19428
2018-05-05 08:47:29 -07:00
epriestley
dc510354c3 Remove explicit "mailKey" from Owners packages
Summary:
Depends on D19426. Ref T13130. Ref T13065. While I'm making changes to Owners for "Ignore generated paths", clean up the "mailKey" column.

We recently (D19399) added code to automatically generate and manage mail keys so we don't need a ton of `mailKey` properties in the future. Migrate existing mail keys and blow away the explicit column on packages.

Test Plan: Ran migration, manually looked at the database and saw sensible data. Edited a package to send some mail, which looked good.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13130, T13065

Differential Revision: https://secure.phabricator.com/D19427
2018-05-05 08:47:08 -07:00
epriestley
5e2af4b9b5 Prepare to support an "Ignore generated files" flag in Owners
Summary:
Depends on D19425. Ref T13130. See PHI251. Now that changesets have a durable "generated" attribute, we can let owners packages check it when we're computing which packages are affected by a revision.

There's no way to actualy configure a package to have this behavior yet.

Test Plan:
  - Created a revision affecting a generated file and a non-generated file.
    - When I faked `mustMatchUngeneratedPaths()` to `return true;`, saw the non-generated file get no packages owning it.
    - Normally: lots of packages owning it).
  - Created a revision affecting only generated files.
    - When I faked things, saw no Owners actions trigger.
    - Normally: some packages added reviewers or subscribers.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13130

Differential Revision: https://secure.phabricator.com/D19426
2018-05-05 08:46:47 -07:00
epriestley
af295341c8 Classify changesets as "generated" at creation time, in addition to display time
Summary:
Ref T13130. See PHI251. Currently, changesets are marked as "generated" (i.e., the file contains generated code and does not normally need to be reviewed) at display time.

An install would like support for having Owners rules ignore generated files. Additionally, future changes anticipate making "generated" and some other similar behaviors more flexible and more general.

To support these, move toward a world where:

  - Changesets have "attributes": today, generated. In the future, perhaps: third-party, highlight-as, encoding, enormous-text-file, etc.
  - Attributes are either "trusted" (usually: the server assigned the attribute) or "untrusted" (usually: the client assigned the attribute). For attributes like "highlight-as", this isn't relevant, but I'd like to provide tools so that you can't make `arc` mark every file as "generated" and sneak past review rules in the future.

Here, the `differential.generated-paths` config can mark a file as "generated" with a trusted attribute. The `@generated`-in-content rule can mark a file as "generated" with an untrusted attribute.

Putting these attributes on changesets at creation time instead of display time will let Owners interact with changesets cheaply: it won't have to render an entire changeset just to figure out if it's generated or not.

Test Plan:
  - Created a revision touching several files, some generated and some not.
  - Saw the generated files get marked properly with attribute metadata in the database, and show/fold as "Generated" in the UI.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13130

Differential Revision: https://secure.phabricator.com/D19425
2018-05-05 08:46:25 -07:00
epriestley
5784e3d3c0 Omit "type" attribute from "<source />" tags in "<video>" to trick Chrome into playing them
Summary:
Fixes T13135. See PHI633. For at least some video files with legitimate MIME type "video/quicktime", Chrome can play them but refuses to if the `<source />` tag has a `type="video/quicktime"` attribute.

To trick Chrome into giving these videos the old college try, omit the "type" attribute. Chrome then tries to play the video, seems to realize it can, and we're back on track.

Since the "type" attribute is theoretically only useful to help browsers select among multiple different alternatives and we're only presenting one alternative, this seems likely safe and reasonable. Omitting "type" also validates. It's hard to be certain that this won't cause any collateral damage, but intuitively it seems like it should be safe and I wasn't able to identify any problems.

Test Plan:
  - Watched a "video/quicktime" MP4 cat video in Chrome/Safari/Firefox.
  - See T13135 for discussion, context, and discussion of the behavior of some smaller reproduction cases.

Reviewers: amckinley, asherkin

Reviewed By: amckinley

Maniphest Tasks: T13135

Differential Revision: https://secure.phabricator.com/D19424
2018-05-04 09:28:47 -07:00
epriestley
332f4ab66d Restore support for using "arc download" to fetch files with no "security.alternate-file-domain"
Summary:
Fixes T13132. I removed this branch in D19156 when tightening the logic for the new CSP header, but there's a legitimate need for it: downloading files via `arc download`, or more generally being an API consumer of files.

This is not completely safe, but attacks I'm aware of (particularly, cookie fixation, where an attacker could potentially force a victim to become logged in to an account they control) are difficult and not very powerful. We already issue clear setup advice about the importance of configuring this option ("Phabricator is currently configured to serve user uploads directly from the same domain as other content. This is a security risk.") and I think there's significant value in letting API clients just GET file data without having to jump through a lot of weird hoops.

Test Plan:
  - With `security.alternate-file-domain` off, tried to `arc download` a file.
  - Before: downloaded an HTML dialog page.
  - After: downloaded the file.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13132

Differential Revision: https://secure.phabricator.com/D19421
2018-05-01 10:08:05 -07:00
epriestley
fb4b9bc2fc Fix an issue where entering the same Owners path for two repositories would incorrectly de-dupe the path
Summary:
Ref T13130. See <https://discourse.phabricator-community.org/t/unable-to-create-owners-package-with-same-path-in-multiple-repositories/1400/1>.

When you edit paths in Owners, we deduplicate similar paths, like `/x/y` and `/x/y/`. However, this logic currently only examines the paths, and incorrectly deduplicates the same path in different repositories.

Instead, consider the repository before deduplicating.

Test Plan:
  - Edited an Owners package and added the path "/" in two different repositories.
  - Before: only one surived the edit.
  - After: both survived.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13130

Differential Revision: https://secure.phabricator.com/D19420
2018-05-01 09:57:37 -07:00
epriestley
7cfac40a22 Pass full Harbormaster URIs to Buildkite
Summary: See PHI611 for details.

Test Plan:
Ran a Buildkite build, saw Buildkite confirm receipt of these parameters in the HTTP response:

{F5562054}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19419
2018-04-30 22:32:50 -07:00
epriestley
ee32c186dd Stop computing ownership for changed paths for Very Large revisions
Summary:
Depends on D19416. Ref T13110. Ref T13130. See PHI598. When rendering a "Very Large" revision (affecting more than 1,000 files) we currently compute the package/changeset ownership map normally.

This is basically a big list of which packages own which of the files affected by the change. We use it to:

  # Show which packages own each file in the table of contents.
  # Show an "(Owns No Changed Paths)" hint in the reviewers list to help catch out-of-date packages that are no longer relevant.

However, this is expensive to build. We don't render the table of contents at all, so (1) is pointless. The value of (2) is very small on these types of changes, and certainly not worth spending many many seconds computing ownership.

Instead, just skip building out these relationships for very large changes.

Test Plan: Viewed a very large change with package owners; verified it no longer built package map data and rendered the package owners with no "(Owns No Changed Paths)" hints.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13130, T13110

Differential Revision: https://secure.phabricator.com/D19418
2018-04-30 15:44:41 -07:00
epriestley
24305cadb9 Hide the "large" diff warning on "very large" diffs
Summary:
Ref T13110. Ref T13130. When a revision is "large" (100 - 1000 files) we hide the actual textual changes by default. When it is "very large" (more than 1000 files) we hide all the changesets by default.

For "very large" diffs, we currently still show the "large" warning, which doesn't really make sense since there aren't any actual changesets.

When a diff is "very large", don't show the "large" warning.

Test Plan:
  - Viewed a small diff (<100 files), saw no warnings.
  - Viewed a large diff (100-1000 files), saw just the large warning.
  - Viewed a very large diff (>1000 files).
    - Before: both "large" and "very large" help warnings.
    - After: just "very large" warnings.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13130, T13110

Differential Revision: https://secure.phabricator.com/D19416
2018-04-30 15:33:20 -07:00
epriestley
afc3099ee7 Add a view option to disable blame in Diffusion and fix some view transition bugs
Summary:
See PHI604. Ref T13130. Ref T13105. There's currently no way to turn blame off in Diffusion. Add a "Hide Blame" option to the "View Options" dropdown so it can be toggled off.

Also fix a couple of bugs around this: for example, if you loaded a Jupyter notebook and then switched to "Source" view, blame would incorrectly fail to activate because the original rendering of the "stage" used an asynchronous engine so `willRenderRef()` wasn't called to populate blame.

Test Plan:
  - Viewed a source file, toggled blame off/on, reloaded page to see state stick in URL.
  - Viewed a Jupyter notebook, toggled to "Source" view, saw blame.
  - Viewed stuff in Files (no blame UI options).
  - Tried to do some invalid stuff like toggle blame on a non-blame engine (options disable properly).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13130, T13105

Differential Revision: https://secure.phabricator.com/D19414
2018-04-30 15:32:23 -07:00
Austin McKinley
dd6e82698a More-robust search for task assignees
Summary: See discussion in D19415.

Test Plan: Searched for some owners, found tasks as expected.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19417
2018-04-30 12:18:09 -07:00
epriestley
ef48a2b2ee Add a "Rule Detail" link to Herald email
Summary:
See PHI285. Ref T13130. After recent changes Herald sends email about rules, but the mail doesn't currently actually include a link to the rule.

Include a link for consistency and ease-of-use.

Test Plan: Edited a rule, looked at the resulting mail, saw a link to the rule.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13130

Differential Revision: https://secure.phabricator.com/D19413
2018-04-30 05:20:12 -07:00
Austin McKinley
9a0dd55442 Extend PhabricatorPolicyCodex interface to handle "interesting" policy defaults
Summary:
Fixes T13128. Ref PHI590. This is a rough-and-ready implementation of a new `PhabricatorPolicyCodex->compareToDefaultPolicy()` method that subclasses can override to handle special cases of policy defaults. Also implements a `PolicyCodex` for Phriction documents, because the default policy of a Phriction document is the policy of the root document.

I might break this change into two parts, one of which maintains the current behavior and another which implements `PhrictionDocumentPolicyCodex`.

Test Plan: Created some Phriction docs, fiddled with policies, observed expected colors in the header. Will test more comprehensively after review for basic reasonable-ness.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, swisspol

Maniphest Tasks: T13128

Differential Revision: https://secure.phabricator.com/D19409
2018-04-27 16:56:11 -07:00
epriestley
5f774f7008 Stop build target start times from being overwritten on reentry
Summary:
See PHI615. Ref T13130. An install is reporting that "Lease Working Copy" build steps always report "Built instantly" after completion.

I'm not 100% sure that this is the fix, but I'm like 99% sure: "Lease Working Copy" build steps yield after they ask Drydock for a lease. They will later reenter `doWork()`, see that the lease is filled, and complete.

Right now, we reset the start time every time we enter `doWork()`. Instead, set it only if it hasn't been set yet.

Test Plan: This is low-risk and a bit tricky to reproduce locally, but I'll run some production builds and see what they look like.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13130

Differential Revision: https://secure.phabricator.com/D19412
2018-04-27 12:25:45 -07:00
epriestley
d40007aa32 Fix an issue where the Herald test console doesn't work with "Content source" rules
Summary:
Ref T13130. See PHI619. Currently, the Herald "Test Console" doesn't pass a "Content Source" to the adapter, so if any rules of the given type execute a "Content source" field rule, they'll fatal.

Provide a content source:

  - If possible, use the content source from the most recent transaction.
  - Otherwise, build a default "web" content source from the current request.

Test Plan:
  - Wrote a "When [content source][is][whatever]" rule for tasks.
  - Ran test console against a task.
  - Before: got a fatal trying to interact with the content source.
  - After: transcript reports sensible content source.
    - Also commented out the "xaction" logic to test the fallback behavior.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13130

Differential Revision: https://secure.phabricator.com/D19411
2018-04-27 12:25:24 -07:00
epriestley
223d7b84dd Recover more gracefully when favicon configuration points at a corrupt/damaged file
Summary:
Ref T13103. Locally, I managed to break the data for a bunch of files by doing `git clean -df` in a working copy that I'd updated to a commit from many many years ago. Since `conf/local.json` wasn't on the gitignore list many years ago, this removed it, and I lost my encryption keyring.

I've symlinked my local config to a version-controlled file now to avoid this specific type of creative self-sabotage in the future, but this has exposed a few cases where we could handle things more gracefully.

One issue is that if your favicon is customized but the file it points at can't actually be loaded, we fail explosively and you really can't do anything to move forward except somehow guess that you need to fix your favicon. Instead, recover more gracefully.

Test Plan:
  - Configure file encryption.
  - Configure a favicon.
  - Remove the encryption key from your keyring.
  - Purge Phabricator's caches.
  - Before: you pretty much dead-end on a fatal that's hard to understand/fix.
  - After: everything works except your favicon.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13103

Differential Revision: https://secure.phabricator.com/D19406
2018-04-27 12:02:32 -07:00
epriestley
9f8e0ad473 Remove unusual unicode marks in Differential action dropdown
Summary:
See <https://twitter.com/HayleyCAnderson/status/988873585363009536>.

Currently, the action dropdown in Differential shows a heavy "X" after "Request Changes" and a heavy checkmark after "Accept Revision".

Although I'm not convinced that the messaging around "Request Changes" is too strong, I do think these marks are out of place in modern Differential. They came from a simpler time when this dropdown had fewer actions, but feel a little weird and inconsistent to me in the modern UI.

Let's try getting rid of them and see how it goes?

Test Plan:
  - Viewed these actions in the dropdown, no longer saw the mark icons.
  - Grepped for these unicode sequences without getting any other hits.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19405
2018-04-27 11:00:56 -07:00
epriestley
b4796d2837 Add "Content type" and "Rule type" fields to Herald rules for Herald rules
Summary:
Depends on D19400. Ref T13130. Currently, when you write Herald rules about other Herald rules, you can't pick a rule type or content type, so there's no way to get notified about edits to just global rules (which is the primary driving use case).

Add a "Content type" field to let the rule match rules that affect revisions, tasks, commits, etc.

Add a "Rule type" field to let the rule match global, personal, or object rules.

Test Plan:
  - Wrote a global rule for other rules about global Herald rules:

{F5540307}

{F5540308}

  - Ran it against itself which matched:

{F5540309}

  - Ran it against another rule (not a global rule about Herald rules), which did not match:

{F5540311}

  - Also reviewed the fields in those transcripts in more detail to make sure they were extracting matching correctly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13130

Differential Revision: https://secure.phabricator.com/D19403
2018-04-25 06:54:48 -07:00
epriestley
cac41d1e48 Support Herald rules for Herald rules
Summary:
Depends on D19399. Ref T13130. This adds basic support for writing Herald rules against Herald rules. See T13130 for a lot more detail.

This needs a bit more work to be useful: for example, there's no way to specify the rule type or subject, so you can't say "notify me when global rules are edited" or "notify me when Maniphest rules are edited". I'll add some fields for that in followup changes to actually solve the original use case.

Test Plan:
  - Wrote Herald rules against Herald rules.
  - Ran them by editing rules and in the test console.
  - Verified they sent some mail with `bin/mail list-outbound`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13130

Differential Revision: https://secure.phabricator.com/D19400
2018-04-25 06:47:19 -07:00
epriestley
1b24b486f5 Manage object mailKeys automatically in Mail instead of storing them on objects
Summary:
Ref T13065. `mailKey`s are a private secret for each object. In some mail configurations, they help us ensure that inbound mail is authentic: when we send you mail, the "Reply-To" is "T123+456+abcdef".

  - The `T123` is the object you're actually replying to.
  - The `456` is your user ID.
  - The `abcdef` is a hash of your user account with the `mailKey`.

Knowing this hash effectively proves that Phabricator has sent you mail about the object before, i.e. that you legitimately control the account you're sending from. Without this, anyone could send mail to any object "From" someone else, and have comments post under their username.

To generate this hash, we need a stable secret per object. (We can't use properties like the PHID because the secret has to be legitimately secret.)

Today, we store these in `mailKey` properties on the actual objects, and manually generate them. This results in tons and tons and tons of copies of this same ~10 lines of code.

Instead, just store them in the Mail application and generate them on demand. This change also anticipates possibly adding flags like "must encrypt" and "original subject", which are other "durable metadata about mail transmission" properties we may have use cases for eventually.

Test Plan:
  - See next change for additional testing and context.
  - Sent mail about Herald rules (next change); saw mail keys generate cleanly.
  - Destroyed a Herald rule with a mail key, saw the mail properties get nuked.
  - Grepped for `getMailKey()` and converted all callsites I could which aren't the copy/pasted boilerplate present in 50 places.
  - Used `bin/mail receive-test --to T123` to test normal mail receipt of older-style objects and make sure that wasn't broken.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13065

Differential Revision: https://secure.phabricator.com/D19399
2018-04-25 06:46:58 -07:00
epriestley
16af0d35e5 In Differential, prevent "Accept" and "Reject" from "Plan Changes + Draft"
Summary:
Ref T13130. See PHI483. Currently, "Plan Changes + Draft" uses rules like "Plan Changes", not rules like "Draft", and allows "Accept".

This isn't consistent with how "Draft" and "Accept" work in other cases. Make "Plan Changes + Draft" more like "Draft" for consistency.

Also fix a string that didn't have a natural English version.

Test Plan:
  - Added a failing build plan.
  - Created a revision.
  - Loaded the revision before builds completed, saw a nicer piece of text about "waiting for builds" instead of "waiting for 2 build(s)".
  - Builds failed, which automatically demoted the reivsion to "Changes Planned + Draft".
  - As the author and as a reviewer, verified all the actions available to me made sense (particularly, no "Accept").
  - Abandoned the revision to test "Abandoned + Draft".
  - As the author and as a reviewer, verified all the actions available to me made sense.
  - Reclaimed the revision, then used "Request Review" to send it to "Needs Review". Verified that actions made sense and, e.g., reviewers could now "Accept" normally.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13130

Differential Revision: https://secure.phabricator.com/D19398
2018-04-23 14:39:36 -07:00
epriestley
8c78cde32f Stop "git blame" from printing "^" markers on root repository commits
Summary: Depends on D19391. Ref T13126. See that task for some details on what's going on here.

Test Plan:
  - Viewed a file which includes lines that were added during the first commit to the repository.
  - Before D19391: fatal.
  - After D19391: blank.
  - After this patch: accurate blame information.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13126

Differential Revision: https://secure.phabricator.com/D19392
2018-04-20 14:13:10 -07:00
epriestley
95e179d9a4 Fix a fatal in the document engine blame view with files that blame to the initial commit
Summary:
Ref T13126. When you view a file using the new document engine view and some lines were introduced in the initial commit to the repository, Git renders "^abc123" in the blame output.

We currently don't do anything about this, and later fail to look it up and fatal.

It's also unlikely-but-conceivably-possible to end up here if a commit has not imported yet or has been nuked with `bin/remove destroy`.

Let the whole thing run without fataling even if a `$commit` is missing. Future refinements could improve this behavior.

Test Plan: Viewed a file with lines introduced in the initial commit, got empty blame instead of a fatal.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13126

Differential Revision: https://secure.phabricator.com/D19391
2018-04-20 14:12:50 -07:00
epriestley
9bf4df2c1d Allow demoted builds to automatically promote if builds pass after a restart
Summary:
Ref T13124. See PHI584. When you create a draft revision and it automatically demotes to "Changes Planned + Draft" because builds fail, let it promote to "Needs Review" automatically if builds pass. Usually, this will be because someone restarted the builds and they worked the second time.

Although I'm a little wary about adding even more state transitions to the diagram in T13110#237736, I think this one is reasonably natural and not ambiguous.

Test Plan:
  - Created a failing build plan with a "Throw Exception" step.
  - Created a revision which hit the build plan, saw it demote to "Changes Planned" when Harbormaster failed.
  - Edited the build plan to remove the "Throw Exception" step, restarted the build, got a pass.
  - Saw revision promote again:

{F5526104}

I didn't exhaustively test that the other 40 state transitions still work properly, but I think the scope of this change is small enough that it's unlikely I did much collateral damage.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13124

Differential Revision: https://secure.phabricator.com/D19380
2018-04-20 10:50:58 -07:00
Austin McKinley
4dc8e2de56 Add unique constraint to AlmanacInterfaces
Summary: See discussion in D19379. The 4-tuple of (device, network, address, port) should be unique.

Test Plan: Created lots of duplicate interfaces, bound those interfaces to various services, observed migration script clean things up correctly.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19388
2018-04-19 19:16:50 -07:00
epriestley
843bfb4fd8 Add a "commits" attachment to "differential.diff.search" for retrieving local commit information
Summary:
Ref T13124. See PHI593.

When you `arc diff` in a Git or Mercurial repository, we upload some information about the local commits in your working copy which the change was generated from.

In the future (for example, with T1508) we may increase the prominence of this feature.

Provide a stable way to read this information back via the API. This roughly mirrors the information we provide about commits in "diffusion.commit.search", although the latter is less fleshed-out today.

Test Plan: Used `differential.diff.search` to retrieve commit information about Git, Mercurial, and Subversion diffs. (There's no info for Subversion, but it doesn't crash or anything.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13124

Differential Revision: https://secure.phabricator.com/D19386
2018-04-19 17:25:06 -07:00
epriestley
19403fdb8e Improve color use in "[+++- ]" element for colorblind users
Summary:
Ref T13127. Users with red/green colorblindness may have difficulty using this element in its current incarnation.

We could give it different behavior if the "Accessibility" option is set for red/green colorblind users, but try a one-size-fits-all approach since the red/green aren't wholly clear anwyay.

Test Plan: {F5530050}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13127

Differential Revision: https://secure.phabricator.com/D19385
2018-04-19 17:24:44 -07:00
epriestley
70d67a3908 Fix the most significant "phantom notification" badness
Summary:
Ref T13124. Ref T13131. Fixes T8953. See PHI512.

When you receieve a notification about an object and then someone hides that object from you (or deletes it), you get a phantom notification which is very difficult to clear.

For now, test that notifications are visible when you open the menu and clear any that are not.

This could be a little more elegant than it is, but the current behavior is very clearly broken. This unbreaks it, at least.

Test Plan:
  - As Alice, configured task stuff to notify me (instead of sending email).
  - As Bailey, added Alice as a subscriber to a task, then commented on it.
  - As Alice, loaded home and saw a notification count. Didn't click it yet.
  - As Bailey, set the task to private.
  - As Alice, clicked the notification bell menu icon.
    - Before change: no unread notifications, bell menu is semi-stuck in a phantom state which you can't clear.
    - After change: bad notifications automatically cleared.

{F5530005}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13131, T13124, T8953

Differential Revision: https://secure.phabricator.com/D19384
2018-04-19 17:24:19 -07:00
Austin McKinley
e81b2173ad Add edge tables for Phlux
Summary: Fixes T13129. This at least makes the existing UI work again before we banish Phlux to the shadow realm.

Test Plan: Edited the visibility for a Phlux variable, didn't get an error. Nothing showed up in the edge tables when I made those changes, but at least it doesn't error out anymore.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13129

Differential Revision: https://secure.phabricator.com/D19387
2018-04-19 15:49:08 -07:00
Austin McKinley
0a83f253ed Add unique constraint for Almanac network names
Summary:
The name of networks should be unique.

Also adds support for exact-name queries for AlamanacNetworks.

Test Plan: Applied migration with existing duplicates, saw networks renamed, attempted to add duplicates, got a nice error message.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19379
2018-04-19 13:41:15 -07:00
epriestley
a817aa6c71 Add an "Abort Older Builds" build step to Harbormaster
Summary:
Ref T13124. See PHI531. When a revision is updated, builds against the older diff tend to stop being relevant. Add an option to abort outstanding older builds automatically.

At least for now, I'm adding this as a build step instead of some kind of special checkbox. An alternate implementation would be some kind of "Edit Options" action on plans with a checkbox like `[X] When this build starts, abort older builds.`

I think adding it as a build step is a bit simpler, and likely to lead to greater consistency and flexibility down the road, make it easier to add options, etc., and since we don't really have any other current use cases for "a bunch of checkboxes". This might change eventually if we add a bunch of checkboxes for some other reason.

The actual step activates //before// the build queues, so it doesn't need to wait in queue before it can actually act. T13088 discusses some plans here if this sticks.

Test Plan:
  - Created a "Sleep for 120 seconds" build plan and triggered it with Herald.
  - Added an "Abort Older Builds" step.
  - Updated a revision several times in a row, saw older builds abort.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13124

Differential Revision: https://secure.phabricator.com/D19376
2018-04-17 14:59:47 -07:00
epriestley
665529ab60 Restore coverage reporting to Diffusion browse UI
Summary:
Depends on D19377. Ref T13125. Ref T13124. Ref T13105. Coverage reporting in Diffusion didn't initially survive the transition to Document Engine; restore it.

This adds some tentative/theoretical support for multiple columns of coverage, but no way to actually produce them in the UI. For now, the labels, codes, and colors are hard coded.

Test Plan:
Added coverage with `diffusion.updatecoverage`, saw coverage in the UI:

{F5525542}

Hovered over coverage, got labels and highlighting.

Double-checked labels for "N" (Not Executable) and "U" (Uncovered). See PHI577.

Faked some multi-column coverage, but you can't currently get this yourself today:

{F5525544}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13125, T13124, T13105

Differential Revision: https://secure.phabricator.com/D19378
2018-04-17 14:51:47 -07:00
epriestley
f9b3673fbb When mail (like "!history" mail) has multiple comments, label them separately
Summary:
Depends on D19372. Ref T13124. See PHI505. Currently, if you `!history` a task with a lot of comments, you get output like this:

> alice added a comment.
> bailey added a comment.
> alice added a comment.
> alice added a comment.
>
> AAAA
>
> BBBB
>
> AAAA
>
> AAAA

This is impossible to read. Put the "alice added a comment." headers above the actual comments for comments after the first.

These types of mail messages are unusual, but occur in several cases:

  - The new `!history` command.
  - Multiple comments on a draft revision before it promotes out of draft.
  - (Probably?) Conduit API updates which submit multiple comment transactions for some reason.

Test Plan: Used `bin/mail receive-test` to send a `!history` command to a task, saw a much more readable rendering of the transaction log in the resulting email.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13124

Differential Revision: https://secure.phabricator.com/D19373
2018-04-16 12:28:24 -07:00
epriestley
25965260c4 Add a rough "!history" email command to get an entire object history via email
Summary:
See PHI505. Ref T13124. If you're an agent of a hostile state trying to exfiltrate corporate secrets, you might find yourself foiled if Phabricator is secured behind a VPN.

To assist users in this situation, provide a "!history" command which will dump the entire history of an object in a nice text format and get through the troublesome VPN.

Some issues with this:

  - You currently get all the "X added a comment." up top, and then all the comments below. This isn't terribly useful.
  - This goes through the "Must Encrypt" flag, but possibly should not? (On the other hand, this is a pretty willful way to bypass it the flag.)

Test Plan: Used `bin/mail receive-test ...` to send `!history` commands, got somewhat-useful response mail.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13124

Differential Revision: https://secure.phabricator.com/D19372
2018-04-16 12:27:52 -07:00
epriestley
b5f23b023e Add an "--auto" flag to "bin/differential migrate-hunk"
Summary:
Depends on D19370. See T13124. See PHI549. The particular install in PHI549 migrated a large amount of data via the fallback hunk migration script, which does not compress hunks.

Add a mode to `bin/differential migrate-hunk` that amounts to "compress all the hunks which would benefit from compression".

Test Plan: Ran `bin/differential migrate-hunk` with `--auto`, `--all`, `--to`, `--id`, and `--dry-run` in various mixtures. Forced a bunch of hunks to raw ("byte") format, saw it cleanly upgrade them to compressed ("gzde") format.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19371
2018-04-16 12:27:26 -07:00
epriestley
e3de7d09c0 Add an "--all" flag to "bin/differential migrate-hunk"
Summary:
Depends on D19369. Ref T13120. Add a flag to migrate every hunk.

This isn't terribly useful on its own, but I'm going to add an `--auto` flag next so that you can run `--auto --all` to migrate hunks to the preferred hunk format.

Test Plan: Ran `bin/differential migrate-hunk --all --to text`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120

Differential Revision: https://secure.phabricator.com/D19370
2018-04-16 12:26:48 -07:00
epriestley
6d1e007076 Try a more conventional spelling of "Convereted"
Summary: This is a good spelling, but maybe a better spelling is possible.

Test Plan: hmmm

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19369
2018-04-16 12:26:29 -07:00
Austin McKinley
0bf0718fad Add isClusterDevice to Almanac query
Summary: Ref T13076. This will be used by the metric collection system to iterate over the cluster devices.

Test Plan: Created some cluster and non-cluster devices, searched and saw expected results.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T13076

Differential Revision: https://secure.phabricator.com/D19368
2018-04-16 10:05:57 -07:00
epriestley
c46be2a70b Allow Maniphest tasks to be queried by workboard Column PHID via SearchEngine
Summary:
Ref T13120. See PHI571. Fixes T5024. This adds a "View as Query" action to workboard columns, which builds a query in Maniphest that has the current query constraints plus an additional constraint to select only tasks in the specified column.

This is a normal query and can be turned into a dashboard panel, added to a menu, edited, saved as a link, etc.

Much of the complexity here is that finding tasks in a given column isn't entirely straightforward because of how board layout works: when you create a task, it isn't immediately placed in columns. It's only actually added to the "Backlog" column on any boards when someone looks at the board.

To get the right behavior, we must do "board layout" for any queried columns before we can constrain results. This isn't enormously efficient, but should be OK for reasonable boards.

Test Plan:
  - Used "View as Query" for normal columns and milestome columns, got appropriate queries in Maniphest.
  - Applied filters to the board (e.g., "Priorities: wishlist"), then used "View As Query" and had my custom filters respected.
  - Queried some large boards/columns with more than a thousand tasks, got results back within a second or so.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T5024

Differential Revision: https://secure.phabricator.com/D19366
2018-04-13 16:07:44 -07:00
epriestley
ca49fffc1b Fix the legacy "25, 50, 100, unlimited" Harbormaster log links to respect generation selection
Summary:
See PHI565. Ref T13120. Although this older log is on the chopping block (see T13088), there's some migration guidance and other complexity around just replacing it.

Until it gets replaced, make clicking the "number of lines" elements respect the current "Build Generation" setting. Prior to this change, clicking the links would lose the generation information and jump you to the most recent build generation.

Also fix some collateral damage from T13105 where we ended up with white text on a white background in some cases.

Test Plan:
  - Restarted a build to get multiple generations.
  - On each generation, clicked the various "25", "50", etc., links.
  - Saw generation and log window sizes both respected by the links.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13120

Differential Revision: https://secure.phabricator.com/D19367
2018-04-13 11:55:44 -07:00
epriestley
6556536d06 Allow repository cluster bindings to be marked as not "writable", making them read-only
Summary:
Depends on D19356. Fixes T10883. Ref T13120.

  - Add a "writable" property to the bindings, defaulting to "true" with a nice dropdown.
  - When selecting hosts, allow callers to request a writable host.
  - If the caller wants a writable host, only return hosts if they're writable.
  - In SVN and Mercurial, we sometimes return only writable hosts when we //could// return read-only hosts, but figuring out if these request are read-only or read-write is currently tricky. Since these repositories can't really cluster yet, this shouldn't matter too much today.

Test Plan:
  - Without any config changes, viewed repositories via web UI and pushed/pulled via SSH and HTTP.
  - Made all nodes in the cluster read-only by disabling "writable", pulled and hit the web UI (worked), tried to push via SSH and HTTP (got errors about read-only).
  - Put everything back, pulled and pushed.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T10883

Differential Revision: https://secure.phabricator.com/D19357
2018-04-12 16:10:36 -07:00
epriestley
7c7e6d555b Give getAlmanacServiceURI() an "options" parameter to prepare for read-only devices
Summary:
Depends on D19355. Ref T10883. Ref T13120. Rather than adding a million parameters here, wrap the selector-parameters in an `$options`.

The next change adds a new "writable" option to support forcing selection of writable hosts.

Test Plan: Pulled and pushed via HTTP and SSH, viewed repositories via Diffusion.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T10883

Differential Revision: https://secure.phabricator.com/D19356
2018-04-12 16:10:12 -07:00
epriestley
6f810d7813 Turn the "closed" property on cluster repositories into a nice boolean
Summary:
Ref T10883. Ref T13120. There's an existing "closed" property on repository services that stops new repositories from being allocated there.

Turn it into a nice boolean.

Test Plan: Toggled the value on/off using a nice `<select />` with helpful labels instead of a text area.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T10883

Differential Revision: https://secure.phabricator.com/D19355
2018-04-12 16:09:32 -07:00
epriestley
4068aaef61 Toggle revision "shouldBroadcast" correctly when "--draft" is used with prototypes off
Summary:
See PHI573. Ref T13120. Drafts were recently changed so that "draft" and "broadcast" are separate flags, and you can have non-broadcasting revisions in states other than "draft" if builds fail on a draft or you abandon a draft.

However, when draft mode is entered with `arc diff --draft` and you have prototypes off, this flag wasn't being set correctly.

Test Plan: Disabled prototypes, created a revision with `arc diff --draft`, observed that `draft.broadcast` is now correctly `false`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120

Differential Revision: https://secure.phabricator.com/D19360
2018-04-12 16:08:43 -07:00
epriestley
c52e10d1ec Respect external unmentionable PHIDs in Differential revision editor
Summary:
See PHI574. Ref T13120. When you `Ref Txx` or `Fixes Txxx`, we mark it "unmentionable" to prevent the task from generating both a reference and a mention.

If you add a reference to an object (like a commit hash) to a custom remarkup field, there's currently no real way to prevent it from generating a mention, except that you can explicitly mark the PHID as unmentionable on the Editor.

This isn't exactly a first-class feature, but we technically do it in `PhabricatorRepositoryCommitMessageParserWorker`, and it probably doesn't hurt or interfere with anything to support it slightly better.

In Differential, respect any existing value and append new values to it rather than overwriting the value.

Test Plan: Edited a revision summary to include `Ref Txxx`, saw only a reference (not a mention) generate.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120

Differential Revision: https://secure.phabricator.com/D19361
2018-04-12 16:07:55 -07:00
epriestley
70056a9072 When creating a file by downloading a URI, truncate the length of the default name
Summary:
See <https://discourse.phabricator-community.org/t/embedding-external-images-url-show-error-for-long-urls/1339>.

When we download a file from a URI, we provide a default name based on the URI. However, if the URI is something like `http://example.com/very-very-very-....-long.jpg` with more than 255 characters, we may suggest a name which won't fit into the `name` column of `PhabricatorFile`.

Instead, suggest a default name no longer than 64 bytes.

Test Plan:
  - Used the `{image ...}` example from the Discourse report locally; got an image with a truncated name.
  - Used a normal `{image ...}`, got an image file with a normal name.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19353
2018-04-12 13:29:53 -07:00
epriestley
ea9187ea92 Allow Almanac properties to be set and deleted via Conduit
Summary:
Depends on D19342. Ref T12414. Ref T13120. This adds an EditEngine extension for editing Almanac properties.

The actual wire format is a little weird. Normally, we'd have a transaction for each property, but since you can pick any property names you want we can't really do that (we'd have to generate infinite transactions).

The transaction wire format anticipates that transactions may eventually get some kind of metadata -- each transaction looks like this:

```
{
  "type": "title",
  "value": "Example title"
}
```

...and we can add more keys there. For example, I could have made this transaction look like this:

```
{
  "type": "property.set",
  "almanac.property.key": "some-key",
  "value": "some-value"
}
```

However, I don't want to just accept any possible key freely, and it might be a decent chunk of work to formalize this better. It also doesn't feel great.

I just built special transaction types intead, so you:

```
{
  "type": "property.set",
  "value": {
   "some-key": "some-value",
   ...
  }
}
```

Internally, we may generate more than one transaction as a result (if the "value" has more than one key).

This feels a bit more natural and is probably easier for clients to use anyway.

Test Plan: Set and deleted Service, Device and Binding properties via the API.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19343
2018-04-11 10:42:10 -07:00
epriestley
c1558031c2 Make various small quality-of-life improvements for Almanac properties
Summary:
Depends on D19341. Ref T12414. Ref T13120.

  - Fix a bug where default-valued properties didn't get rendered in grey as they're supposed to (as a hint that the value isn't customized).
  - When resetting a builtin property won't do anything, visually disable the button as a hint.
  - Allow Services to specify properties on their Bindings.
  - Specify that repository bindings have a "protocol" property, so it becomes an explicit thing in the UI. Previously, you had to read the documentation to figure this out.
  - When editing bindings, use the EditField and its configuration if possible. This turns the "Protocol" property into a dropdown in the UI where you select between "http", "https" and "ssh".
  - Give the "protocol" binding a smart default based on the port number of the corresponding interface.

Test Plan:
  - Viewed properties on Services, Devices and Bindings.
  - Saw them render sensibly, and grey out + grey button when a builtin value has a default setting.
  - Saw "Protocol" appear as a default property on repository cluster bindings and get a smart value.
  - Edited "protocol", got a nice dropdown.

{F5518791}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19342
2018-04-11 10:38:41 -07:00
epriestley
d56a37b636 Allow Almanac Bindings to be enabled/disabled via API and support the "properties" attachment
Summary:
Depends on D19340. Ref T12414. Ref T13120. See T12414 for some discussion about direction here.

Since I think retaining "enabled/disabled" as a simple flag is reasonable, expose it via the API for readers and writers.

Also expose binding properties.

Test Plan:
  - Searched for bindings and properties with "alamanc.binding.search".
  - Enabled and disabled bindings with "almanac.binding.edit".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19341
2018-04-11 10:38:09 -07:00
epriestley
208504a5e3 Provide "almanac.binding.search" and "almanac.binding.edit"
Summary:
Depends on D19338. Ref T13120. Ref T12414. These are the last of the new API methods.

This stuff still doesn't work:

  - You can't actually enable/disable bindings yet. I want to take a look at the use cases and consider changing "disabled" to "status", or providing a different way to solve the problem.
  - You can't edit properties via the API. I expect to enable this for all `AlmanacPropertyInterface` objects with an extension in a future change.

Test Plan:
  - Searched for bindings via API.
  - Viewed binding web UI for API methods.
  - Created bindings via API.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19340
2018-04-11 10:37:38 -07:00
epriestley
e502df509d Implement "almanac.interface.search" and "almanac.interface.edit"
Summary: Depends on D19337. Ref T13120. Ref T12414. These are slightly more substantive than namespace/network, but pretty much standard fare.

Test Plan:
  - Searched for interfaces with "almanac.interface.search".
  - Created and edited interfaces with "almanac.interface.edit".
  - Created and edited interfaces with web UI since some stuff got tweaked.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19338
2018-04-11 10:35:03 -07:00
epriestley
10947c8684 Add "almanac.namespace.edit" and "almanac.namespace.search" API methods
Summary: Depends on D19336. Ref T13120. Ref T12414. These are simple, straightforward, and uninteresting.

Test Plan:
  - Searched for namespaces with "almanac.namespace.search".
  - Created and edited namespaces with "almanac.namespace.edit".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19337
2018-04-11 10:34:30 -07:00
epriestley
9022e14082 Use a more conventional spelling of "Almanac" for "almanac.service.edit" class
Summary: Depends on D19335. Ref T13120. Ref T12414. There are many good ways to spell "almanac", but stick with convention here.

Test Plan: (O_O)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19336
2018-04-11 10:34:04 -07:00
epriestley
a8c4da13c0 Add "almanac.network.edit" and "almanac.network.search" API methods
Summary: Depends on D19334. Ref T13120. Ref T12414. These are pretty straightforward, but no one really has a use case for them anyway today so they're primarily just for completeness.

Test Plan:
  - Queried networks with `almanac.network.search`.
  - Created and edited networks with `almanac.network.edit`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19335
2018-04-11 10:33:41 -07:00
epriestley
4bce3fc8e6 Modularize Almanac property transactions
Summary:
Depends on D19329. Ref T13120. Ref T12414. Recent changes have mostly modularized Almanac transactions, but the "property" transactions remained written in an older style with the logic on the Editor/Transaction classes.

This moves them to modern modular transactions. These end up being a little bit copy-pastey, but it doesn't feel too terribly bad.

Test Plan: Created, edited, and deleted properties on services, devices and bindings. Grepped for removed constants.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19334
2018-04-11 10:33:18 -07:00
epriestley
71c77fcc3a Modularize transactions for Almanac Device
Summary:
Depends on D19328. Ref T13120. Ref T12414.

Prior work has left us with just a NAME transaction here, which is straightforward to modularize.

Test Plan:
  - Created and renamed devices.
  - Tried to set no name, a bad name, a duplicate name (got errors).
  - Tried to create/rename into a namespace I could not edit (got an error).
  - Grepped for `AlmanacDeviceTransaction::`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19329
2018-04-11 10:31:46 -07:00
epriestley
4e156a0385 Remove TYPE_INTERFACE transaction from Almanac Device
Summary:
Depends on D19325. Ref T13120. Ref T12414.

This no longer has any callers in the upstream or in Phacility support libraries, so get rid of it.

This will make modularizing Device transactions significantly easier, since the other transactions are reasonable, normal sorts of transactions.

For existing devices, this leaves some "author edited this object." transactions in the log. I might just leave those since they aren't really hurting anything, or maybe I'll clean them up or hide them later once I have more confidence that these changes are stable.

Test Plan: Grepped for `TYPE_INTERFACE` and `AlmanacDeviceTransaction`, found no callsites.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19328
2018-04-11 10:31:25 -07:00
epriestley
d240969e47 Use Interface transactions, not Device transactions, to destroy Interfaces
Summary:
Depends on D19324. Ref T13120. Ref T12414.

This moves "Destroy Interface" to use Interface transactions instead of Device transactions, so we can ultimately get rid of the complex and difficult-to-modernize `AlmanacDeviceTransaction::TYPE_INTERFACE`.

This transaction is a bit weird since it makes the interface delete itself, but this should work OK for now. At some point in the future I'd probably want to change this into more of a "disable" action, but I don't think we face any immediate peril by retaining this behavior for now.

Test Plan:
  - Destroyed interfaces on devices using the web UI, saw them vanish.
  - Ran daemons, nothing fataled/exploded even though the transaction is weird and destroys the object it affects.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19325
2018-04-11 10:30:15 -07:00
epriestley
6ccf35f9a2 Edit Interfaces in Almanac with EditEngine
Summary:
Depends on D19323. Ref T13120. Ref T12414.

Move editing to modern stuff and fix some implementation errors from D19323 (mostly copy/paste stuff).

Test Plan:
  - Created and edited interfaces.
  - Tried to create/edit an interface with a bogus/empty address/port, got errors.
  - Tried to create an interface on a bogus device, got an error.
  - Tried to create an interface on a device I could not edit, got an error.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19324
2018-04-11 10:29:50 -07:00
epriestley
f9c6a69d9c Add skeleton code for Almanac Interfaces to have real transactions
Summary:
Depends on D19322. Ref T13120. Ref T12414.

Currently, `AlmanacDevice` has a bit of a beast of a `TYPE_INTERFACE` transaction that fully creates a complex Interface object. This isn't very flexible or consistent, and Interfaces are complex enough to reasonably have their own object behaviors (for example, they have their own PHIDs).

The complexity of this transaction makes modularizing `AlmanacDevice` transactions tricky. To simplify this, move Interface toward having its own set of normal transactions.

This change just adds some reasonable-looking transactions; it doesn't actually hook them up in the UI or make them reachable. I'll test that they actually work as I swap the UI over.

We may also have some code using the `TYPE_INTERFACE` transaction in Phacility support stuff, so that may need to wait a week to actually phase out.

Test Plan: Ran `bin/storage upgrade` and `arc liberate`. This code isn't reachable yet.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19323
2018-04-11 10:29:26 -07:00
epriestley
580409b562 Modularize Almanac Network transactions
Summary: Depends on D19321. Ref T13120. Ref T12414. Move transactions for Almanac Networks (just "name") to ModularTransactions.

Test Plan:
  - Created a new network.
  - Renamed a network.
  - Tried to create a network with no name (got an error).
  - Grepped for `AlmanacNetworkTransaction::`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19322
2018-04-11 10:29:05 -07:00
epriestley
f62494355d Modularize Almanac Binding transactions
Summary: Depends on D19320. Ref T13120. Ref T12414. Move transactions for Almanac Bindings to ModularTransactions.

Test Plan:
  - Created a new binding.
  - Tried to create a duplicate binding, got an error.
  - Edited a binding to rebind it to a different device.
  - Disabled and enabled bindings.
  - Grepped for `AlmanacBindingTransaction::` constants.

When a binding is created, it currently renders a bad "changed the interface from ??? to X" transaction. This is because creation isn't currently using EditEngine. I plan to swap it shortly, which will turn this into a real "Create" transaction and fix the issue.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19321
2018-04-11 10:28:42 -07:00
epriestley
5ada1211cd Modularize Almanac Namespace transactions
Summary: Depends on D19318. Ref T13120. Ref T12414. Move transactions for Almanac Namespaces ("name" is the only meaningful one) to ModularTransactions.

Test Plan:
  - Created a new namespace.
  - Edited a namespace.
  - Tried to choose no name, an invalid name, a duplicate name, and a name in a namespace I can't edit; got appropriate errors.
  - Grepped for `AlmanacNamespaceTransaction::TYPE_NAME`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19320
2018-04-11 10:24:10 -07:00
epriestley
6983479e4f Allow "almanac.service.edit" to create services
Summary:
Depends on D19317. Ref T13120. Ref T12414. See PHI145. See PHI473.

This adds a Conduit-only "type" transaction for Almanac services. This is very similar to the approach in D18849 for Drydock blueprints.

Test Plan:
  - Tried to create an empty service via "almanac.service.edit", was told to pick a type.
  - Tried to pick a bad type, was told to pick a good type.
  - Created a new Almanac service via "almanac.service.edit".
  - Tried to edit the service to change the type, wasn't allowed to.
  - Created and edited via the web UI, nothing changed from before.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19318
2018-04-11 10:23:50 -07:00
epriestley
c428f60a97 Partially modularize AlmanacService transactions
Summary:
Ref T13120. Ref T12414. See PHI145. See PHI473. This partially modernizes AlmanacService transactions by moving them to ModularTransactions.

This isn't complete because the "update property" and "remove property" transactions aren't modularized. They still //work//, since the parent Editor implements them, but they no longer render properly on the timeline since the `Transaction` object no longer has rendering logic for them.

Tentatively, I'm going to try to convert the rest of the Almanac objects and then modularize those transactions. (Currently, all of Binding, Device, Namespace and Service support properties, although they can only actually be edited on Service, Device and Binding.)

If that turns out to be really tricky for some reason I can just copy/paste the timeline rendering for now, but I think it won't be too hard.

Test Plan:
  - Created and edited Services.
  - Tried to create a service with: a bad name, no name, a name which put it in a namespace I can't edit (got errors in all cases).
  - Edited and removed properties. The edits worked, the timeline just renders a generic story now ('X edited this object (transaction type "almanac:property:update").').

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19317
2018-04-11 10:22:34 -07:00
Austin McKinley
0755482bf0 Add transactions for installing/uninstalling applications
Summary: Fixes T11476.

Test Plan:
 - Installed/uninstalled the Conpherence application
 - Observed correct timeline stories
 - Observed correct config in database
 - Observed 404 for application page

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11476

Differential Revision: https://secure.phabricator.com/D19339
2018-04-11 08:54:55 -07:00
Austin McKinley
d398bcd67c Fix argument ordering in error message
Summary:
Before:
```
$ ./config set phabricator.base-uri local.phacility.com:8080
Usage Exception: Config option 'http://' is invalid. The URI must start with https://' or 'phabricator.base-uri'.
```
After:
```
$ ./config set phabricator.base-uri local.phacility.com:8080
Usage Exception: Config option 'phabricator.base-uri' is invalid. The URI must start with http://' or 'https://'.
```

Test Plan: See above

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19330
2018-04-10 10:18:51 -07:00
epriestley
1680211702 Remove dead "Service Lock" code from Almanac
Summary:
Depends on D19315. Ref T13120. Ref T12414. See PHI145. See PHI473. I want to move Almanac services to ModularTransactions but ran into this old piece of dead/unused code along the way.

Long ago, Almanac services could be individually "locked", but this didn't really work out very well. It was replaced by "Can Manage Cluster Services" in D15339 and prior changes, but not all of the old "Lock" code got cleaned up.

I don't expect to restore this feature, so clean it up now.

Test Plan:
  - Grepped for `AlmanacServiceTransaction::TYPE_LOCK`, `TYPE_LOCK`, etc.
  - Grepped for `updateServiceLock()`, no callsites.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13120, T12414

Differential Revision: https://secure.phabricator.com/D19316
2018-04-09 11:38:04 -07:00
epriestley
72ab8640c5 Narrowly fix web UI fatal for "almanac.service.edit" Conduit API method
Summary:
See T13120. See T12414. See PHI145. See PHI473. Almanac services require a type before they can do anything, and EditEngine currently builds one with no type. We then fatal when trying to do mundane things like generate documentation.

Instead, build a generic but complete Service for documentation generation in the web UI. This is similar to the previous Drydock Blueprint change from D18849 (or some earlier diff in that series).

(You still probably can't use this method to //create// a service; I'll fix that in the next change.)

Test Plan:
  - Viewed "almanac.service.edit" in the web UI.
    - Before: immediate fatal ("No Almanac service type "" exists!").
    - After: Page works. No claims about the method doing anything useful.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19315
2018-04-09 11:37:39 -07:00
epriestley
472bc3d90a Colorize lines in blame under DocumentEngine, to show relative age of changes
Summary:
Depends on D19313. Ref T13105. Fixes T13015. We lost the coloration for ages in the switch to Document Engine.

Restore it, and use a wider range of colors to make the information more clear.

Test Plan: Viewed some blame, saw a nice explosion of bright colors. This is a cornerstone of good design.

Maniphest Tasks: T13105, T13015

Differential Revision: https://secure.phabricator.com/D19314
2018-04-09 06:11:47 -07:00
epriestley
eca7dc25f2 Use javelin_tag(), not phutil_tag(), to render revision blame tooltips properly
Summary: Depends on D19310. Ref T13105. The "meta" value was not populating correctly because this used `phutil_tag()`.

Test Plan: Will verify on `secure`.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19311
2018-04-09 06:10:09 -07:00
epriestley
09c6d42b95 Mostly make blame work with DocumentEngine
Summary: Ref T13105. This needs refinement but blame sort of works again, now.

Test Plan: Viewed files in Diffusion and Files; saw blame in Diffusion when viewing in source mode.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19309
2018-04-09 04:48:21 -07:00
epriestley
90a614778c Make repository symbol references work with DocumentEngine
Summary: Ref T13105. Ref T13047. This makes symbol indexes work with DocumentEngine in Files, and restores support in Diffusion.

Test Plan: Command-clicked stuff, got taken to the symbol index with reasonable metadata in Diffusion, Differential and Files.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105, T13047

Differential Revision: https://secure.phabricator.com/D19307
2018-04-09 04:47:28 -07:00
epriestley
0363febeb2 Disable default syntax highlighting for large files in DocumentEngine
Summary: Ref T13105. See also T7895. When users render very large files as source via DocumentEngine, skip highlighting.

Test Plan: Fiddled with the limit, viewed files, saw highlighting degrade.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19306
2018-04-09 04:47:08 -07:00
epriestley
6dea2ba3b3 Fix DocumentEngine line behaviors in Diffusion
Summary:
Ref T13105. Fixes some issues with line linking and highlighting under DocumentEngine:

  - Adding `$1-3` to the URI didn't work correctly with query parameters.
  - Reading `$1-3` from the URI didn't work correctly because Diffusion parses them slightly abnormally.

Test Plan: Clicked/dragged lines to select them. Observed URI. Reloaded page, got the right selection.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19305
2018-04-09 04:46:47 -07:00
epriestley
1fde4a9450 Move Diffusion browse rendering to DocumentEngine, breaking almost all features
Summary:
Ref T13105. This breaks about 9,000 features but moves Diffusion to DocumentEngine for rendering. See T13105 for a more complete list of all the broken stuff.

But you can't bake a software without breaking all the features every time you make a change, right?

Test Plan: Viewed various files in Diffusion, used DocumentEngine features like highlighting and rendering engine selection.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Subscribers: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19302
2018-04-09 04:46:26 -07:00
epriestley
245132a0b2 Pull file Document Engine rendering out of "Files" application controllers
Summary:
Ref T13105. This separates document rendering from the Controllers which trigger it so it can be reused elsewhere (notably, in Diffusion).

This shouldn't cause any application behavior to change, it just pulls the rendering logic out so it can be reused elsewhere.

Test Plan: Viewed various types of files in Files; toggled rendering, highlighting, and encoding.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19301
2018-04-09 04:45:58 -07:00
epriestley
7d4e25614d Remove the ability to disable blame in Diffusion
Summary: Ref T13105. Given that we now load blame with AJAX, it's not clear that there's any benefit to disabling it. This would also interact oddly with the document engine.

Test Plan: Viewed files in Diffusion, no longer saw blame-related options.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19300
2018-04-09 04:45:16 -07:00
epriestley
9bb338c038 Revert the alternate menu names for applications
Summary: This reverts D18524. See that revision for discussion.

Test Plan: Viewed home menu, saw application names as menu items.

Differential Revision: https://secure.phabricator.com/D19308
2018-04-08 10:20:24 -07:00
epriestley
af87f414e8 Stop the debugging view for typeahead datasources from fataling
Summary: Fixes T13119. Ref T13120. This isn't the world's most elegant patch, but restores the debugging version of this view to service.

Test Plan: Viewed debugging phage (at `/typeahead/class/`). Used the actual proxy (by changing a datasource custom field from the comment area).

Maniphest Tasks: T13120, T13119

Differential Revision: https://secure.phabricator.com/D19304
2018-04-08 06:16:56 -07:00
epriestley
f01c2e3694 Remove "Large Changes" documentation and make some minor behavioral improvements
Summary:
Depends on D19296. Ref T13110.

  - Remove the "Large Changesets" documentation since we now degrade very large changesets and I don't have any evidence that anyone has ever tried to follow any of the recommendations in this document.
  - Remove references to it.
  - When an older revision doesn't have denormalized size information on the Revision object itself, don't render a scale element (instead of rendering a bogus one).
  - Try to improve terminology consistency around "Large Change" (100-1000 files) vs "Very Large Change" (1000+ files) vs "Enormous Change" (too large to hold in memory).

Test Plan: Viewed revisions; grepped for documentation.

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19298
2018-04-05 06:40:46 -07:00
epriestley
1b363a831e When a revision changes more than 1,000 files, don't show the changes on the main page
Summary: Depends on D19295. Ref T13110. Degrade the review UX when users try to interact with changes which are too large to receive human review.

Test Plan: Reduced the "very large" limit, browsed some changes, saw various elements degrade.

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19296
2018-04-05 06:40:22 -07:00
epriestley
8c8e7f07b5 Add a standalone view for browsing changesets of very large revisions
Summary: Ref T13110. Installs have various reasons for sending unreviewable changes (changes where the text of the change will never be reviewed by a human) through Differential anyway. Prepare for accommodating this more gracefully by building a standalone changeset list page which paginates the changesets.

Test Plan: Clicked the new "Changeset List" button on a revision, was taken to a separate page.

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19295
2018-04-05 06:35:06 -07:00
epriestley
3de002c841 Fix a commit hook issue where pushing dangerous changes would fatal before hitting the dragon bureaucrats
Summary: See <https://discourse.phabricator-community.org/t/php-fatal-when-using-git-push-d/1317>. The behavioral changes for Herald on initial import from D19265 could leave `$all_updates` undefined if we throw early enough.

Test Plan: Pushed a dangerous change, saw dragon bureaucrats again.

Differential Revision: https://secure.phabricator.com/D19297
2018-04-05 06:19:49 -07:00
epriestley
e70c9f72a4 Show revision sizes using a perplexing, inexplicable symbol code
Summary: Ref T13110. See PHI230. Show revision sizes on a roughly logarithmic scale from 1-7 stars. See D16322 for theorycrafting on this element.

Test Plan: Looked at some revisions, saw plausible-looking size markers.

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19294
2018-04-03 12:49:27 -07:00
epriestley
e40aec0210 When a revision has more than 7 reviewers, render only the first 7 in the list view
Summary:
See PHI489. Ref T13110. At least for now, this just shows "..." at the end since you can click the revision to see the whole list anyway.

Also remove the older-style external Handle passing in favor of lazy construction via HandlePool.

Test Plan: Viewed revisions, fiddled with the 7 limit, got sensible-seeming "..." behavior.

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19293
2018-04-03 12:47:43 -07:00
epriestley
592d72e006 Move PhabricatorModularTransaction slightly closer to having "final" methods again
Summary: Depends on D19290. Ref T13110. Differential still has some hacks in place which require these methods to "very temporarily" be nonfinal, but the badness can be slightly reduced nowadays.

Test Plan: Loaded some pages, nothing fataled.

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19291
2018-04-03 11:13:58 -07:00
epriestley
6f520e0534 Clean up an old transaction state flag
Summary: Depends on D19289. Ref T13110. This flag has been obsolete for some time and has no callers.

Test Plan: Grepped for `hasReviewTransaction`, no hits.

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19290
2018-04-03 11:13:31 -07:00
epriestley
804f9817c3 When a draft's builds fail and it demotes to "Changes Planned + Draft", notify the author (only) via email
Summary:
Depends on D19288. Ref T13110. In addition to kicking revisions back to "Changes Planned" when builds fail, notify the author that they need to fix their awful garbage change.

(The actual email could be more useful than it currently is.)

Test Plan: Created a revision with failing remote builds, saw email about the problem generate.

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19289
2018-04-03 11:11:28 -07:00
epriestley
f4f3311312 When reclaiming an "Abandoned + Draft" revision, return it to "Draft", not "Needs Review"
Summary: Depends on D19287. Ref T13110. Currently, "Abandon" and then "Reclaim" moves you out of "Draft" without setting the "Should Broadcast" flag. Keep these revisions in draft instead.

Test Plan: Reclaimed an abandoned + draft revision, got a draft revision instead of a "needs review + nonbroadcast" revision (which isn't a meaningful state).

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19288
2018-04-03 11:11:06 -07:00
epriestley
adf8fdef0e When remote builds fail, demote revisions to "Changes Planned + But, Still A Draft"
Summary:
Depends on D19286. Ref T13110. After builds fail remote builds, put revisions back in the author's queue.

This doesn't actually notify the author quite yet.

Test Plan: Made a failing build plan run on revisions, created a revision, saw it demote after builds failed.

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19287
2018-04-03 11:10:45 -07:00
epriestley
d9bd36039f When a non-broadcasting revision is updated, put it in "Draft", not "Needs Review"
Summary: Depends on D19285. Ref T13110. When you update an "Abandoned + But, Never Promoted" revision or (in the future) a "Changes Planned + But, Never Promoted" revision, return it to the "Draft" state rather than promoting it.

Test Plan: Updated an "Abandoned + Draft" revision, saw it return to "Draft".

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19286
2018-04-03 11:10:13 -07:00
epriestley
615d27c8e9 Show an additional "Draft" tag on non-broadcasting revisions in a non-draft state
Summary:
Depends on D19284. Ref T13110. It's now possible to get a revision into a "Abandoned + But, Never Promoted From Draft" state. Show this in the header and provide the draft hint above the comment area.

Also, remove `shouldBroadcast()`. The method `getShouldBroadcast()` now has the same meaning.

Finally, migrate existing drafts to `shouldBroadcast = false` and default `shouldBroadcast` to `true`. If we don't do this, every older revision becomes a non-broadcasting revision because this flag was not explicitly set on revision creation before, only on promotion out of draft.

Test Plan: Ran migration; abandoned draft revisions and ended up in a draft + abandoned state.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19285
2018-04-03 11:09:49 -07:00
epriestley
38e788c99a Partially decouple revision broadcasting from revision draft state
Summary:
Depends on D19283. Ref T13110. To enable "Changes Planned + But, Still A Draft" and "Abandoned + But, Never Promoted From Draft" states, decouple the "broadcast" flag from the "draft" state.

Broadcast behavior is now based only on the `shouldBroadcast` flag, and revisions in any state may have this flag.

Revisions gain this flag when created as a non-draft, or when they leave the draft state for the first time.

There are probably still some ways you can get the wrong result here -- maybe abandon + update -- but those can be cleaned up as they arise.

Test Plan: Kinda poked it a bit but I'll vet this more heavily at the end of this sequence.

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19284
2018-04-03 11:09:26 -07:00
epriestley
3b5a7d1c88 Rename the Differential "hasBroadcast" flag to "shouldBroadcast"
Summary:
Depends on D19282. Ref T13110. I want to introduce "Changes Planned + Still A Draft" and "Abandoned + Still A Draft" states, at a minimum.

I think the "hasBroadcast" flag is effectively identical to a hypothetical "stillADraft" flag, so rename it to "shouldBroadcast" to better match its intended behavior.

This just changes labels, not any behavior.

Test Plan: Grepped for `hasBroadcast` and `HAS_BROADCAST`.

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19283
2018-04-03 11:09:02 -07:00
epriestley
f350b9e464 Explicitly condition Differential draft promotion on only "impactful" builds
Summary:
Depends on D19281. This increases consistency between build timeline publishing and revision draft promotion.

There's no real behavioral change here (switching how publishing worked already changed the beahvior) but this sends more callsites down the same code paths.

Since the builds we're looking at include completed builds, change the term "active" to "impactful". This describes the same set of builds, but hopefully describes them more accurately.

Test Plan: Created a local revision, saw it plausibly interact with draft status and promote. There are a lot of moving parts here and some stuff may well have slipped through.

Differential Revision: https://secure.phabricator.com/D19282
2018-04-03 11:06:46 -07:00
epriestley
51461f18c1 When publishing buildables in Differential, ignore autobuilds (local lint and unit)
Summary:
Depends on D19280. Ref T13110. Although Harbormaster cares about all builds, Differential does not practically care about local lint and unit results in determining build status.

In Differential, orient publishing around "remote builds" instead of "builds".

This does not yet change any of the draft logic, it just makes the timeline story use newer logic.

Test Plan: Used `bin/harbormaster publish` (with some guard-clause removal) to publish some buildables to revisions without anything crashing.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19281
2018-04-03 11:02:12 -07:00
epriestley
ada0c9126c Provide a modular buildable transaction in Diffusion
Summary:
Depends on D19279. Ref T13110. This implements the existing publishing logic for buildables, but does so via ModularTransactions instead of a core transaction type.

Since each application is implementing build transactions independently, this removes the core type.

Next, Differential will get a similar treatment.

Test Plan: Used `bin/harbormaster publish` (with some commenting-out-guard-clauses) to publish a commit Buildable; saw unchanged feed behavior.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19280
2018-04-03 11:01:37 -07:00
epriestley
c20b4e365b Move structural build publishing logic to BuildEngine, provide "bin/harbormaster publish"
Summary:
Depends on D19278. Ref T13110. This moves most of the structural logic for publishing builds to BuildableEngine and provides a `bin/harbormaster publish` to make publishing easy to retry/debug.

This intentionally removes the bit which actually does anything when builds publish. Followup changes will implement application-specific versions of the publishing logic in Differential and Diffusion.

Test Plan: Ran `bin/harbormaster publish Bxxx`, saw it do nothing (but not crash).

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19279
2018-04-03 10:58:27 -07:00
epriestley
95c9d403f4 Make objects implementing BuildableInterface produce a BuildableEngine
Summary:
Ref T13110. Currently, build status is published the same way for every Buildable by the BuildEngine.

I want to change this to delegate publishing to each Buildable, particularly so that Differential may use more detailed rules for handling builds and drafts.

Rather than add additional methods to the existing `BuildableInterface`, add an engine generator method instead. This is a pattern which has seen more use recently (e.g., in Ferret) and lets us pay a little more upfront to pull complex pieces of logic out of the main class and let them use inheritence more easily. If we had Traits that might cover this to some degree.

I'd expect to eventually reduce the size of `BuildableInterface` and move the `CircleCI` and `BuildKite` interfaces so that the `BuildableEngine` implements them instead of the main object.

Here, this new engine does nothing and is never instantiated. In upcoming changes, publishing logic will move into it so that Differential can handle publishing differently.

Test Plan: Ran `arc liberate`, loaded pages, grepped for `BuildableInterface`.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13110

Differential Revision: https://secure.phabricator.com/D19278
2018-04-03 10:57:51 -07:00
epriestley
7189cb7ba8 Support text encoding and syntax highlighting options in document rendering
Summary: Depends on D19273. Ref T13105. Adds "Change Text Encoding..." and "Highlight As..." options when rendering documents, and makes an effort to automatically detect and handle text encoding.

Test Plan:
  - Uploaded a Shift-JIS file, saw it auto-detect as Shift-JIS.
  - Converted files between encodings.
  - Highlighted various things as "Rainbow", etc.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19274
2018-03-30 11:28:52 -07:00
epriestley
ccbc8a430f Make Jupyter notebooks use the fast builtin Python highlighter
Summary:
Ref T13105. This is silly, but "py" and "python" end up in different places today, and "py" is ~100x faster than "python".

See also T3626 for longer-term plans on this.

Test Plan: Reloaded a Jupyter notebook, saw it render almost instantly instead of taking a few seconds.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19273
2018-03-30 11:26:48 -07:00
epriestley
66392e5b8b Add a rough "bin/repository unpublish" workflow to attempt to cleanup improperly published repositories
Summary:
Ref T13114. See PHI514. This makes some attempt to undo the damage caused by incorrectly publishing a repository.

Don't run this.

Test Plan: Yikes.

Maniphest Tasks: T13114

Differential Revision: https://secure.phabricator.com/D19271
2018-03-30 08:46:11 -07:00
epriestley
7f9a9bc800 Make Harbormaster objects destructible
Summary:
Ref T13114. See PHI511. Ref T13072. This makes Buildables, Builds, Targets and Artifacts destructible with `bin/remove destroy`.

This might not be totally exhaustive. In particular:

  - File artifacts won't destroy the file. This is sort of okay because file artifacts are currently just a file reference, but probably shouldn't be how things work in the long term.
  - `BuildCommand` doesn't get cleaned up, but `BuildMessage` does on `Build`. See T13072 for more.

Test Plan: Used `bin/remove destroy` to nuke a bunch of builds, buildables, etc. Loaded stuff in the web UI and it all looked like it got nuked properly.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13114, T13072

Differential Revision: https://secure.phabricator.com/D19269
2018-03-29 13:01:14 -07:00
epriestley
7915445543 Fix two issues with Differential updates and Owners
Summary:
Ref T13114.

  - Followup fix for D19267, which didn't work correctly with //new// revision creation.
  - Followup fix for changes in T11015. Some of the querying logic was still handling "/x.y" and "/x.y/" differently. Instead, normalize consistently to "/x.y/"

Test Plan:
  - Created a new revision cleanly.
  - Created a package owning only a `example.txt` file and saw Differential find it as an owning package in the table of contents.

Maniphest Tasks: T13114

Differential Revision: https://secure.phabricator.com/D19268
2018-03-29 11:32:23 -07:00
epriestley
93cb6e3bde Make updating a revision with the same active diff a no-op
Summary: Ref T13114. See PHI515. Updating a revision with the same, currently active diff became an error at some point (probably D19175). This is inconsistent; make it an allowable no-op instead.

Test Plan:
  - Updated a revision's diff via Conduit.
  - Updated to the same diff, no-op.
  - Tried to update a different revision, error ("already attached elsewhere").
  - Updated with a different diff.
  - Tried to update with the original diff, error ("previously attached version").

Maniphest Tasks: T13114

Differential Revision: https://secure.phabricator.com/D19267
2018-03-29 09:59:39 -07:00
epriestley
74216ea8e0 Disable Herald and enormous change protection for repository initial imports
Summary: See PHI514. Ref T13114. Ref T8951. When a push is an "initial import" (a push of at least 7 commits to an empty repository) don't run Herald or enormous change protection.

Test Plan: Pushed some non-initial changes to a repository, and some initial changes.

Maniphest Tasks: T13114, T8951

Differential Revision: https://secure.phabricator.com/D19265
2018-03-29 08:05:07 -07:00
epriestley
5cb6832572 Fix usage of fprintf() in bin/drydock command
Summary: See PHI513. `fprintf()` takes `(thing, pattern, args, ...)` but we aren't passing a `pattern`, so if the command returns a "%" in the output we get an error.

Test Plan:
  - Installed `bytes`, a great useful program which prints all the bytes, on my HoaxOS(tm) system (see D19102).

```
epriestley@orbital ~/dev/phabricator $ ./bin/drydock command --lease 76287 -- bytes # Before patch.
[2018-03-29 02:09:08] ERROR 2: fprintf(): Too few arguments at [/Users/epriestley/dev/core/lib/phabricator/src/applications/drydock/management/DrydockManagementCommandWorkflow.php:60]
arcanist(head=experimental, ref.master=b8c9c385a7f5, ref.experimental=925c60e7b837), corgi(head=master, ref.master=6371578c9d32), instances(head=master, ref.master=d983b9517924), ledger(head=master, ref.master=4da4a24b8779), libcore(), phabricator(head=hoax1, ref.master=b586ee065a75, ref.hoax1=f8d7480bbdd1, custom=4), phutil(head=master, ref.master=1ad42491e44a), secure(head=master, ref.master=988cf9bd7958), services(head=master, ref.master=6b3fb8d8dd0a)
  #0 fprintf(resource, string) called at [<phabricator>/src/applications/drydock/management/DrydockManagementCommandWorkflow.php:60]
  #1 DrydockManagementCommandWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:441]
  #2 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:333]
  #3 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/drydock/drydock_control.php:21]
epriestley@orbital ~/dev/phabricator $ ./bin/drydock command --lease 76287 -- bytes # After patch.

!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
```

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19264
2018-03-28 16:19:55 -07:00
epriestley
b586ee065a Stop evaluating Herald rules when writing "someone mentioned this somewhere else." transactions
Summary: Ref T13114. See PHI510. Firing Herald on mentioned objects tends to feel arbitrary and can substantially slow down edits which mention many objects.

Test Plan: Mentioned tasks on other tasks; verified that the normal path is hit normally, the new Herald-free path is hit on the mentioned object, and both still work fine and show up in the timeline.

Maniphest Tasks: T13114

Differential Revision: https://secure.phabricator.com/D19263
2018-03-28 15:35:34 -07:00
epriestley
c5b244bfd0 Render directly embedded image data represented as a string in Jupyter notebooks
Summary: Depends on D19259. Ref T13105. Some examples represent image data as `["da", "ta"]` while others represent it as `"data"`. Accept either.

Test Plan: Rendered example notebooks with both kinds of images.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19260
2018-03-28 15:08:44 -07:00
epriestley
38999e25ac Support logged-out access to the document rendering endpoint
Summary: Ref T13105. Currently, logged-out users can't render documents via the endpoint even if they otherwise have access to the file.

Test Plan: Viewed a file as a logged-out user and re-rendered it via Ajax.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19258
2018-03-28 15:01:36 -07:00
epriestley
f583406ba9 Drop uniqueness constraint on PushEvent request ID
Summary: See <https://discourse.phabricator-community.org/t/pushing-to-mercurial-repository-fails/1275/1>. Mercurial may invoke hooks multiple times per push.

Test Plan: Pushed to Mercurial, saw key constraint failure.

Differential Revision: https://secure.phabricator.com/D19257
2018-03-26 07:02:15 -07:00
epriestley
bba1b185f8 Improve minor client behaviors for document rendering
Summary:
Ref T13105. This adds various small client-side improvements to document rendering.

  - In the menu, show which renderer is in use.
  - Make linking to lines work.
  - Make URIs persist information about which rendering engine is in use.
  - Improve the UI feedback for transitions between document types.
  - Load slower documents asynchronously by default.
  - Discard irrelevant requests if you spam the view menu.

Test Plan: Loaded files, linked to lines, swapped between modes, copy/pasted URLs.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19256
2018-03-23 14:09:31 -07:00
epriestley
4906364751 Add a JSON document rendering engine
Summary: Depends on D19254. This engine just formats JSON files in a nicer, more readable way.

Test Plan: Looked at some JSON files, saw them become formatted nicely.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Differential Revision: https://secure.phabricator.com/D19255
2018-03-23 12:29:05 -07:00
epriestley
d2727d24da Add an abstract "Text" document engine and a "Source" document engine
Summary: Ref T13105. Allow normal text files to be rendered as documents, and add a "source code" rendering engine.

Test Plan: Viewed some source code.

Reviewers: mydeveloperday

Reviewed By: mydeveloperday

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19254
2018-03-23 12:28:43 -07:00
epriestley
cbf3d3c371 Add a very rough, proof-of-concept Jupyter notebook document engine
Summary:
Depends on D19252. Ref T13105. This very roughly renders Jupyter notebooks.

It's probably better than showing the raw JSON, but not by much.

Test Plan:
  - Viewed various notebooks with various cell types, including markdown, code, stdout, stderr, images, HTML, and Javascript.
  - HTML and Javascript are not live-fired since they're wildly dangerous.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19253
2018-03-23 07:14:45 -07:00
epriestley
fb4ce851c4 Add a PDF document "rendering" engine
Summary:
Depends on D19251. Ref T13105. This adds rendering engine support for PDFs.

It doesn't actually render them, it just renders a link which you can click to view them in a new window. This is much easier than actually rendering them inline and at least 95% as good most of the time (and probably more-than-100%-as-good some of the time).

This makes PDF a viewable MIME type by default and adds a narrow CSP exception for it. See also T13112.

Test Plan:
  - Viewed PDFs in Files, got a link to view them in a new tab.
  - Clicked the link in Safari, Chrome, and Firefox; got inline PDFs.
  - Verified primary CSP is still `object-src 'none'` with `curl ...`.
  - Interacted with the vanilla lightbox element to check that it still works.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19252
2018-03-23 07:14:17 -07:00
epriestley
8b658706a8 Add a basic Remarkup document rendering engine
Summary:
Ref T13105. Although Markdown is trickier to deal with, we can handle Remarkup easily.

This may need some support for encoding options.

Test Plan: Viewed `.remarkup` files, got remarkup document presentation by default. Viewed other text files, got an option to render as remarkup.

Reviewers: avivey

Reviewed By: avivey

Subscribers: mydeveloperday, avivey

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19251
2018-03-23 07:07:50 -07:00
epriestley
df3c937dab Record lock timing information on PushEvents
Summary:
Depends on D19249. Ref T13109. Add timing information to the `PushEvent`:

  - `writeWait`: Time spent waiting for a write lock.
  - `readWait`: Time spent waiting for a read lock.
  - `hostWait`: Roughly, total time spent on the leaf node.

The primary goal here is to see if `readWait` is meaningful in the wild. If it is, that motivates smarter routing, and the value of smarter routing can be demonstrated by looking for a reduction in read wait times.

Test Plan: Pushed some stuff, saw reasonable timing values in the table. Saw timing information in "Export Data".

Maniphest Tasks: T13109

Differential Revision: https://secure.phabricator.com/D19250
2018-03-22 13:46:01 -07:00
epriestley
69bff489d4 Generate a random unique "Request ID" for SSH requests so processes can coordinate better
Summary:
Depends on D19247. Ref T13109. When we receive an SSH request, generate a random unique ID for the request. Then thread it down through the process tree.

The immediate goal is to let the `ssh-exec` process coordinate with `commit-hook` process and log information about read and write lock wait times. Today, there's no way for `ssh-exec` to interact with the `PushEvent`, but this is the most helpful place to store this data for users.

Test Plan: Made pushes, saw the `PushEvent` table populate with a random request ID. Exported data and saw the ID preserved in the export.

Maniphest Tasks: T13109

Differential Revision: https://secure.phabricator.com/D19249
2018-03-22 13:44:30 -07:00
epriestley
859b274970 Provide more information to users during git push while waiting for write locks
Summary:
Ref T13109. Make it slightly more clear what the scope of the write and read locks are, and slightly more clear that we're actively acquiring locks, not just sitting around waiting.

While waiting on another writer, show who we're waiting on so you can walk over to their desk and glare at them.

Test Plan:
Added `sleep(15)` after `willWrite()`. Pushed in two windows. Saw new, more informative messages. In the second window, saw the new guidance:

> # Waiting for hector to finish writing (on device "repo1.local.phacility.net" for 11s)...

Reviewers: asherkin

Reviewed By: asherkin

Subscribers: asherkin

Maniphest Tasks: T13109

Differential Revision: https://secure.phabricator.com/D19247
2018-03-22 13:42:18 -07:00
epriestley
6ed123e080 Propagate "unexpandable" PHIDs to feed notification recipient expansion
Summary:
See PHI466. Ref T13108. Somewhat recently, new rules were added so that "Resigning" from a revision takes you off the default recipient list, even if you're still a member of a project or package that is still a reviewer or subscriber.

However, these rules don't currently apply to the similar expansion which occurs in notifications. If you resign from a revision you may still get some notifications (just not email) if a package or project you're a member of is a reviewer or subscriber.

(Possibly these should eventually share more code, but just get things working for now.)

Test Plan:
  - Created a revision as A.
  - Added B as a reviewer.
  - Added a package B is an owner for as a reviewer.
  - As B, resigned. (Make sure B is also not an explicit subscriber.)
  - Commented on the revision as A.
    - Before: B is included in the expanded notification recipient list.
    - After: B is no longer included in the expanded notification recipient list.

Maniphest Tasks: T13108

Differential Revision: https://secure.phabricator.com/D19244
2018-03-21 11:55:52 -07:00
Tino Breddin
73b68bc2a6 Fix a possible count(null) in DifferentialRevisionActionTransaction
Summary:
This change prevents the following error when using PHP 7.2:

```
ERROR 2: count(): Parameter must be an array or an object that implements Countable at [/usr/local/lib/php/phabricator/src/applications/differential/xaction/DifferentialRevisionActionTransaction.php:132]
```

A similar issue was fixed in D18964

Test Plan: Tested in a live system.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19242
2018-03-21 07:39:34 -07:00
epriestley
4aafce6862 Add filesize limits for document rendering engines and support partial/complete rendering
Summary:
Depends on D19238. Ref T13105. Give document engines some reasonable automatic support for degrading gracefully when someone tries to hexdump a 100MB file or similar.

Also, make "Video" sort above "Audio" for files which could be rendered either way.

Test Plan: Viewed audio, video, image, and other files. Adjusted limits and saw full, partial, and fallback/error rendering.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19239
2018-03-19 15:18:34 -07:00
epriestley
f646153f4d Add an async driver for document rendering and a crude "Hexdump" document engine
Summary: Depends on D19237. Ref T13105. This adds a (very basic) "Hexdump" engine (mostly just to have a second option to switch to) and a selector for choosing view modes.

Test Plan: Viewed some files, switched between audio/video/image/hexdump.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19238
2018-03-19 15:18:05 -07:00
epriestley
01f22a8d06 Roughly modularize document rendering in Files
Summary:
Ref T13105. This change begins modularizing document rendering. I'm starting in Files since it's the use case with the smallest amount of complexity.

Currently, we hard-coding the inline rendering for images, audio, and video. Instead, use the modular engine pattern to make rendering flexible and extensible.

There aren't any options for switching modes yet and none of the renderers do anything fancy. This API is also probably very unstable.

Test Plan: Viewwed images, audio, video, and other files. Saw reasonable renderings, with "nothing can render this" for any other file type.

Maniphest Tasks: T13105

Differential Revision: https://secure.phabricator.com/D19237
2018-03-19 15:17:04 -07:00
epriestley
c5e4bd8187 Fix some minor errors (DarkConsole warning, unstable Ferret sort)
Summary:
DarkConsole could warn when "Analyze Query Plans" was not active.

`msort()` is not stable, so Ferret results with similar relevance could be returned out-of-order.

Test Plan: Saw fewer traces and more-stable result ordering.

Differential Revision: https://secure.phabricator.com/D19236
2018-03-18 15:12:25 -07:00
epriestley
7e43b74055 Give all commands from DiffusionCommandEngine a default 15 minute timeout
Summary:
Ref T13108. See PHI364. See the task and issue for discussion.

If a `git fetch` during synchronization hangs, the whole node currently hangs. While the causes of a `git fetch` hang aren't clear, we don't expect synchronization to ever reasonably take more than 15 minutes, so add a default timeout.

Test Plan: Will deploy and observe; this is difficult to reproduce or test directly.

Maniphest Tasks: T13108

Differential Revision: https://secure.phabricator.com/D19235
2018-03-16 17:22:03 -07:00
epriestley
fa6cd200e8 Reduce the severity of policy fatals when building the Harbormaster "build status" element
Summary:
See PHI430. Ref T13102. When the "Build Status" element raises a policy exception, we currently fatal the whole page rather than raising a normal policy error.

This is because the policy check happens very late in page construction, long after we've made the decision to show the page instead of a policy error, and gets treated as a rendering error.

In turn, this is because the rendering is event-based rather than using a more modern Engine + EngineExtension sort of construct, so some of the actual logic runs way later than it should.

Since unwinding all of this isn't trivial and the current behavior is materially bad, limit the damage here for now by just hiding the element. See T13088 for notes on handling this in a more nuanced way in the future.

Test Plan:
  - Created a revision visible to "Public".
  - Ran a build against it with a build plan visible to "All Users".
  - Viewed revision in an incognito window.
    - Before patch: Policy fatal with a red "rendering phase" error box.
    - After patch: Mostly-functional page with a missing "Build Status" element.
  - Viewed revision as a user with a normal session, saw the same UI before and after the change.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13102

Differential Revision: https://secure.phabricator.com/D19232
2018-03-16 13:27:57 -07:00
epriestley
c216fd4072 Allow projects to be queried by slug in "project.search"
Summary:
Ref T13102. See PHI461. An install is interested in querying projects by slug.

I think I omitted this capability originally only because we're not consistent about what slugs are called (they are "Slugs" internally, but "Hashtags" in the UI).

However, this ship has sort of already sailed because the results have a "slug" field. Just expose this as "slugs" for consistency with the existing API field and try to smooth thing over with a little documentation hint.

Test Plan: Queried for projects by slug, got the desired results back.

Maniphest Tasks: T13102

Differential Revision: https://secure.phabricator.com/D19230
2018-03-16 13:08:40 -07:00
epriestley
2b5c73fc3d In "Analyze Query Plans" mode, collect service call stack traces in DarkConsole
Summary: Ref T13106. When profiling service queries, there's no convenient way to easily get a sense of why a query was issued. Add a mode to collect traces for each query to make this more clear. This is rough, but works well enough to be useful.

Test Plan: Clicked "Analyze Query Plans", got stack traces for each service call.

Maniphest Tasks: T13106

Differential Revision: https://secure.phabricator.com/D19221
2018-03-14 20:34:34 -07:00
epriestley
af8269d2fb Allow draft revisions to be commandeered
Summary:
See PHI457. There's no real reason not to allow this, it just wasn't clear if it was useful. See D18626.

An install had a user `arc diff` and then sprint out the door to take a very long vacation before the builds finished. One failed, so the revision is stuck as a draft forever. This seems like a reasonable motivation for allowing "Commandeer".

Test Plan: Successfully commandeered a draft.

Differential Revision: https://secure.phabricator.com/D19228
2018-03-14 14:04:31 -07:00
epriestley
f348721aed When loading project membership to evaluate the "Subscribers" policy, use the ominipotent viewer
Summary: See PHI448. Ref T13106. The current implementation here can end up in an infinite stack if, e.g., a project uses "Visible to: Subscribers".

Test Plan: Will push.

Maniphest Tasks: T13106

Differential Revision: https://secure.phabricator.com/D19226
2018-03-14 12:59:31 -07:00
epriestley
ce6e020d5d Don't make an expensive, unused call to test if a viewer can reassign a task
Summary: Depends on D19224. Ref T13106. Computing this is expensive and the value is not used. This came from D15432, but we never actually shipped that feature.

Test Plan: Saw local query cost drop from 139 to 110 with no change in functionality. Grepped for removed symbols.

Maniphest Tasks: T13106

Differential Revision: https://secure.phabricator.com/D19225
2018-03-14 12:46:27 -07:00
epriestley
d80a53abcc Skip loading file transform sources when we know a file is not transformed
Summary:
Depends on D19223. Ref T13106. When we're loading a file, we currently test if it's a transformed version of another file (usually, a thumbnail) and apply policy behavior if it is.

We know that builtins and profile images are never transforms and that the policy behavior for these files doesn't matter anyway. Skip loading transforms for these files.

Test Plan: Saw local queries drop from 146 to 139 with no change in behavior.

Maniphest Tasks: T13106

Differential Revision: https://secure.phabricator.com/D19224
2018-03-14 12:45:06 -07:00
epriestley
31bd3679f0 Skip loading attached objects for files when we know the file is visible
Summary:
Depends on D19222. Ref T13106. We currently execute an edge query (and possibly an object query) when loading builtin files, but this is never necessary because we know these files are always visible.

Instead, skip this logic for builtin files and profile image files; these files have global visibility and will never get a different policy result because of file attachment information.

(In theory, we could additionally skip this for files with the most open visibility policy or some other trivially visible policy like the user's PHID, but we do actually care about the attachment data some of the time.)

Test Plan: Saw queries drop from 151 to 145 on local test page. Checked file attachment data in Files, saw it still working correctly.

Maniphest Tasks: T13106

Differential Revision: https://secure.phabricator.com/D19223
2018-03-14 12:36:46 -07:00
epriestley
49e6358fce Bulk load builtin project default profile images
Summary: Depends on D19221. Ref T13106. When we fall back to default profile images for projects, bulk load them instead of doing individual queries.

Test Plan: Saw local task drop from 199 queries to 151 queries with the same actual outcome. Saw custom and default profile images on the project list page.

Maniphest Tasks: T13106

Differential Revision: https://secure.phabricator.com/D19222
2018-03-14 12:35:15 -07:00
epriestley
dc7e40ff3f Fix the DarkConsole inline error log stack trace expansion behavior for Content-Security-Policy
Summary:
See PHI451. Ref T13102. DarkConsole uses an ancient inline "onclick" handler to expand the stack traces for errors.

The new Content-Security-Policy prevents this from functioning.

Replace this with a more modern behavior-driven action instead.

Test Plan:
  - Clicked some errors in DarkConsole, saw stack traces appear.
  - Grepped for `onclick` and `jsprintf()` to see if I could find any more of these, but came up empty.

Maniphest Tasks: T13102

Differential Revision: https://secure.phabricator.com/D19218
2018-03-13 16:45:20 -07:00
epriestley
dfd8b0225b Add a UI element for reviewing older generations of Harbormaster builds
Summary:
See PHI446. Ref T13088. Currently, there's no way to access older generations of a build unless you know the secret `?g=1` URI magic.

When a build has multiple generations, show a history table and let users click to see older run information.

This is currently very basic. It would be nice to show when each generation started, who started/restarted it, and what the build status was at the time the build was restarted. There's currently no convenient source for this information so just add a bare-bones, working version of this for now.

Test Plan:
Viewed pending, single-run and multi-restart builds. Saw table on builds with more than one generation. Clicked table entries to see different build data.

{F5471160}

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19217
2018-03-13 16:15:11 -07:00
epriestley
0bf8e33bb6 Issue setup guidance recommending MySQLi and MySQL Native Driver
Summary:
Fixes T12994. We need `MYSQLI_ASYNC` to implement client-side query timeouts, and we need MySQLi + MySQL Native Driver to get `MYSQLI_ASYNC`.

Recommend users install MySQLi and MySQL Native Driver if they don't have them. These are generally the defaults and best practice anyway, but Ubuntu makes it easy to use the older stuff.

All the cases we're currently aware of stem from `apt-get install php5-mysql` (which explicitly selects the non-native driver) so issue particular guidance about `php5-mysqlnd`.

Test Plan:
  - Faked both issues locally, reviewed the text.
  - Will deploy to `secure`, which currently has the non-native driver.

Maniphest Tasks: T12994

Differential Revision: https://secure.phabricator.com/D19216
2018-03-13 12:38:09 -07:00
epriestley
2b19f91936 Allow Doorkeeper references to have multiple display variations (full, short, etc.)
Summary:
Ref T13102. An install has a custom rule for bridging JIRA references via Doorkeeper and would like to be able to render them as `JIRA-123` instead of `JIRA JIRA-123 Full JIRA title`.

I think it's reasonable to imagine future support upstream for `JIRA-123`, `{JIRA-123}`, and so on, although we do not support these today. We can take a small step toward eventual support by letting the rendering pipeline understand different view modes.

This adds an optional `name` (the default text rendered before we do the OAuth sync) and an optional `view`, which can be `short` or `full`.

Test Plan:
I tested this primarily with Asana, since it's less of a pain to set up than JIRA. The logic should be similar, hopefully.

I changed `DoorkeeperAsanaRemarkupRule` to specify `name` and `view`, e.g `'view' => (mt_rand(0, 1) ? 'short' : 'full')`. Then I made a bunch of Asana references in a comment and saw them randomly go short or long.

Maniphest Tasks: T13102

Differential Revision: https://secure.phabricator.com/D19215
2018-03-13 11:29:52 -07:00
epriestley
a4a390fe2d Use "-dispose background" to improve reassembly of GIFs with transparency
Summary:
Fixes T5741. We break GIFs apart with "-coalesce" which completely rasterizes each frame, but stitch them back together without specifying "-dispose".

This produces the default "-dispose none" behavior, which causes GIF frames to "pile up" if they contain transparency.

Instead, use "-dispose background" so that the previous frame is erased before each new frame is drawn.

Test Plan: See T5741 for additional details.

Maniphest Tasks: T5741

Differential Revision: https://secure.phabricator.com/D19214
2018-03-13 09:19:53 -07:00
epriestley
598d0c04e7 When computing the "Subscribers" policy, use materialized membership
Summary:
Fixes T13104. The "Subscribers" policy implementation still uses older logic to query project membership and misses parent projects and milestones which a user is a member of.

Instead of doing an edge query for explicit membership, use a project query to find all projects the viewer belongs to.

Test Plan:
  - Created a parent project A.
  - Created a subproject B.
  - As Bailey, created a task with "Visible To: Bailey, Subscribers".
  - Added parent project A as a task subscriber.

Then:

  - As Alice, verified I could not see the task.
  - As Alice, joined subproject B.
    - Before patch: still unable to see the task.
    - After patch: can see the task.
  - Removed parent project A as a subscriber, verified I could no longer see the task.

Maniphest Tasks: T13104

Differential Revision: https://secure.phabricator.com/D19213
2018-03-13 08:30:03 -07:00
epriestley
1e93b49b1b Allow custom actions in Differential to explicitly override "accept" stickiness
Summary:
See PHI431. Ref T13102. An install is interested in a custom "non-sticky" accept action, roughly.

On the one hand, this is a pretty hacky patch. However, I suspect it inches us closer to T731, and I'm generally comfortable with exploring the realms of "Accept Next Update", "Unblock Without Accepting", etc., as long as most of it doesn't end up enabled by default in the upstream.

Test Plan:
  - Accepted and updated revisions normally, saw accepts respect global stickiness.
  - Modified the "Accept" action to explicitly be unsticky, saw nonsticky accept behavior after update.

Maniphest Tasks: T13102

Differential Revision: https://secure.phabricator.com/D19211
2018-03-12 17:10:43 -07:00
epriestley
df8d4dff67 Raise a warning when mentioning a user in a comment on a draft revision
Summary: See PHI433. Ref T13102. Users in the wild have mixed expecations about exactly what "draft" means. Recent changes have tried to make behavior more clear. As part of clarifying messaging, make it explicit that `@mention` does not work on drafts by showing users a warning when they try to `@mention` a user.

Test Plan:
  - Mentioned users on drafts, got a warning.
  - Posted normal comments on drafts, no warning.
  - Posted normal/mention comments on non-drafts, no warning.

Maniphest Tasks: T13102

Differential Revision: https://secure.phabricator.com/D19210
2018-03-12 17:03:14 -07:00
epriestley
3c4f31e4b9 Dynamically composite favicons from customizable sources
Summary: Ref T13103. Make favicons customizable, and perform dynamic compositing to add marker to indicate things like "unread messages".

Test Plan: Viewed favicons in Safari, Firefox and Chrome. With unread messages, saw pink dot composited into icon.

Maniphest Tasks: T13103

Differential Revision: https://secure.phabricator.com/D19209
2018-03-12 15:28:41 -07:00
epriestley
9d0cf3c8b8 Before anyone notices, break the API
Summary: See PHI439. Use slightly richer "dominion" return values for consistency.

Test Plan: Fetched results with `owners.search` API method.

Differential Revision: https://secure.phabricator.com/D19208
2018-03-09 12:21:18 -08:00
epriestley
3e992c6713 Add audit, review, and dominion information to "owners.search" API method
Summary:
See PHI439. This fills in additional information about Owners packages.

Also removes dead `primaryOwnerPHID`.

Test Plan: Called `owners.search` and reviewed the results. Grepped for `primaryOwnerPHID`.

Differential Revision: https://secure.phabricator.com/D19207
2018-03-09 12:11:13 -08:00
epriestley
1763b516b1 Fix missing parameter in parent call for Differential button text
Summary: See <https://discourse.phabricator-community.org/t/openning-any-differential-fails-with-undefined-variable-object/1216/1>.

Test Plan: Loaded any //non//-draft revision.

Differential Revision: https://secure.phabricator.com/D19205
2018-03-09 05:22:57 -08:00
epriestley
2de06a5375 Add some more UI reminder text about draft revisions
Summary: See PHI433. This beefs up reminder texts for drafts a little bit since some users in the wild aren't always seeing/remembering the existing, fairly subtle hints.

Test Plan: Created a reivsion with `--draft`, viewed it, saw richer reminders.

Differential Revision: https://secure.phabricator.com/D19204
2018-03-08 12:07:40 -08:00
epriestley
10b3ddf426 Possibly fix memes in email
Summary:
Depends on D19201. Ref T13101. This likely produces relatively stable-ish image references for email.

They currently TTL after 30 days but this makes the jokes more exclusive and special so it's a feature, not a bug.

Test Plan: I'm just going to test this in production because I'm a ninja superstar developer.

Maniphest Tasks: T13101

Differential Revision: https://secure.phabricator.com/D19203
2018-03-08 11:09:21 -08:00
epriestley
a3d282d33e Somewhat improve meme transform code so it is merely very bad
Summary: Depends on D19200. Fixes T5258. Ref T13101. Attempt to simplify and modernize this code and improve error handling.

Test Plan: did real hard dank memes

Maniphest Tasks: T13101, T5258

Differential Revision: https://secure.phabricator.com/D19201
2018-03-08 11:08:55 -08:00
epriestley
c7408f2797 PhabricatorMemeEngine HA HA HA HA
Summary:
Depends on D19198. Ref T13101. Ref T5258. Pull compositing logic out of the `Controller`.

This is moving toward fixing memes in email.

Test Plan: Used new and old memes. Used API memes.

Maniphest Tasks: T13101, T5258

Differential Revision: https://secure.phabricator.com/D19200
2018-03-08 11:06:52 -08:00
epriestley
a099a06265 Remove some old image transform code with no callsites
Summary: Ref T13101. Ref T5258. This old image transform code no longer has callsites.

Test Plan: Grepped for removed methods, no hits.

Maniphest Tasks: T13101, T5258

Differential Revision: https://secure.phabricator.com/D19198
2018-03-08 11:04:53 -08:00
epriestley
fc1ee20efe Support repository query by short name in Diffusion
Summary: See PHI432. Ref T13099. Short names never made it to the UI/API but seem stable now, so support them.

Test Plan: {F5465173}

Maniphest Tasks: T13099

Differential Revision: https://secure.phabricator.com/D19202
2018-03-08 10:55:24 -08:00
epriestley
98cac2cc29 Always serve "{meme ...}" from the CDN domain, never from the primary domain
Summary:
Ref T13101. This is a minimal change to make "{meme ...}" work with the new Content-Security-Policy by using an Ajax request to generate the image and then swapping the source on the client.

This could be much cleaner (see T5258, etc).

Test Plan: Used `{meme, src=cat6, above=i am, below=cat}`, chuckled completely unironically.

Maniphest Tasks: T13101

Differential Revision: https://secure.phabricator.com/D19196
2018-03-08 07:47:02 -08:00
epriestley
6095d88998 Don't require prototypes for "{image ...}"
Summary: Depends on D19194. Fixes T4190. This should be in good-enough shape now to release and support more generally.

Test Plan: Used `{image ...}` in remarkup.

Maniphest Tasks: T4190

Differential Revision: https://secure.phabricator.com/D19195
2018-03-08 07:04:23 -08:00
epriestley
b30535a36f When rendering "{image ...}" images, check the cache and just render a direct "<img />" tag if possible
Summary: Depends on D19193. Ref T13101. Fixes T4190. Before we render a fancy AJAX placeholder, check if we already have a valid cache for the image. If we do, render a direct `<img />` tag. This is a little cleaner and, e.g., avoids flicker in Safari, at least.

Test Plan: Rendered `{image ...}` rules in remarkup with new and existing URIs.

Maniphest Tasks: T13101, T4190

Differential Revision: https://secure.phabricator.com/D19194
2018-03-08 07:03:55 -08:00
epriestley
9d3a722eb1 When proxying an "{image ...}" image fails, show the user an error message
Summary:
Depends on D19192. Ref T4190. Ref T13101. Instead of directly including the proxy endpoint with `<img src="..." />`, emit a placeholder and use AJAX to make the request. If the proxy fetch fails, replace the placeholder with an error message.

This isn't the most polished implementation imaginable, but it's much less mysterious about errors.

Test Plan: Used `{image ...}` for valid and invalid images, got images and useful error messages respectively.

Maniphest Tasks: T13101, T4190

Differential Revision: https://secure.phabricator.com/D19193
2018-03-08 07:03:26 -08:00
epriestley
01bbd71b96 Separate the "{img ...}" remarkup rule into separate parse and markup phases
Summary:
Ref T13101. Ref T4190. This rule is currently single-phase but I'd like to check for a valid proxied image in cache already and just emit an `<img ... />` tag pointing at it if we have one.

To support batching these lookups, split the rule into a parse phase (where we extract URIs) and a markup phase (where we build tags).

Test Plan: Used `{img ...}` in Remarkup with no apparent behavioral changes. (This change should do nothing on its own.)

Maniphest Tasks: T13101, T4190

Differential Revision: https://secure.phabricator.com/D19192
2018-03-08 07:02:59 -08:00
epriestley
a4cc1373d3 Use a tokenizer, not a gigantic poorly-ordered "<select />", to choose repositories in Owners
Summary: Depends on D19190. Fixes T12590. Ref T13099. Replaces the barely-usable, gigantic, poorly ordered "<select />" control with a tokenizer. Attempts to fix various minor issues.

Test Plan:
  - Edited paths: include/exclude paths, from different repositories, different actual paths.
  - Used "Add New Path" to add rows, got repository selector prepopulated with last value.
  - Used "remove".
  - Used validation typeahead, got reasonable behaviors?

The error behavior if you delete the repository for a path is a little sketchy still, but roughly okay.

Maniphest Tasks: T13099, T12590

Differential Revision: https://secure.phabricator.com/D19191
2018-03-07 20:57:24 -08:00
epriestley
b41a0e6ddd Fix broken suggestion/validation for Owners paths in repositories with short names
Summary:
Depends on D19189. Ref T12590. The "validate" and "complete" endpoints for this UI could incorrectly return redirect responses. These aren't critical to the behavior of Owners, but they're nice to have, and shouldn't redirect.

Instead, skip the canonicalizing redirect for AJAX requests.

Test Plan: Edited Owners paths in a repository with a short name, got completion/validation again.

Maniphest Tasks: T12590

Differential Revision: https://secure.phabricator.com/D19190
2018-03-07 18:31:25 -08:00
epriestley
ab0ac7f61b Remove very old "owners-default-path" code from Owners
Summary: Ref T12590. This is ancient code which was used to prefill `/trunk/tfb/www/` or similar at Facebook. I don't think it ever had a UI and no install has asked for this feature since 2011.

Test Plan: Grepped for affected symbols, edited paths in Owners.

Maniphest Tasks: T12590

Differential Revision: https://secure.phabricator.com/D19189
2018-03-07 18:25:27 -08:00
epriestley
c6a042b59a Correct line highlighting behavior in Diffusion
Summary: See <https://discourse.phabricator-community.org/t/line-highlighting-in-diffusion-breaks-url/1207>. Ref T13088. This was disrupted by changes for the new Harbormaster build logs and now needs an explicit base URI.

Test Plan: Clicked lines and dragged across line ranges in Diffusion, observed correct URI behavior.

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19187
2018-03-07 07:07:06 -08:00
epriestley
28854ae812 Return a integer JSON type from "*.edit" endpoints for the object ID
Summary: See PHI425. See T12678. This should be an integer, but may be a string.

Test Plan: Called `differential.revision.edit`, observed integer in result instead of string.

Differential Revision: https://secure.phabricator.com/D19186
2018-03-07 06:27:35 -08:00
epriestley
9462f8aa89 Remove client OAuth redirect code which was only partially cleaned up
See T13099. I took a different approach here but didn't fully clean up
the old one.
2018-03-06 20:41:13 -08:00
epriestley
516aaad341 Use "pathIndex" in some owners package queries to improve query plans
Summary: Depends on D19184. Ref T11015. Now that we have a digest index column, we can improve some of the queries a bit.

Test Plan:
  - Ran queries from revision pages before and after with and without EXPLAIN.
  - Saw the same results with much better EXPLAIN plans.
  - Fragment size is now fixed at 12 bytes per fragment, so we can shove more of them in a single query.

Maniphest Tasks: T11015

Differential Revision: https://secure.phabricator.com/D19185
2018-03-06 20:33:18 -08:00
epriestley
df1e9ce646 Treat Owners paths like "/src/backend" and "/src/backend/" identically
Summary:
Depends on D19183. Ref T11015. Currently, adding a trailing slash works great and omitting it mysteriously doesn't work.

Store a normalized version with an unconditional trailing slash for the lookup logic to operate on, and a separate display version which tracks what the user actually typed.

Test Plan:
  - Entered "/src/main.c", "/src/main.c/", saw them de-duplicate.
  - Entered "/src/main.c", saw it stay that way in the UI but appear as "/src/main.c/" internally.
  - Added a rule for "/src/applications/owners" (no slash), created a revision touching paths in that directory, saw Owners fire for it.
  - Changed the display value of a path only ("/src/main.c" to "/src/main.c/"), saw the update reflected in the UI without any beahvioral change.

Maniphest Tasks: T11015

Differential Revision: https://secure.phabricator.com/D19184
2018-03-06 20:31:46 -08:00
epriestley
adde4089b4 Allow owners paths to be arbitrarily long and add storage for display paths
Summary:
Depends on D19182. Ref T11015. This changes `path` from `text255` to `longtext` because paths may be arbitrarily long.

It adds `pathDisplay` to prepare for display paths and storage paths having different values. For now, `pathDisplay` is copied from `path` and always has the same value.

Test Plan:
  - Ran migration, checked database for sanity (all `pathDisplay` and `path` values identical).
  - Added new paths, saw `pathDisplay` and `path` get the same values.
  - Added an unreasonably enormous path with far more than 255 characters.

Maniphest Tasks: T11015

Differential Revision: https://secure.phabricator.com/D19183
2018-03-06 20:31:22 -08:00
epriestley
8cb273a053 Add a unique key to OwnersPath on "<packageID, repositoryPHID, pathIndex>"
Summary:
Depends on D19181. Ref T11015. This nukes duplicates from the table if they exist, then adds a unique key.

(Duplicates should not exist and can not be added with any recent version of the web UI.)

Test Plan:
  - Tried to add duplicates with web UI, didn't have any luck.
  - Explicitly added duplicates with manual `INSERT`s.
  - Viewed packages in web UI and saw duplicates.
  - Ran migrations, got a clean purge and a nice unique key.
  - There's still no way to actually hit a duplicate key error in the UI (unless you can collide hashes, I suppose), this is purely a correctness/robustness change.

Maniphest Tasks: T11015

Differential Revision: https://secure.phabricator.com/D19182
2018-03-06 20:30:59 -08:00
epriestley
1bf4422c74 Add and populate a pathIndex column for OwnersPath
Summary: Ref T11015. This supports making path names arbitrarily long and putting a proper unique key on the table.

Test Plan:
  - Migrated, checked database, saw nice digested indexes.
  - Edited a package, saw new rows update with digested indexes.

Maniphest Tasks: T11015

Differential Revision: https://secure.phabricator.com/D19181
2018-03-06 20:30:33 -08:00
epriestley
d14a0f4787 Add "All" and "With Non-Owner Author" options for all Owners Package autoreview rules
Summary: Ref T13099. See PHI424. Fixes T11664. Several installs are interested in having these behaviors available in Owners by default and they aren't difficult to provide, it just makes the UI kind of messy. But I think there's enough general interest to justify it, now.

Test Plan: Created a package which owns "/" with a "With Non-Owner Author" review rule which I own. Created a revision, no package reviewer. Changed rule to "All", updated revision, got package reviewer.

Maniphest Tasks: T13099, T11664

Differential Revision: https://secure.phabricator.com/D19180
2018-03-06 19:01:58 -08:00
epriestley
e57dbcda33 Hide "abraham landed Dxyz irresponsibly" stories from feed
Summary:
Ref T13099. Ref T12787. See PHI417. Differential has new "irresponsible" warnings in the timeline somewhat recently, but these publish feed stories that don't link to the revision or have other relevant details, so they're confusing on the balance.

These have a high strength so they render on top, but we actually just want to hide them from the feed and let "abraham closed Dxyz by committing rXzzz." be the primary story.

Modularize things more so that we can get this behavior. Also, respect `shouldHideForFeed()` at display time, not just publishing time.

Test Plan: Used `bin/differential attach-commit` on a non-accepted revision to "irresponsibly land" a revision. Verified that feed story now shows "closed by commit" instead of "closed irresponsibly".

Maniphest Tasks: T13099, T12787

Differential Revision: https://secure.phabricator.com/D19179
2018-03-06 17:48:03 -08:00
epriestley
573bf15124 Provide a more tailored error message when a Herald rule fails because of PCRE limits
Summary: Ref T13100. Since rules may begin failing for PRCE configuration reasons soon, provide a more complete explanation of possible causes in the UI.

Test Plan: Faked this, hit it via test console, saw explanation in web UI.

Maniphest Tasks: T13100

Differential Revision: https://secure.phabricator.com/D19178
2018-03-06 12:18:58 -08:00
epriestley
dbccfb234f Perform a client-side redirect after OAuth server authorization
Summary:
Ref T13099. See that task for discussion. Chrome is unhappy with an MFA form submitting to an endpoint which redirects you to an OAuth URI.

Instead, do the redirect entirely on the client.

Chrome's rationale here isn't obvious, so we may be able to revert this at some point.

Test Plan: Went through the OAuth flow locally, was redirected on the client. Will verify in production.

Maniphest Tasks: T13099

Differential Revision: https://secure.phabricator.com/D19177
2018-03-06 12:18:27 -08:00
epriestley
f392896209 Return commit information for Revision "close" and "update" transactions over the Conduit API
Summary: Depends on D19175. Ref T13099. This fills in "close" and "update" transactions so that they show which commit(s) caused the action.

Test Plan: Used `transaction.search` to query some revisions, saw commit PHID information.

Maniphest Tasks: T13099

Differential Revision: https://secure.phabricator.com/D19176
2018-03-06 09:12:59 -08:00
epriestley
743d1ac426 Mostly modularize the Differential "update" transaction
Summary: Ref T13099. Move most of the "Update" logic to modular transactions

Test Plan: Created and updated revisions. Flushed the task queue. Grepped for `TYPE_UPDATE`. Reviewed update transactions in the timeline and feed.

Maniphest Tasks: T13099

Differential Revision: https://secure.phabricator.com/D19175
2018-03-06 09:10:32 -08:00
epriestley
44f0664d2c Add a "lock log" for debugging where locks are being held
Summary: Depends on D19173. Ref T13096. Adds an optional, disabled-by-default lock log to make it easier to figure out what is acquiring and holding locks.

Test Plan: Ran `bin/lock log --enable`, `--disable`, `--name`, etc. Saw sensible-looking output with log enabled and daemons restarted. Saw no additional output with log disabled and daemons restarted.

Maniphest Tasks: T13096

Differential Revision: https://secure.phabricator.com/D19174
2018-03-05 17:55:34 -08:00
epriestley
fd367adbaf Parameterize PhabricatorGlobalLock
Summary:
Ref T13096. Currently, we do a fair amount of clever digesting and string manipulation to build lock names which are less than 64 characters long while still being reasonably readable.

Instead, do more of this automatically. This will let lock acquisition become simpler and make it more possible to build a useful lock log.

Test Plan: Ran `bin/repository update`, saw a reasonable lock acquire and release.

Maniphest Tasks: T13096

Differential Revision: https://secure.phabricator.com/D19173
2018-03-05 15:30:27 -08:00
epriestley
5844952153 Show lint messages in deleted files on the left-hand side of the change
Summary:
See PHI416. If you raise a lint message in a deleted file, we don't render any text on the right hand side so the message never displays.

This is occasionally still legitimate/useful, e.g. to display a "don't delete this file" message. At least for now, show these messages on the left.

Test Plan: Posted a lint message on a deleted file via `harbormaster.sendmessage`, viewed revision, saw file expand with synthetic inline for lint.

Differential Revision: https://secure.phabricator.com/D19171
2018-03-04 09:14:10 -08:00
epriestley
2121f2dea6 Don't require project edit permission to create a project with members other than yourself
Summary: See PHI193. Previously, see similar D18763. Skip this legacy-style policy check when creating a project, since we know you can add members, even if the policy doesn't actually resolve in your favor.

Test Plan:
  - Created a project with edit policy "Members of project" and myself, plus any other user (so the code goes down this path, not the "join/leave" path) as members.

Differential Revision: https://secure.phabricator.com/D19169
2018-03-01 18:46:03 -08:00
epriestley
14fe941c34 Reduce the cost of generating default user profile images
Summary:
See PHI413. You can pre-generate these with `bin/people profileimage --all`, but they're needlessly expensive to generate.

Streamline the workflow and cache some of the cacheable parts to reduce the generation cost.

Test Plan:
  - Ran `bin/people profileimage --all` and saw cost drop from {nav 15.801s > 4.839s}.
  - Set `defaultProfileImagePHID` to `NULL` in `phabricator_user.user` and purged caches with `bin/cache purge --all`.
  - Loaded user directory.
  - Saw default images regenerate relatively quickly.

Differential Revision: https://secure.phabricator.com/D19168
2018-03-01 16:53:17 -08:00
epriestley
1f40e50f7e Improve live Harbormaster log follow behaviors
Summary:
Depends on D19166. Ref T13088. When the user scrolls away from a followed log, break the focus lock.

Let users stop following a live log.

Show when lines are added more clearly.

Don't refresh quite as quickly give users a better shot at clicking the stop button.

These behaviors can probably be refined but are at least more plausible and less actively user-hostile than the first version of this behavior was.

Test Plan: Used `write-log --rate` to write a large log slowly. Clicked "Follow Log", followed for a bit. Scrolled away, still got live updates but no more scroll lock. Clicked stop, no more updates.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19167
2018-03-01 13:11:22 -08:00
epriestley
4e91ad276d Prevent copying Harbormaster build log line numbers with CSS psuedocontent instead of ZWS
Summary:
Depends on D19165. Ref T13088. Currently, in other applications, we use Zero Width Spaces and Javascript "copy" listeners to prevent line numbers from being copied. This isn't terribly elegant.

Modern browsers support a second approach: using psuedo-elements with `content`. Try this in Harbormaster since it's conceptually cleaner, at least. One immediate drawback is that Command-F can't find this text either.

Test Plan: In Safari, Chrome and Firefox, highlighted ranges of lines and copy/pasted text. Got just text (no line numbers) in all cases.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19166
2018-03-01 13:03:40 -08:00
epriestley
73619c4643 Share the Paste line highlighting behavior for Harbormaster build logs
Summary: Depends on D19164. Ref T13088. Now that the JS behaviors are generic, use them on the Harbormaster standalone page.

Test Plan: Clicked lines and dragged across line ranges. Reloaded pages. Saw expected highlighting behavior in the client and on the server across reloads.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19165
2018-03-01 12:57:30 -08:00
epriestley
49af4165bc Support rendering arbitrary sections in the middle of a Harbormaster build log so links to line 3500 work
Summary:
Depends on D19162. Ref T13088. When a user links to `$1234`, we need to render a default view of the log with a piece at the head, a piece at the end, and a piece in the middle.

We also need to figure out the offset for line 1234, or multiple offsets for "1234-2345".

Since the logic views/reads mostly anticipated this it isn't too much of a mess, although there are a couple of bugs this exposes with view specifications that use combinations of parameters which were previously impossible.

Test Plan: Viewed a large log with no line marker. Viewed `$1`. Viewed `$end`. Viewed `$35-40`, etc. Expanded context around logs.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19163
2018-03-01 11:18:21 -08:00
epriestley
4466402c5a Move Paste line range reading code into AphrontRequest
Summary: Ref T13088. This lifts the code for parsing "$x-y" line ranges in URIs into AphrontRequest so Diffusion, Paste, Harbormaster, etc., can share it.

Test Plan: Viewed lines, line ranges, no lines, negative line ranges, line ranges with 0, and extremely long line ranges in Paste.

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19162
2018-03-01 11:15:06 -08:00
epriestley
94d340fcff Include OAuth targets in "form-action" Content-Security-Policy
Summary:
Ref T4340. Some "Register/Login" and "Link External Account" buttons are forms which submit to third-party sites. Whitelist these targets when pages render an OAuth form.

Safari, at least, also prevents a redirect to a third-party domain after a form submission to the local domain, so when we first redirect locally (as with Twitter and other OAuth1 providers) we need to authorize an additional URI.

Test Plan: Clicked all my registration buttons locally without hitting CSP issues.

Maniphest Tasks: T4340

Differential Revision: https://secure.phabricator.com/D19159
2018-02-28 19:28:35 -08:00
epriestley
ab579f2511 Never generate file download forms which point to the CDN domain, tighten "form-action" CSP
Summary:
Depends on D19155. Ref T13094. Ref T4340.

We can't currently implement a strict `form-action 'self'` content security policy because some file downloads rely on a `<form />` which sometimes POSTs to the CDN domain.

Broadly, stop generating these forms. We just redirect instead, and show an interstitial confirm dialog if no CDN domain is configured. This makes the UX for installs with no CDN domain a little worse and the UX for everyone else better.

Then, implement the stricter Content-Security-Policy.

This also removes extra confirm dialogs for downloading Harbormaster build logs and data exports.

Test Plan:
  - Went through the plain data export, data export with bulk jobs, ssh key generation, calendar ICS download, Diffusion data, Paste data, Harbormaster log data, and normal file data download workflows with a CDN domain.
  - Went through all those workflows again without a CDN domain.
  - Grepped for affected symbols (`getCDNURI()`, `getDownloadURI()`).
  - Added an evil form to a page, tried to submit it, was rejected.
  - Went through the ReCaptcha and Stripe flows again to see if they're submitting any forms.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13094, T4340

Differential Revision: https://secure.phabricator.com/D19156
2018-02-28 17:20:12 -08:00
epriestley
afc98f5d5d Remove defunct "download" route in Files pointing to nonexistent controller
Summary:
Depends on D19154. Ref T13094. This controller was removed at some point and this route no longer works.

I plan to add a new `download/` route to let us tighten the `form-action` Content Security Policy.

Test Plan: Grepped for the route and controller, no hits.

Maniphest Tasks: T13094

Differential Revision: https://secure.phabricator.com/D19155
2018-02-28 17:19:52 -08:00
epriestley
f114b2dd7d When viewing a live build log, trap users in a small personal hell where nothing but slavish devotion to the log exists
Summary: Depends on D19152. Ref T13088. This adds live log tailing. It is probably not the final version of this feature because it prevents escape once you begin tailing a log.

Test Plan: Used `bin/harbormaster write-log --rate ...` to write a log slowly. Viewed it in the web UI. Clicked "Follow Log". Followed the log until the write finished, a lifetime later.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19153
2018-02-28 12:38:41 -08:00
epriestley
21ddfe442e Add a "--rate" flag to bin/harbormaster write-log to support testing live log streaming
Summary: Depends on D19151. Ref T13088. While dramatically less exciting than using `lolcat` and less general than `pv`, this should do the job adequately.

Test Plan: Piped a sizable log into `bin/harbormaster write-log` with `--rate 2048`, saw a progress bar. Loaded the log in the web UI and saw it grow as the page reloaded.

Reviewers: yelirekim

Reviewed By: yelirekim

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19152
2018-02-28 12:37:04 -08:00
epriestley
5a2213ef82 Provide API read access to Harbormaster build logs
Summary:
Depends on D19150. Ref T13088. Allow clients to retrieve information about build logs, including log data, over the API.

(To fetch log data, take the `filePHID` to `file.search`, then issue a normal GET against the URI. Use a `Content-Range` header to get part of the log.)

Test Plan: Ran `harbormaster.log.search`, got sensible-looking results.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19151
2018-02-28 12:36:03 -08:00
epriestley
dc6a66f7f4 Add a "(prototype)" link to the standalone build log on build pages
Summary: Depends on D19149. Ref T13088. Since the new log requires a bunch of log reprocessing, the cutover is going to require at least some time for installs to run migrations. Add a link in the UI to ease the transition, smooth over some behaviors a little, and fix a fetch issue where we'd request past the end of the log (since this is now enforced).

Test Plan: Viewed a traditional Harbormaster build, saw links to the new standalone log pages.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19150
2018-02-28 12:34:08 -08:00
epriestley
143033dc1f When showing a small piece of a Harbormaster build log, load a small piece of data instead of the entire log
Summary: Depends on D19148. Ref T13088. The new rendering always executes range requests for data it needs, and we can satisfy these requests by loading the smallest number of chunks which span that range.

Test Plan: Piped 50,000 lines of Apache log into Harbormaster, viewed it in the new UI, got sensible rendering times and a reasonable amount of data actually going over the wire.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19149
2018-02-28 12:32:26 -08:00
epriestley
985d499f50 Fix a Harbormaster build log issue where too few header lines were returned when expanding logs in the middle
Summary: Ref T13088. This variable bled through from an earlier loop and caused us to drop some of the lines in the middle.

Test Plan: Clicked "Show More", got an equal number of header and footer lines.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19148
2018-02-28 12:30:29 -08:00
epriestley
ada4f65565 Stop sending Phriction edit mail to the previous/current content author
Summary:
See <https://discourse.phabricator-community.org/t/previous-author-of-phriction-page-always-receives-mail-when-new-author-edits-the-page/1155>.

After the "send users email when they are removed from the reviewer list" change which includes old To/Cc on the list, Phriction now emails the previous editor.

This is weird and unintended. The behavior isn't necessarily always bad, but if it's sometimes desirable we should make it explicit.

For now, just get rid of the weird side effect.

Test Plan: Edited a Phriction page.

Differential Revision: https://secure.phabricator.com/D19147
2018-02-27 15:04:59 -08:00
epriestley
8cbfb386bb When evaluating the "Branches" Herald field from the test console, use the current viewer
Summary: Ref T13093. Depends on D19145. See PHI398. Previously, see D18933. This provides the current viewer to `ConduitCall` so that we don't try to use device credentials from unprivileged web hosts.

Test Plan: Evaluated the "Branches" field locally, saw an appropriate field value.

Maniphest Tasks: T13093

Differential Revision: https://secure.phabricator.com/D19146
2018-02-27 14:37:36 -08:00
epriestley
80fe382e3d Add "Committer's projects" and "Author's projects" fields to Herald commit rules
Summary:
Ref T13093. See PHI396. These are possibly somewhat niche, but reasonable to support and consistent with the existing "Pusher's projects".

Also relabel "Pusher's projects" and "Project tags" for consistency and, hopefully, clarity.

Test Plan:
  - Created new "commit" and "hook: commit content" Herald rules which run against "Author's projects" and "Committer's projects".
  - Test console'd the "Commit" rules.
  - Pushed through the "Hook" rule.
  - In all cases, saw fields populate appropriately.

Maniphest Tasks: T13093

Differential Revision: https://secure.phabricator.com/D19145
2018-02-27 14:33:12 -08:00
epriestley
dba4c4bdf6 Emit a "Content-Security-Policy" HTTP header
Summary:
See PHI399. Ref T4340. This header provides an additional layer of protection against various attacks, including XSS attacks which embed inline `<script ...>` or `onhover="..."` content into the document.

**style-src**: The "unsafe-inline" directive affects both `style="..."` and `<style>`. We use a lot of `style="..."`, some very legitimately, so we can't realistically get away from this any time soon. We only use one `<style>` (for monospaced font preferences) but can't disable `<style>` without disabling `style="..."`.

**img-src**: We use "data:" URIs to inline small images into CSS, and there's a significant performance benefit from doing this. There doesn't seem to be a way to allow "data" URIs in CSS without allowing them in the document itself.

**script-src** and **frame-src**: For a small number of flows (Recaptcha, Stripe) we embed external javascript, some of which embeds child elements (or additional resources) into the document. We now whitelist these narrowly on the respective pages.

This won't work with Quicksand, so I've blacklisted it for now.

**connect-src**: We need to include `'self'` for AJAX to work, and any websocket URIs.

**Clickjacking**: We now have three layers of protection:

  - X-Frame-Options: works in older browsers.
  - `frame-ancestors 'none'`: does the same thing.
  - Explicit framebust in JX.Stratcom after initialization: works in ancient IE.

We could probably drop the explicit framebust but it wasn't difficult to retain.

**script tags**: We previously used an inline `<script>` tag to start Javelin. I've moved this to `<data data-javelin-init ...>` tags, which seems to work properly.

**`__DEV__`**: We previously used an inline `<script>` tag to set the `__DEV__` mode flag. I tried using the "initialization" tags for this, but they fire too late. I moved it to `<html data-developer-mode="1">`, which seems OK everywhere.

**CSP Scope**: Only the CSP header on the original request appears to matter -- you can't refine the scope by emitting headers on CSS/JS. To reduce confusion, I disabled the headers on those response types. More headers could be disabled, although we're likely already deep in the land of diminishing returns.

**Initialization**: The initialization sequence has changed slightly. Previously, we waited for the <script> in bottom of the document to evaluate. Now, we go fishing for tags when domcontentready fires.

Test Plan:
  - Browsed around in Firefox, Safari and Chrome looking for console warnings. Interacted with various Javascript behaviors. Enabled Quicksand.
  - Disabled all the framebusting, launched a clickjacking attack, verified that each layer of protection is individually effective.
  - Verified that the XHProf iframe in Darkconsole and the PHPAST frame layout work properly.
  - Enabled notifications, verified no complaints about connecting to Aphlict.
  - Hit `__DEV__` mode warnings based on the new data attribute.
  - Tried to do sketchy stuff with `data:` URIs and SVGs. This works but doesn't seem to be able to do anything dangerous.
  - Went through the Stripe and Recaptcha workflows.
  - Dumped and examined the CSP headers with `curl`, etc.
  - Added a raw <script> tag to a page (as though I'd found an XSS attack), verified it was no longer executed.

Maniphest Tasks: T4340

Differential Revision: https://secure.phabricator.com/D19143
2018-02-27 10:17:30 -08:00
epriestley
f450c6c55b Fix some of the most egregious errors in Harbormaster log paging
Summary:
Depends on D19141. Ref T13088. Some of the fundamental log behaviors like "loading the correct rows" are now a bit better behaved.

The UI is a little less garbage, too.

Test Plan: Viewed some logs and loaded more context by clicking the buttons.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19142
2018-02-26 17:59:13 -08:00
epriestley
11d1dc484b Sort of make Harbormaster build logs page properly
Summary: Depends on D19139. Ref T13088. This doesn't actually work, but is close enough that a skilled attacker might be able to briefly deceive a small child.

Test Plan:
  - Viewed some very small logs under very controlled conditions, saw content.
  - Larger logs vaguely do something resembling working correctly.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19141
2018-02-26 17:58:33 -08:00
epriestley
6dc341be87 As Harbormaster logs are processed, build a sparse map of byte offsets to line numbers
Summary:
Depends on D19138. Ref T13088. When we want to read the last part of a logfile //and show accurate line numbers//, we need to be able to get from byte offsets to line numbers somehow.

Our fundamental unit must remain byte offsets, because a test can emit an arbitrarily long line, and we should accommodate it cleanly if a test emits 2GB of the letter "A".

To support going from byte offsets to line numbers, compute a map with periodic line markers throughout the offsets of the file. From here, we can figure out the line numbers for arbitrary positions in the file with only a constant amount of work.

Test Plan: Added unit tests; ran unit tests.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19139
2018-02-26 17:56:52 -08:00
epriestley
d6311044bb Store the Harbormaster log chunk format on the log record
Summary: Depends on D19137. Ref T13088. This allows `rebuild-log` to skip work if the chunks are already compressed. It also prepares for a future GC which is looking for "text" or "gzip" chunks to throw away in favor of archival into Files; such a GC can use this column to find collectable logs and then write "file" to it, meaning "chunks are gone, this data is only available in Files".

Test Plan: Ran migration, saw logs populate as "text". Ran `rebuild-log`, saw logs rebuild as "gzip".

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19138
2018-02-26 17:56:14 -08:00
epriestley
46d735d312 Add "--all" and an explicit "--force" flag to bin/harbormaster rebuild-log
Summary: Depends on D19136. Ref T13088. Since it's probably impractical to do all the migrations these changes imply during `bin/storage upgrade`, provide some support for performing them online.

Test Plan: Ran `bin/harbormaster rebuild-log` with `--all`, `--id`, and with and without `--force`.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19137
2018-02-26 17:55:38 -08:00
epriestley
57e3d607f5 In Harbormaster, record byte length on the build logs
Summary: Depends on D19135. Ref T13088. Denormalize the total log size onto the log itself. This makes reasoning about the log at display time easier, and we don't need to fish around in the database as much to figure out what we're dealing with.

Test Plan: Ran `bin/harbormaster rebuild-log`, saw an existing log populate. Ran `bin/harbormaster write-log`, saw new log write with proper length information.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19136
2018-02-26 17:54:47 -08:00
epriestley
d152bd5836 Manage log locks on the Log object to prepare for multiple writers
Summary:
Depends on D19134. Ref T13088. Future changes will support API writers, so push the log lock into the Log object.

Allow open/close ("this process is writing to this log") to be separate from live/final ("this log is still generating more data").

Test Plan: Wrote logs with `bin/harbormater write-log` and updated logs with `bin/harbormaster rebuild-log`.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19135
2018-02-26 17:54:17 -08:00
epriestley
e920e2b143 Implement DestructibleInterface on BuildLog
Summary: Depends on D19133. Ref T13088. Allows build logs to be formally destroyed, cleaning up their chunks and file data.

Test Plan:
  - Used `bin/remove destroy` to destroy a log, verified chunks and files were removed.
  - Used `bin/harbormaster rebuild-log` to force a log to rebuild, verified files were destroyed and regenerated.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19134
2018-02-26 17:53:38 -08:00
epriestley
9b4295ed60 Add a very basic standalone view for build logs with a "Download Log" button
Summary: Depends on D19132. Ref T13088. This implements an extremely skeletal dedicated log page with a more-or-less functional "Download Log" button.

Test Plan: Downloaded a recent log. Tried to download an old (un-finalized) log, couldn't. Used `bin/harbormaster write-log` to get a convenient standalone link to a log.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19133
2018-02-26 17:53:10 -08:00
epriestley
8a2604cf06 Add a "filePHID" to HarbormasterBuildLog and copy logs into Files during finalization
Summary: Depends on D19131. Ref T13088. During log finalization, stream the log into Files to support "Download Log", archive to Files, and API access.

Test Plan: Ran `write-log` and `rebuild-log`, saw Files objects generate with log content and appropriate permissions.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19132
2018-02-26 17:52:39 -08:00
epriestley
32c6b649dd Move Harbormaster log compression to the worker task queue
Summary: Depends on D19130. Ref T13088. Currently, when a build log is closed we compress it in the same process. Separate this out into a dedicated worker since the plan is to do a lot more work during finalization, none of which needs to happen inline during builds (or, particuarly, inline during a Conduit call for API writes in the future).

Test Plan: Ran `bin/harbormaster write-log --trace`, saw compression run inline.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19131
2018-02-26 17:51:58 -08:00
epriestley
cd4c4dc2ff Add bin/harbormaster write-log to write some arbitrary content into a new Harbormaster log
Summary: Ref T13088. This is currently minimal but the modify-execute development loop on build logs is extremely long without it.

Test Plan: Ran `echo hi | ./bin/harbormaster write-log --target 12345`, saw the log show up in the web UI.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13088

Differential Revision: https://secure.phabricator.com/D19130
2018-02-26 17:51:13 -08:00
epriestley
4c7370a1a3 Make the filetree view width sticky across show/hide and reload
Summary:
Ref T13090. The default width changed recently to become much wider, but the behavior on this control isn't great. Instead:

  - Pick a default width somewhere between the two.
  - Make the width sticky across show/hide (pressing "f" twice remembers your width instead of resetting it).
  - Make the width sticky across reloads (dragging the bar, then reloading the page keeps the bar in the same place).

Test Plan:
  - Without settings, loaded page: got medium-width bar.
  - Dragged bar wide/narrow, toggled on/off with "f", got persistent width.
  - Dragged bar wide/narrow, reloaded page, got persistent width.
  - Dragged bar wide/narrow, toggled it off, reloaded page, toggled it on, got persistent width.

Maniphest Tasks: T13090

Differential Revision: https://secure.phabricator.com/D19129
2018-02-22 13:47:41 -08:00
epriestley
8ae01fdc6b Fix documentation behaviors for the new proxy functions for custom datasource fields
Summary: Ref T13090. The doc string in "any()" wasn't specified correctly and the help page wasn't getting enough supporting data to build properly.

Test Plan: Viewed "Reference: Advanced Functions" for a custom datasource field and got more helpful help.

Maniphest Tasks: T13090

Differential Revision: https://secure.phabricator.com/D19128
2018-02-22 13:01:02 -08:00
epriestley
3203fd9eea Support "Any Value" and "No Value" search constraints for datasource Custom Fields
Summary: Depends on D19126. Ref T13090. For datasource custom fields, this proxies the datasource and provides "none()" and "any()" functions to allow you to search for objects with no values or any values.

Test Plan:
  - Created a custom "Owning Group" field in Maniphest using a Projects datasource.
  - For a task with no owner assigned, searched for "none()" (hit) and "any()" (miss).
  - Assigned the task to an owning project.
  - Searched for "none()" (miss), "any()" (hit), the project it is now a member of (hit) and some random other project (miss).

Maniphest Tasks: T13090

Differential Revision: https://secure.phabricator.com/D19127
2018-02-22 12:50:05 -08:00
epriestley
4cb62ca0d6 Support "phriction.document.search" queries by "parentPaths" or "ancestorPaths"
Summary: Ref T13090. Ref T13077. This adds `parentPaths` and `ancestorPaths` constraints to `phriction.document.query`. These should be a little more usable than the internal `slugPrefix` / `depth` stuff -- that's technically more powerful, but requires callers to know more slug normalization rules. We could perhaps expose `minDepth` / `maxDepth` in the future.

Test Plan: Ran valid and invalid `parentPaths` and `ancestorPaths` queries for `/`, `aaa/`, `AAA/`, etc. Got sensible-seeming results.

Maniphest Tasks: T13090, T13077

Differential Revision: https://secure.phabricator.com/D19125
2018-02-22 12:49:13 -08:00
epriestley
ffcfc04652 Add some delivery diagnostic headers to outbound mail
Summary:
Fixes T13087. Ref T13090. An install ran into a situation where mail was being double-delivered, and it wasn't immediately clear where in the pipeline the issue lay.

This change adds some headers which should rule out (or, at least, render very unlikely) some possible causes if we encounter similar issues in the future.

The `X-Phabricator-Mail-ID` header stores the ID of the `MetaMTAMail` storage object so we can distinguish between two messages sent to two different targets and one message which may have been split or re-sent. It also makes it easier to know what to `bin/mail show-outbound --id <id>` and where to find the message in the web UI for additional information.

The `X-Phabricator-Send-Attempt` is a unique value per attempt. If two mail messages are delivered with the same attempt value, the split is probably downstream from Phabricator. If they have different attempt values, the split is probably in Phabricator.

(In this case, the split was somewhere downstream from us, since sending mail with `/usr/bin/mail` also resulted in duplicates.)

Test Plan: Send some mail, inspected it with `bin/mail show-outbound --id <id>`, saw new headers with sensible/expected values.

Maniphest Tasks: T13090, T13087

Differential Revision: https://secure.phabricator.com/D19124
2018-02-22 12:47:20 -08:00
epriestley
2085716da6 Make dashboard arrange actions (move, add, remove) work again after read locking from "chaos reduction"
Summary:
See PHI385. Ref T13054. Ref T13083. The dashboard "arrange" operations (add, remove, move) rely on doing `$dashboard->setThing(...)` and then applying transactions.

This no longer works after the read locking change from T13054. To make this function again, just add an explicit `save()` after layout adjustment. This should be more nuanced eventually, but all arrange operations are nonfunctional in a corrupting way at HEAD of `master`/`stable`, so stop the bleeding first.

Test Plan:
  - Created new empty and template dashboards.
  - Moved panels.
  - Added new and existing panels.
  - Removed panels.

Maniphest Tasks: T13083, T13054

Differential Revision: https://secure.phabricator.com/D19123
2018-02-20 17:23:24 -08:00
epriestley
2fb266de7c Fix some of the most obvious bugs in fact generation from Maniphest tasks
Summary:
Depends on D19121. Ref T13083. Group transactions and show groups in the debugging view.

Fix some of the most obvious issues with fact generation:

  - No more 0-point facts.
  - Engine can now generate at least one of every type of fact.

Test Plan: Generated facts, viewed them in the debugging view, fact generation largely appeared to align with reality. No more "no facts in storage" facts.

Subscribers: yelirekim

Maniphest Tasks: T13083

Differential Revision: https://secure.phabricator.com/D19122
2018-02-19 12:07:28 -08:00
epriestley
46ce4c7aef Provide a page for examining the facts an object generates
Summary:
Depends on D19120. Ref T13083. When you write a fact engine, it's currently somewhat difficult to figure out exactly what it's doing. It would also be difficult to diagnose bugs or report them to the upstream.

To ease this, add a page which shows all the facts an object generates. This allows you to iterate on an engine quickly without needing to reanalyze facts, take a screenshot, easily compare the timeline to the fact view, etc.

Test Plan: Viewed the object fact page for several objects.

Subscribers: yelirekim

Maniphest Tasks: T13083

Differential Revision: https://secure.phabricator.com/D19121
2018-02-19 12:06:36 -08:00
epriestley
e3a1a32444 Extract count/point data from tasks in Fact engines
Summary:
Depends on D19119. Ref T13083. This is probably still very buggy, but I'm planning to build support tools to make debugging facts easier shortly.

This generates a large number of datapoints, at least, and can render some charts which aren't all completely broken in an obvious way.

Test Plan: Ran `bin/fact analyze --all`, got some charts with lines that went up and down in the web UI.

Subscribers: yelirekim

Maniphest Tasks: T13083

Differential Revision: https://secure.phabricator.com/D19120
2018-02-19 12:06:03 -08:00
epriestley
0dee34b3fa Make Facts more modern, DRY, and dimensional
Summary:
Ref T13083. Facts has a fair amount of weird hardcoding and duplication of responsibilities. Reduce this somewhat: no more hard-coded fact aggregates, no more database-driven list of available facts, etc. Generally, derive all objective truth from FactEngines. This is more similar to how most other modern applications work.

For clarity, hopefully: rename "FactSpec" to "Fact". Rename "RawFact" to "Datapoint".

Split the fairly optimistic "RawFact" table into an "IntDatapoint" table with less stuff in it, then dimension tables for the object PHIDs and key names. This is primarily aimed at reducing the row size of each datapoint. At the time I originally wrote this code we hadn't experimented much with storing similar data in multiple tables, but this is now more common and has worked well elsewhere (CustomFields, Edges, Ferret) so I don't anticipate this causing issues. If we need more complex or multidimension/multivalue tables later we can accommodate them. The queries a single table supports (like "all facts of all kinds in some time window") don't make any sense as far as I can tell and could likely be UNION ALL'd anyway.

Remove all the aggregation stuff for now, it's not really clear to me what this should look like.

Test Plan: Ran `bin/fact analyze` and viewed web UI. Nothing exploded too violently.

Subscribers: yelirekim

Maniphest Tasks: T13083

Differential Revision: https://secure.phabricator.com/D19119
2018-02-19 12:05:19 -08:00
epriestley
05a4c55c52 Explicitly add rel="noreferrer" to all external links
Summary: See D19117. Instead of automatically figuring this out inside `phutil_tag()`, explicitly add rel="noreferrer" at the application level to all external links.

Test Plan:
  - Grepped for `_blank`, `isValidRemoteURIForLink`, checked all callsites for user-controlled data.
  - Created a link menu item, verified noreferrer in markup.
  - Created a link custom field, verified no referrer in markup.
  - Verified noreferrer for `{nav href=...}`.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19118
2018-02-17 17:46:11 -08:00
epriestley
cb2f710606 Provide the document content as a context object when rendering Phriction documents
Summary: Ref T13077. The context object wasn't being passed into the engine properly here, affecting relative link rendering in Phriction.

Test Plan: Viewed rendered Phriction documents with relative links, got clean renders.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19115
2018-02-16 12:42:41 -08:00
epriestley
66a7ca49b2 Fix incorrect context extraction for relative Phriction links on Phriction pages
Summary: Ref T13077. This content extraction rule wasn't right and caused rendering on Phriction pages to extract context improperly.

Test Plan: Viewed pages in Phriction with relative links to other documents.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19114
2018-02-16 12:33:40 -08:00
epriestley
db3ef4021a Freeze the "phriction.info" Conduit API method
Summary: Ref T13077. Freeze "phriction.info" in favor of the more modern "phriction.document.search".

Test Plan: Reviewed older method in web UI, saw frozen markers.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19113
2018-02-16 12:14:23 -08:00
epriestley
6d3177a3bf Allow "phriction.document.search" to query by path
Summary: Ref T13077. Adds a "paths" constraint to the API query.

Test Plan: Used paths constraint to fetch documents.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19112
2018-02-16 11:40:43 -08:00
epriestley
0202c36b62 Suggest Phurl URLs on "((..." in Remarkup text areas
Summary: Depends on D19108. Ref T12241. Ref T13077. See D19108. This extends the `[[ ...` autocompleter to `((...` for Phurl URLs.

Test Plan: Typed `((th`, got `((thing))` suggested.

Reviewers: avivey

Reviewed By: avivey

Maniphest Tasks: T13077, T12241

Differential Revision: https://secure.phabricator.com/D19109
2018-02-16 09:56:39 -08:00
epriestley
8771b7d5c4 Add autocomplete for Phriction documents on "[[ ..." in Remarkup
Summary: Depends on D19107. Ref T13077. The underlying datasource may need some adjustment but this appears to work properly locally.

Test Plan: Typed `[[ por` locally, was suggested "Porcupine Facts". Typed `[[ / ]]`, saw it render as a reference to the wiki root instead of the install root.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19108
2018-02-16 09:56:18 -08:00
epriestley
f82206a4d1 Add a rough Quick Search datasource for Phriction documents
Summary:
Depends on D19106. Fixes T5941. Ref T13077. Allows you to find Phriction documents as suggestions from global quick search.

Also supports `w` to jump to Phriction and `w query` to query Phriction.

The actual query logic for the datasource may need some tweaking after it collides with reality, but seems to produce fairly reasonable results in local testing against synthetic data.

Test Plan: Searched for "Porcupine Facts", "Travel Companions", and other useful local pages. Searched for `w`. Searched for `w zebra facts`.

Maniphest Tasks: T13077, T5941

Differential Revision: https://secure.phabricator.com/D19107
2018-02-16 09:55:54 -08:00
epriestley
b8bb4d3ad5 Accept either "[[ %24doge ]]" or "[[ $doge ]]" as references to the "/w/$doge/" Phriction document
Summary:
Depends on D19105. Ref T13077. Fixes T12344.

The `[[ ... ]]` syntax accepts and handles characters which would require URL encoding if they appeared in URIs. For example, `[[ 100% Natural Cheese Dust ]]` is a legitimate, supported piece of remarkup syntax, and does not need to be written as `... 100%25 Natural ...`.

Likewise, `[[ BUY $DOGE ]]` is legitimate and does not need to be written as `[[ BUY %24DOGE ]]`. This piece of syntax creates a link to `/w/buy_$doge/`. This may or may not appear in your browser's URL bar as `/w/buy_%24doge/`, but internally "$" is a valid slug character and you'll see `buy_$doge` over Conduit, etc.

However, since users may reasonably copy paths from their browser URL bar, they may have unnecessary URL encoding. The syntax `[[ buy_$doge ]]` is legitimate, but a user copy/pasting may write `[[ buy_%24doge ]]` instead.

Currently, this extra URL encoding causes links to break, since `[[ buy_%24doge ]]` is a treated as link to `/w/buy_24doge/`, just like `[[ Fresh 100%AB Blood ]]` is a link to `/w/fresh_100_ab_blood/`.

To fix this:

  - When the target for a link can be URL decoded, try to do lookups on both the un-decoded and decoded variations.
  - If the un-decoded variation works, great: use it. This preserves behavior for all existing, working links.
  - If the un-decoded variation fails but the decoded variation works, okay: we'll assume you copy-pasted a URL-encoded version and strip URL encoding.
  - If both fail, same behavior as before.

Also, use a different spelling for "existent".

See T13084 for some "attacks" based on this behavior. I think the usability affordance this behavior provides greatly outweighs the very mild threat those attacks represent.

Test Plan:
  - Created links to existing, nonexisting, and existing-but-not-visible documents, all of which worked normally.
  - Created links to `[[ $doge ]]` and `[[ %24doge ]]`, saw them both go to the right place.
  - Performed the "attacks" in T13084.

Maniphest Tasks: T13077, T12344

Differential Revision: https://secure.phabricator.com/D19106
2018-02-16 09:55:32 -08:00
epriestley
bfdc9411f7 Provide context objects for remarkup mail rendering, fixing Phriction relative URIs in initial email
Summary:
Fixes T10969. Ref T13077. When you create a Phriction document with a relative link (`[[ ./path/to/page ]]`) the initial email currently points to the wrong place.

This is because the context object (the page) isn't passed to the markup engine. Without this context, the relative link is rendered as though it appeared somewhere else (like a task or revision) where relative links don't make sense.

Test Plan: Created a new Phriction document with a relative link to `[[ ./porcupine_facts/starmap ]]`, saw a usable link in the resulting email.

Maniphest Tasks: T13077, T10969

Differential Revision: https://secure.phabricator.com/D19105
2018-02-16 09:55:04 -08:00
epriestley
f713e1dfc1 Add Owners Package support for "Commit Hook: Content" Herald rules
Summary:
See PHI370. Support the "Affected packages" and "Affected package owners" Herald fields in pre-commit hooks.

I believe there's no technical reason these fields aren't supported and this was just overlooked.

Test Plan: Wrote a rule which makes use of the new fields, pushed commits through it. Checked transcripts and saw sensible-looking values.

Differential Revision: https://secure.phabricator.com/D19104
2018-02-16 09:49:24 -08:00
epriestley
45403b162a Flesh out "phriction.document.search" slightly and provide page text for content/documents
Summary: Depends on D19100. Ref T13077. Adds a "content" attachment to get the actual page text. This works on both "phriction.document.search" and "phriction.content.search".

Test Plan: Called both API methods with the attachment, saw proper text content returned.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19103
2018-02-15 18:24:37 -08:00
epriestley
143350fdba Give Phriction documents modern string status constants instead of numeric constants
Summary:
Depends on D19099. Ref T13077. Updates Phriction documents to string constants to make API interactions cleaner and statuses more practical to extend.

This does not seem to require any transaction migrations because none of the Phriction transactions actually store status values: status is always a side effect of other edits.

Test Plan: Created, edited, deleted, moved documents. Saw appropriate UI cues. Browsed and filtered documents by status in the index.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19100
2018-02-15 18:23:41 -08:00
epriestley
c1056f6dab Partially clean up Phriction document status constants; introduce "phriction.document.search"
Summary:
Depends on D19098. Ref T13077.

Phriction status constants currently use the "bag of statuses" approach typical of older code, and store integers in the database.

This fixes the "bag of statuses" stuff; a future change will fix the integers.

Also adds a skeleton for `phriction.document.search`, but doesn't implement the Conduit interface yet.

Test Plan: Searched for documents with various status constraints. Grepped for removed status constants. Viewed document list.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19099
2018-02-15 18:19:10 -08:00
epriestley
48eb82f229 Freeze "phriction.history" in favor of "phriction.content.search"
Summary: Depends on D19097. Ref T13077. Freeze the older method now that the newer one is available.

Test Plan: Viewed the older method's page and saw it frozen; called it to make sure I didn't break it by accident.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19098
2018-02-15 18:15:18 -08:00
epriestley
8101bf74e9 Introduce a "phriction.content.search" API method to replace "phriction.history"
Summary: Depends on D19096. Ref T13077. Adds a new "v3" API method for Phriction document content, to replace the existing "phriction.history" call.

Test Plan: Made various calls via web API console.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19097
2018-02-15 18:12:54 -08:00
epriestley
a965d8d6ae Make PhrictionContent "description" non-nullable
Summary:
Depends on D19095. Ref T6203. Ref T13077. This column is nullable in an inconsistent way. Make it non-nullable.

Also clean up one more content query on the history view.

Test Plan: Ran migration, then created and edited documents without providing a descriptino or hitting `NULL` exceptions.

Maniphest Tasks: T13077, T6203

Differential Revision: https://secure.phabricator.com/D19096
2018-02-15 17:55:11 -08:00
epriestley
f742d00c28 Mostly use PhrictionContentQuery to load PhrictionContent objects
Summary: Depends on D19094. Ref T13077. Use modern infrastructure to perform these loads. I left a couple of calls in the older API methods unconverted.

Test Plan: Viewed documents. Viewed older versions. Viewed diffs. Did revert edits to older versions.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19095
2018-02-15 17:44:43 -08:00
epriestley
9404e2b3d4 Implement PolicyInterface, ExtendedPolicyInterface, and DestructibleInterface on PhrictionContent
Summary:
Depends on D19093. Ref T13077. Although content objects normally don't have any edges today, they may in the future.

Also implement Policy stuff properly.

Test Plan: Used `bin/remove destroy` to destroy a document, verified it also loaded and destroyed the correspoding Content correctly by looking at `--trace` and the database rows.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19094
2018-02-15 17:44:11 -08:00
epriestley
b2c829f274 Move PhrictionContent from RemarkupInterface (deprecated) to PHUIRemarkupView
Summary:
Depends on D19092. Ref T13077. This modernizes markup rendering for PhrictionContent.

This is a little messy because table of contents generation isn't straightforward.

Test Plan: Viewed Phriction documents with and without 3+ headers, saw ToC vs no ToC. Edited/previewed documents. Grepped for affected symbols. Checked DarkConsole for sensible cache behavior.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19093
2018-02-15 17:40:51 -08:00
epriestley
e492c717c6 Give PhrictionContent objects (older versions of wiki pages) legitimate PHIDs
Summary: Ref T13077. Prepares for modern API access to document history using standard "v3" APIs.

Test Plan: Ran migration, verified PHIDs appeared in the database. Created/edited a document, got even more PHIDs in the database.

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19092
2018-02-15 17:39:07 -08:00
epriestley
463dda98ed Render Drydock logs in text from "bin/drydock lease"; in HTML in web views
Summary: Ref T13073. The new log output from `bin/drydock lease` currently uses HTML handle rendering, but should render to text.

Test Plan: Ran `bin/drydock lease` and saw normal text in log output. Viewed the same logs from the web UI and saw HTML.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19101
2018-02-15 17:25:34 -08:00
epriestley
fe294d4034 Allow third-party code to extend upstream datasources via EngineExtension
Summary: Depends on D19089. Fixes T13079. This is likely not the final form of this, but creates a defensible extension point.

Test Plan: See T13079 for discussion.

Maniphest Tasks: T13079

Differential Revision: https://secure.phabricator.com/D19090
2018-02-14 18:11:51 -08:00
epriestley
d6edc3f4cc Support evaluation of complex tokenizer functions
Summary:
Depends on D19088. Ref T13079.

> Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.
> - Greenspun's Tenth Rule

Move us a step closer to this noble goal.

This doesn't implement any `viewer(project())` stuff but it looks like the API doesn't need to change to do that in the future.

Test Plan: Grimmaced in pain.

Maniphest Tasks: T13079

Differential Revision: https://secure.phabricator.com/D19089
2018-02-14 18:11:15 -08:00
epriestley
4bccb1547d Modularize the "jump nav" behaviors in global search
Summary: Depends on D19087. Ref T13079. This still doesn't feel like the most clean, general system in the world, but is a step forward from hard-coded `switch()` stuff.

Test Plan:
- Jumped to `r`.
- Jumped to `a`.
- Jumped to `r poe` (multiple results).
- Jumped to `r poetry` (one result).
- Jumped to `r syzygy` (no results).
- Jumped to `p`.
- Jumped to `p robot` (multiple results); `p assessment` (one result).
  - The behavior for `p <string>` has changed slightly but should be more powerful now (it's consistent with `r <string>`).
- Jumped to `s <symbol>` and `s <context>-><symbol>`.
- Jumped to `d`.
- Jumped to `f`.
- Jumped to `t`.
- Jumped to `T123`, `D123`, `@dog`, `PHID-DREV-abcd`, etc.

Maniphest Tasks: T13079

Differential Revision: https://secure.phabricator.com/D19088
2018-02-14 18:08:07 -08:00
epriestley
abe5fd57b0 Rename "QuickSearch" Engine/EngineExtension to "Datasource"
Summary: Ref T13079. This recently-introduced Engine/EngineExtension are a good fit for adding more datasource functions in general, but we didn't think quite big enough in naming them.

Test Plan: Used quick search typeahead, hit applications/users/monograms/symbols/etc.

Maniphest Tasks: T13079

Differential Revision: https://secure.phabricator.com/D19087
2018-02-14 18:03:03 -08:00
epriestley
f74e6bbf8d Make "phabricator.silent" disable build steps which rely on external services
Summary:
Depends on D19084. Fixes T13078. When `phabricator.silent` is enabled, immediately fail the "HTTP Request", "CircleCI" and "Buildkite" build steps.

This doesn't feel quite as clean as most of the other behavior of `phabricator.silent`, since these calls are not exactly notifications in the same way that email is, and failing to make these calls means that builds run differently (whereas failing to deliver email doesn't really do anything).

However, I suspect that this behavior is almost always reasonable/correct, and that we can probably get away with it until this grey area between "notifications" and "external service calls" is more clearly defined.

Test Plan:
  - Created a build with HTTP, CircleCI, and Buildkite steps.
  - Put install in `phabricator.silent` mode: all three steps failed with "declining, because silent" messages.
  - Put install back in normal mode: all three steps made HTTP requests.
  - Read updated documentation.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13078

Differential Revision: https://secure.phabricator.com/D19085
2018-02-14 11:41:47 -08:00
epriestley
a2453706ab When "phabricator.silent" is enabled, don't call webhooks
Summary: Ref T13078. The `phabricator.silent` configuration flag should disable webhook calls, since this is consistent with the documented and desired behavior.

Test Plan: Enabled `phabricator.silent`, made test hook calls, saw them fail with a "silent" failure reason.

Maniphest Tasks: T13078

Differential Revision: https://secure.phabricator.com/D19084
2018-02-14 11:40:19 -08:00
epriestley
6bfd0ff275 Support "encoding", which is required by PHPMailerLite, in SES adapter
Summary: See <https://discourse.phabricator-community.org/t/amazon-ses-adapator-broken-after-upgrade/1121>. The adapter class tree is a mess and this property is read by the parent class.

Test Plan: Configured an SES mailer, used `bin/mail send-test` to reproduce the issue before the patch and observe it working after the patch.

Differential Revision: https://secure.phabricator.com/D19083
2018-02-14 10:57:07 -08:00
epriestley
743f0d65ea Fix a mail stamp issue with blocking reviewers
Summary: Revisions with blocking reviewers had this stamp built incorrectly, which cascaded into trying to use `array()` as a PHID. Recover so these tasks succeed.

Test Plan: Will deploy production.

Differential Revision: https://secure.phabricator.com/D19082
2018-02-13 17:56:21 -08:00
epriestley
a1baedbd9a Lock resources briefly while acquiring leases on them to prevent acquiring near-death resources
Summary:
Depends on D19078. Ref T13073. Currently, there is a narrow window where we can acquire a resource after a reclaim has started against it.

To prevent this, briefly lock resources before acquiring them and make sure they're still good. If a resource isn't good, throw the lease back in the pool.

Test Plan:
This is tricky. You need:

  - Hoax blueprint with limits and a rule where leases of a given "flavor" can only be satisfied by resources of the same flavor.
  - Reduce the 3-minute "wait before resources can be released" to 3 seconds.
  - Limit Hoaxes to 1.
  - Allocate one "cherry" flavored Hoax and release the lease.
  - Add a `sleep(15)` to `releaseResource()` in `DrydockResourceUpdateWorker`, after the `canReclaimResource()` check, with a `print`.

Now:

  - Run `bin/phd debug task` in two windows.
  - Run `bin/drydock lease --type host --attributes flavor=banana` in a third window.
  - This will start to reclaim the existing "cherry" resource. Once one of the `phd` windows prints the "RECLAIMING" message run `bin/drydock lease --type host --attributes flavor=cherry` in a fourth window.
  - Before patch: the "cherry" lease acquired immediately, then was released and destroyed moments later.
  - After patch: the "cherry" lease yields.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19080
2018-02-13 13:22:13 -08:00
epriestley
619943bea0 Reduce collision rate for concurrency-limiting slot locks
Summary:
Depends on D19077. Ref T13073. When we're using slot locks to enforce a limit (e.g., maximum of 5 simultaneous things) we currently load locks owned by the blueprint to identify which slots are likely to be free.

However, this isn't right: the blueprint doesn't own these locks. The resources do.

We still get the right behavior eventually, but we incorrectly identify that every slot lock is always free, so as the slots fill up we'll tend to guess wrong more and more often.

Instead, load the slot locks by name explicitly.

Test Plan: Implemented lock-based limiting on `HoaxBlueprint`, `var_dump()`'d the candidate locks, saw correct test state for locks. Acquired leases without releasing, got all of the slots filled without any slot lock collisions (previously, the last slot or two tended to collide a lot).

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19078
2018-02-13 13:21:32 -08:00
epriestley
2994753d23 Don't let bin/drydock lease --attributes overwrite blueprints
Summary:
Depends on D19076. Ref T13073. Blueprints are stored as an attribute and `setAttributes()` overwrites all attributes.

This is sorta junk but make it less obviously broken, at least.

Test Plan: Ran `bin/drydock lease --type working-copy --attributes x=y` without instantly getting a fatal about "no blueprint PHIDs".

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19077
2018-02-13 13:20:36 -08:00
epriestley
30a0b103e6 When a lease acquires a resource but the resource fails to activate, throw the lease back in the pool
Summary:
Depends on D19075. Ref T13073. If a lease acquires a resource but finds that the resource builds directly into a dead state (which can happen for any number of reasonable reasons), reset the lease and throw it back in the pool.

This isn't the lease's fault and it hasn't caused any side effects or done anything we can't undo, so we can safely reset it and throw it back in the pool.

Test Plan:
  - Created a blueprint which throws from `allocateResource()` so that resources never activate.
  - Tried to lease it with `bin/drydock lease ...`.
  - Before patch: lease was broken and destroyed after a failed activation.
  - After patch: lease was returned to the pool after a failed activation.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19076
2018-02-13 13:17:54 -08:00
epriestley
06bbf237fe Give Drydock Resources more modern status treatment
Summary: Ref T13073. Depends on D19074. Update icons and UI for resource status.

Test Plan: Viewed resources in detail view and list view, saw better status icons.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19075
2018-02-13 13:16:46 -08:00
epriestley
27c3793d40 Give Drydock Leases more modern status treatment
Summary:
Depends on D19073. Ref T13073. Give leases a normal header tag and try to wrangle their status constants a bit.

Also, try to capture the "status class" pattern a bit. Since we target PHP 5.2.3 we can't use `static::` so the actual subclass is kind of a mess. Not exactly sure if I want to stick with this or not. We could consider targeting PHP 5.3.0 instead to get `static::` / late static binding.

Test Plan: Viewed leases and lease lists, saw better and more conventional status information.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19074
2018-02-13 13:15:57 -08:00
epriestley
07028cfc30 When bin/drydock lease is interrupted, release leases
Summary:
Depends on D19072. Ref T13073. Currently, you can leave leases stranded by using `^C` to interrupt the script. Handle signals and release leases on destruction if they haven't activated yet.

Also, print out more useful information before and after activation.

Test Plan: Mashed ^C while runnning `bin/drydock lease ... --trace`, saw the lease release.

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19073
2018-02-13 13:14:21 -08:00
epriestley
b833e324bd While waiting for a "bin/drydock" lease to activate, entertain the user with log output
Summary: Depends on D19071. Ref T13073. While the daemons are supposedly doing things, show the user any logs they generate. There's often something relevant but unearthing it can be involved.

Test Plan: {F5427773}

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19072
2018-02-13 13:13:52 -08:00
epriestley
3ec80a36db In Drydock log views, respect newlines
Summary: Depends on D19070. Ref T13073. Some messages contain an interesting story or a clever anecdote. Respect newlines during rendering to preserve authorial intent.

Test Plan:
Viewed a message with linebreaks and could still read it.

{F5427754}

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19071
2018-02-13 13:12:31 -08:00
epriestley
4dd32dca3e When a Drydock Blueprint promises us a resource but can't deliver, continue believing in it
Summary:
Ref T13073. When a Blueprint says it will be able to allocate a resource but then throws an exception while attempting that allocation, we currently fail the lease permanently.

This is excessively harsh. This blueprint may have the best of intentions and have encountered a legitimately unforseeable failure (like a `vm.new` call to build a VM failed) and be able to succeed in the future.

Even if this blueprint is a dirty liar, other blueprints (or existing resources) may be able to satisfy the lease in the future.

Even if every blueprint is implemented incorrectly, leaving the lease alive lets it converge to success after the blueprints are fixed.

Instead of failing, log the issue and yield.

(In the future, it might make sense to distinguish more narrowly between "actually, all the resources are used up" and all other failure types, since the former is likely more routine and less concerning.)

Test Plan:
  - Wrote a broken `Hoax` blueprint which always claims it can allocate but never actually allocates (just `throw` in `allocateResource()`).
  - Used `bin/phd drydock lease` to acquire a Hoax lease.
  - Before patch: lease abruptly failed permanently.
  - After patch: lease yields after allocation fails.

{F5427747}

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19070
2018-02-13 13:11:55 -08:00
epriestley
6a4d5ce3c9 Don't allow Herald Diff rules to "Call Webhooks"
Summary: Like "Commit Hook" rules, these also fire oddly and don't have an object PHID or a list of transactions.

Test Plan: Verified that "Call Webhooks" was no longer available from Diff rules, but still available from other rule types.

Differential Revision: https://secure.phabricator.com/D19069
2018-02-12 17:52:05 -08:00
epriestley
894e9dd852 Update a handful of missed HarbormasterBuildableStatus constants
Summary: See <https://discourse.phabricator-community.org/t/exception-undefined-class-status-building/1103>.

Test Plan: Used `grep` more carefully.

Differential Revision: https://secure.phabricator.com/D19068
2018-02-12 15:33:24 -08:00
epriestley
11c9994134 Allow "arc diff --plan-changes" to work with drafts enabled
Summary:
See PHI346. Ref T13054. If you have prototypes enabled on the server but use `master` / `stable` on the client and run `arc diff --plan-changes`, the transition is rejected because "Draft -> Changes Planned" isn't currently a legal transition.

Allow this transition if not coming from the web UI (to keep it out of the dropdown).

Test Plan:
  - Ran `arc diff --plan-changes` on `master`, got a "Changes Planned" revision instead of a validation error.
  - Ran `arc diff` without `--plan-changes`, got a draft, verified that "Plan Changes" still doesn't appear in the action dropdown.

Maniphest Tasks: T13054

Differential Revision: https://secure.phabricator.com/D19067
2018-02-12 13:15:13 -08:00
epriestley
6f508a2258 Update buildable containerPHIDs in a proper way via BuildWorker rather than via sneaky uncoordinated write
Summary:
Depends on D19065. Ref T13054. Instead of just updating `containerPHID` and hoping for the best, queue a proper BuildWorker to process a "your container has changed, update it" message.

We also need to remove a (superfluous) `withContainerPHIDs()` when loading active diffs for a revision.

Test Plan:
  - Without daemons, created a revision and saw builds stick in "preparing" with no container PHID, but also stay in draft mode.
  - With daemons, saw builds actually build and get the right container PHID.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13054

Differential Revision: https://secure.phabricator.com/D19066
2018-02-12 12:18:52 -08:00
epriestley
66f20595e4 Start buildables in "PREPARING", move them to "BUILDING" after builds queue
Summary:
Depends on D19064. Ref T13054. See that task for additional discussion.

When buildables are created by `arc` and have lint/unit messages, they can currently pass or fail before Herald triggers actual builds. This puts them in a pre-build state where they can't complete until Herald says it's okay.

On its own, this change intentionally strands `arc diff --only` diffs in the "PREPARING" stage forever.

Test Plan:
  - Ran a build with `bin/harbormaster`, saw it build normally.
  - Ran a build with web UI, saw it build normally.
  - Ran a build with `arc diff`, saw it build normally.
  - Ran a build with `arc diff --only`, saw it hang in "PREPARING" forever.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13054

Differential Revision: https://secure.phabricator.com/D19065
2018-02-12 12:18:29 -08:00
epriestley
f939a2b12e Make Harbormaster buildable status more of a nice flexible map and less of a bunch of switch statements
Summary: Depends on D19063. Ref T13054. Prepare for the addition of a new `PREPARING` status by getting rid of the "scattered mess of switch statements" pattern of status management.

Test Plan: Searched/browsed buildables. Viewed buildables. Viewed revisions. Grepped for all affected symbols.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13054

Differential Revision: https://secure.phabricator.com/D19064
2018-02-12 12:18:06 -08:00
epriestley
c42bbd6f5c Rename HarbormasterBuildMessage "buildTargetPHID" to "receiverPHID"
Summary: Ref T13054. Companion storage change for D19062.

Test Plan: Applied migration and adjustments. Viewed messages in Harbormaster; created them with `harbormaster.sendmessage`; processed them with `bin/phd debug task`.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13054

Differential Revision: https://secure.phabricator.com/D19063
2018-02-12 12:17:44 -08:00
epriestley
ed0ba41cd2 Allow a HarbormasterBuildMessage to be sent to any object
Summary:
See T13054. This prepares for Buildables to be sent messages ("attach", "done scheduling builds") to fix races between Harbormaster and Differential.

The `buildTargetPHID` is replaced with a `recipientPHID` in the API. An additional change will fix the storage.

In the future, this table could probably also replace `HarbormasterBuildCommand` now, which is approximately the same bus, but for Builds.

Test Plan: Viewed builds with messages. Sent messages with `harbormaster.sendmessage`. Processed messages with `bin/phd debug task`.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19062
2018-02-12 12:16:03 -08:00
epriestley
4fa99374be Prevent "Call webhooks" Herald action from appearing in UI for adapters which can't fire it
Summary:
See <https://discourse.phabricator-community.org/t/herald-webhook-is-not-called-but-herald-transcript-tells-me-the-opposite/1098>.

The "Commit Hook" events don't operate on objects and don't use TransactionEditors. They can't call webhooks in a normal way and currently don't call webhooks at all. Stop offering these actions in the UI.

The  "Outbound Mail" event also fires oddly and likely doesn't make much sense to hook anyway.

Test Plan: Verified that these events no longer offer "Call webhooks", while normal events still do.

Differential Revision: https://secure.phabricator.com/D19061
2018-02-11 06:15:29 -08:00
epriestley
5e6e9fcc56 When purging drafts after a transaction edit, purge all drafts
Summary: Fixes T13071. See that task for discusison. I think this `<= version` constraint is needless in normal cases (it should match everything in the table anyway), and slightly harmful in bizarre cases where a draft somehow gets a much larger ID than it should have.

Test Plan:
  - Gave a draft an unreasonably large ID.
  - Pre-patch, observed: submitting comments on the draft's object does not clear the draft.
  - Post-patch: submitting comments on the draft's object now clears the draft correctly.
  - Also added comments/actions, reloaded pages, saw drafts stick properly.

Maniphest Tasks: T13071

Differential Revision: https://secure.phabricator.com/D19060
2018-02-11 06:01:09 -08:00
epriestley
653bc0fa01 Read lock all transaction edits
Summary: Ref T13054. Fixes T12714. Applies read locks to all transactions instead of only a very select subset (chat messages in Conpherence).

Test Plan: See <T13054#235650> for discussion and testing.

Maniphest Tasks: T13054, T12714

Differential Revision: https://secure.phabricator.com/D19059
2018-02-10 20:07:46 -08:00
epriestley
f43d08c2bb Completely remove the legacy hunk table
Summary: Depends on D19056. Fixes T8475. Ref T13054. Merges "ModernHunk" back into "Hunk".

Test Plan: Grepped for `modernhunk`. Reviewed revisions. Created a new revision. Used `bin/differential migrate-hunk` to migrate hunks between storage formats and back.

Maniphest Tasks: T13054, T8475

Differential Revision: https://secure.phabricator.com/D19057
2018-02-10 16:12:50 -08:00
epriestley
b0d1d46a73 Drop the legacy hunk table
Summary: Ref T13054. Ref T8475. This table has had no readers or writers for more than a year after it was migrated to the modern table.

Test Plan: Ran migration, verified that all the data was still around.

Maniphest Tasks: T13054, T8475

Differential Revision: https://secure.phabricator.com/D19056
2018-02-10 16:09:31 -08:00
epriestley
a2d02aed22 When a build is aborted, fail the buildable
Summary:
Ref T13054. Fixes T10746. Fixes T11154. This is really a one-line fix (include `ABORTED` in `BuildEngine->updateBuildable()`) but try to structure the code a little more clearly too and reduce (at least slightly) the number of random lists of status attributes spread throughout the codebase.

Also add a header tag for buildable status.

Test Plan: Aborted a build, saw buildable fail properly.

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13054, T11154, T10746

Differential Revision: https://secure.phabricator.com/D19055
2018-02-10 16:08:41 -08:00
epriestley
d47c5de9d0 Fix PHPMailer/SMTP configuration typo for legacy SMTP configurations
Summary: See <https://discourse.phabricator-community.org/t/phpmailer-cannot-read-configuration-value-because-of-typo/1094>.

Test Plan: Browsed a dictionary. (This doesn't get hit when configured via `cluster.mailers`, which is how I previously re-tested SMTP configuration.)

Differential Revision: https://secure.phabricator.com/D19058
2018-02-10 16:06:11 -08:00
epriestley
c64aae052f Make sure auditors are attached to commits on new pathways
Companion change to D19022 for commits. Mentioning and subscribing to commits
can load them without audit data.
2018-02-09 17:09:00 -08:00
epriestley
9b7d5b74d4 Purge ssh-auth key cache after trust/untrust
Summary: See PHI358. The `bin/almanac [un]trust-key` workflows don't properly purge the SSH key cache, but should.

Test Plan:
  - Added key `ssh-rsa xyz` to a device.
  - Used `bin/ssh-auth | grep xyz` to test for the presence of the key.
  - Before patch: Saw it not present, trusted it, saw it still not present.
  - After patch: Saw it not present, trusted it, saw it now present. Untrusted it, saw it no longer present.

Differential Revision: https://secure.phabricator.com/D19053
2018-02-09 14:58:45 -08:00
epriestley
4fef0a6128 Allow a wider range of characters in macro names, including emoji
Summary:
Fixes T6121. See PHI357.

  - Allow emoji and other unicode (like Chinese characters) as long as you have at least three of them.
  - Disallow macros with only latin symbols. These were previously allowed.

Test Plan: Created a macro for "🐶🐶🐶", then used it in a comment.

Maniphest Tasks: T6121

Differential Revision: https://secure.phabricator.com/D19051
2018-02-09 14:34:30 -08:00
epriestley
64177cb16e Document how webhooks work
Summary: Depends on D19049. Ref T11330. Adds some documentation for webhooks.

Test Plan: Read the documentation and found it to be exceptionally accurate and helpful.

Maniphest Tasks: T11330

Differential Revision: https://secure.phabricator.com/D19050
2018-02-09 13:57:19 -08:00
epriestley
98c701ffc5 Add a "Call webhooks" action to Herald
Summary: Depends on D19048. Fixes T11330.

Test Plan: Wrote rules to call webhooks selectively, saw them fire appropriately with correct trigger attribution.

Maniphest Tasks: T11330

Differential Revision: https://secure.phabricator.com/D19049
2018-02-09 13:56:57 -08:00
epriestley
41d28abfcc Trigger all "Firehose" webhooks on all transactional edits
Summary: Depends on D19047. Ref T11330. Triggers every firehose hook on every edit; prepares for Herald triggers.

Test Plan: Configured a firehose hook, edited some objects, saw callbacks.

Maniphest Tasks: T11330

Differential Revision: https://secure.phabricator.com/D19048
2018-02-09 13:56:34 -08:00
epriestley
4887c6aa80 Allow "transaction.search" to be constrained by PHIDs
Summary: Depends on D19046. Ref T11330. Supports querying for specific transactions while responding to webhooks.

Test Plan: Called `transaction.search` with and without PHID constraints.

Maniphest Tasks: T11330

Differential Revision: https://secure.phabricator.com/D19047
2018-02-09 13:56:16 -08:00
epriestley
dc2995c4ca Refine core webhook implementation somewhat
Summary:
Depends on D19045. Ref T11330.

  - View/regenerate HMAC keys.
  - Pretty JSON.
  - Readable status transactions.
  - test, silent, secure flags.
  - Dates on request view.
  - More icons.
  - Can test any object.
  - GC for requests.

Test Plan: Went through each feature poking at it in the web UI and with `bin/webhook call ...` / `bin/garbage collect ...`.

Subscribers: ftdysa

Maniphest Tasks: T11330

Differential Revision: https://secure.phabricator.com/D19046
2018-02-09 13:55:55 -08:00
epriestley
0470125d9e Add skeleton code for webhooks
Summary: Ref T11330. Adds general support for webhooks. This is still rough and missing a lot of pieces -- and not yet useful for anything -- but can make HTTP requests.

Test Plan: Used `bin/webhook call ...` to complete requests to a test endpoint.

Maniphest Tasks: T11330

Differential Revision: https://secure.phabricator.com/D19045
2018-02-09 13:55:04 -08:00
epriestley
9386e436fe Remove red coloration from "Logout" menu item
Summary: I made the red stronger (always visible, not just a hover state) for the "Mute" feature, but this made Logout look a little intense. Just make it normal-colored, logging out isn't a big deal.

Test Plan: No longer saw bright red logout action in profile dropdown menu.

Differential Revision: https://secure.phabricator.com/D19044
2018-02-09 05:26:38 -08:00
epriestley
09b446b269 Don't run older mail setup checks if "cluster.mailers" is configured
Summary: Ref T12677. Skip these checks if we're doing the new stuff. Also, allow priority to be unspecified.

Test Plan: Will deploy.

Maniphest Tasks: T12677

Differential Revision: https://secure.phabricator.com/D19043
2018-02-08 17:51:32 -08:00
epriestley
7d4362690f Fix transposed name/email in Mailgun adapter
Summary: Ref T12677. This argument order was swapped.

Test Plan: Will push/verify.

Maniphest Tasks: T12677

Differential Revision: https://secure.phabricator.com/D19042
2018-02-08 17:49:32 -08:00
epriestley
261a4a0e51 Add inline comment counts to the filetree view
Summary: See PHI356. Adds inline comment and done counts to the filetree. Also makes the filetree wider by default.

Test Plan: Fiddled with filetrees in different browsers on different revisions. Added inlines, marked them done/undone.

Differential Revision: https://secure.phabricator.com/D19041
2018-02-08 17:15:36 -08:00
epriestley
6ea1b8df9b Colorize filetree for adds, moves, and deletes
Summary: See PHI356. Makes it easier to pick out change types in the filetree view in Differential.

Test Plan: Created a diff with adds, copies, moves, deletions, and binary files. Viewed in Differential, had an easier time picking stuff out.

Differential Revision: https://secure.phabricator.com/D19040
2018-02-08 16:11:35 -08:00
epriestley
e26a784dcf Allow tasks to be filtered and ordered by closed date
Summary: Depends on D19038. Fixes T4434. Updates the SearchEngine and Query to handle these fields.

Test Plan: Filtered and ordered by date and closer.

Maniphest Tasks: T4434

Differential Revision: https://secure.phabricator.com/D19039
2018-02-08 15:42:26 -08:00
epriestley
4c4707e467 Provide task closed date via Conduit API, data export pipeline, and in list UI
Summary:
Depends on D19037. Ref T4434. Adds closed date to `maniphest.search` and "Export Data".

When a task has been closed, show the closed date with a checkmark in the UI instead of the modified date.

Test Plan:
  - Exported data to CSV, saw close information.
  - Saw close information in `/maniphest/`.
  - Queried for close information via `maniphest.search`.

Maniphest Tasks: T4434

Differential Revision: https://secure.phabricator.com/D19038
2018-02-08 15:41:54 -08:00
epriestley
f028aa6f60 Track closed date and closing user for tasks explicitly
Summary:
Ref T4434. Although some of the use cases for this data are better fits for Facts, this data is reasonable to track separately.

I have an approximate view of it already ("closed, ordered by date modified") that's useful to review things that were fixed recently. This lets us make that view more effective.

This just adds (and populates) the storage. Followups will add Conduit, Export, Search, and UI support.

This is slightly tricky because merges work oddly (see T13020).

Test Plan:
  - Ran migration, checked database for sensible results.
  - Created a task in open/closed status, got the right database values.
  - Modified a task to close/open it, got the right values.
  - Merged an open task, got updates.

Maniphest Tasks: T4434

Differential Revision: https://secure.phabricator.com/D19037
2018-02-08 15:40:49 -08:00
epriestley
d1e273daf6 Remove completely pointless load of every repository when viewing a repository URI
Summary:
See D18176. This query has no effect (other than wasting resources) and the result is unused.

`$repository` already has the URI loaded because we load them unconditionally during request initialization.

Test Plan: Viewed repository URIs.

Subscribers: jmeador

Differential Revision: https://secure.phabricator.com/D19036
2018-02-08 12:47:48 -08:00
epriestley
ab04d2179b Add "Mute/Unmute" for subscribable objects
Summary: Ref T13053. See PHI126. Add an explicit "Mute" action to kill mail and notifications for a particular object.

Test Plan: Muted and umuted an object while interacting with it. Saw mail route appropriately.

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19033
2018-02-08 11:06:22 -08:00
epriestley
bca9c08953 Add an "Acting user" field to Herald
Summary:
Ref T13053. Fixes T7804. Adds "Acting user" so you can have "always email me" stuff skip things you did or keep an eye on suspicious interns.

For the test console, the current user is the acting user.

For pushes, the pusher is the acting user.

Test Plan: Wrote acting user rules, triggered them via test console and via multiple actors on real objects.

Maniphest Tasks: T13053, T7804

Differential Revision: https://secure.phabricator.com/D19031
2018-02-08 09:52:18 -08:00
epriestley
6186f0aa91 Briefly document mail stamps and remove obsolete header documentation
Summary: Fixes T10189. Ref T13053. We haven't sent these headers in a very long time. Briefly mention the new stamps header instead, although I expect to integrate stamp documentation into the UI in a more cohesive way in the future.

Test Plan: Read documentation.

Maniphest Tasks: T13053, T10189

Differential Revision: https://secure.phabricator.com/D19030
2018-02-08 09:31:12 -08:00
epriestley
bae9f459ab Pass HTML bodies to push email
Summary: Depends on D19028. Ref T13053. Fixes T6576. An HTML body was built here, but not passed to the actual mail message.

Test Plan: Will verify production push mail.

Maniphest Tasks: T13053, T6576

Differential Revision: https://secure.phabricator.com/D19029
2018-02-08 09:12:03 -08:00
epriestley
a8f937d313 Only add the Mail "STAMPS" body section if there are stamps
Summary: Ref T13053. Some mail (like push notification mail) doesn't currently generate any stamps. Drop this section if there aren't any stamps on the mail.

Test Plan: Will check push mail in production.

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19028
2018-02-08 09:09:28 -08:00
epriestley
942b17a980 Improve correctness of email address escaping in Mailgun/Postmark
Summary: Ref T13053. Uses the changes in D19026 to escape mail addresses. Those rules may not be right yet, but they're at least all in one place, have test coverage, and aren't obviously incorrect.

Test Plan: Will vet this more extensively when re-testing all mailers.

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19027
2018-02-08 09:02:50 -08:00
epriestley
948b0ceca4 Configure a whitelist of remote addresses for Postmark inbound webhooks
Summary:
Ref T13053. Postmark support recommends testing requests against a whitelist of known remote addresses to determine request authenticity. Today, the list can be found here:

<https://postmarkapp.com/support/article/800-ips-for-firewalls>

This is potentially less robust than, e.g., HMAC verification, since they may need to add new datacenters or support IPv6 or something. Users might also have weird network topologies where everything is proxied, and this makes testing/simulating more difficult.

Allow users to configure the list so that they don't need to hack things apart if Postmark adds a new datacenter or remote addresses are unreliable for some other reason, but ship with safe defaults for today.

Test Plan:
Tried to make local requests, got kicked out. Added `0.0.0.0/0` to the list, stopped getting kicked out.

I don't have a convenient way to route real Postmark traffic to my development laptop with an authentic remote address so I haven't verified that the published remote address is legitimate, but I'll vet that in production when I go through all the other mailers.

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19025
2018-02-08 08:23:14 -08:00
epriestley
2bb4fc9ece Fix a Phortune billing issue where subscription autopay could charge disabled cards
Summary:
See support email. There's nothing tricky here, we were just missing a check. The different parts of this got built at different times so I think this was simply overlooked.

Also add a redundant check just to future-proof this and be on the safe side.

Test Plan: Used `bin/phortune invoice` to charge a pact subscription. After deleting the card, the charge failed with an appropriate error.

Reviewers: amckinley

Differential Revision: https://secure.phabricator.com/D19020
2018-02-08 06:30:59 -08:00
epriestley
d0a2e3c54f Fix an issue where some Differential edit pathways may not have reviewers attached
Summary:
Depends on D19021. Ref T13053. When you "Subscribe", or make some other types of edits, we don't necessarily have reviewer data, but may now need it to do the new recipient list logic.

I don't have a totally clean way to deal with this in the general case in mind, but just load it for now so that things don't fatal.

Test Plan: Subscribed to a revision with the "Subscribe" action.

Reviewers: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19022
2018-02-08 06:30:00 -08:00
epriestley
1cd3a59378 When users resign from revisions, stop expanding projects/packages to include them
Summary:
Depends on D19019. Ref T13053. Fixes T12689. See PHI178.

Currently, if `@alice` resigns from a revision but `#alice-fan-club` is still a subscriber or reviewer, she'll continue to get mail. This is undesirable.

When users are associated with an object but have explicitly disengaged in an individal role (currently, only resign in audit/differential) mark them "unexpandable", so that they can no longer be included through implicit membership in a group (a project or package).

`@alice` can still get mail if she's a explicit recipient: as an author, owner, or if she adds herself back as a subscriber.

Test Plan:
  - Added `@ducker` and `#users-named-ducker` as reviewers. Ducker got mail.
  - Resigned as ducker, stopped getting future mail.
  - Subscribed explicitly, got mail again.
  - (Plus some `var_dump()` sanity checking in the internals.)

Reviewers: amckinley

Maniphest Tasks: T13053, T12689

Differential Revision: https://secure.phabricator.com/D19021
2018-02-08 06:29:13 -08:00
epriestley
f214abb63f When a change removes recipients from an object, send them one last email
Summary:
Depends on D19018. Fixes T4776. Ref T13053. When you remove someone from an object's recipient list (for example, by removing them a reviewer, auditor, subscriber, owner or author) we currently do not send them mail about it because they're no longer connected to the object.

In many of these cases (Commandeer, Reassign) the actual action in the UI adds them back to the object somehow (as a reviewer or subscriber, respectively) so this doesn't actually matter. However, there's no recovery mechanism for reviewer or subscriber removal.

This is slightly bad from a policy/threat viewpoint since it means an attacker can remove all the recipients of an object "somewhat" silently. This isn't really silent, but it's less un-silent than it should be.

It's also just not very good from a human interaction perspective: if Alice removes Bob as a reviewer, possibly "against his will", he should be notified about that. In the good case, Alice wrote a nice goodbye note that he should get to read. In the bad case, he should get a chance to correct the mistake.

Also add a `removed(@user)` mail stamp so you can route these locally if you want.

Test Plan:
  - Created and edited some different objects without catching anything broken.
  - Removed subscribers from tasks, saw the final email include the removed recipients with a `removed()` stamp.

I'm not totally sure this doesn't have any surprising behavior or break any weird objects, but I think anything that crops up should be easy to fix.

Reviewers: amckinley

Subscribers: sophiebits

Maniphest Tasks: T13053, T4776

Differential Revision: https://secure.phabricator.com/D19019
2018-02-08 06:28:11 -08:00
epriestley
dbe479f0d9 Don't send error/exception mail to unverified addresses
Summary:
Depends on D19017. Fixes T12491. Ref T13053. After SES threw us in the dungeon for sending mail to a spamtrap we changed outbound mail rules to stop sending to unverified addresses, except a small amount of registration mail which we can't avoid.

However, we'll still reply to random inbound messages with a helpful error, even if the sender is unverified.

Instead, only send exception mail back if we know who the sender is.

Test Plan: Processed inbound mail with `scripts/mail/mail_handler.php`. No more outbound mail for "bad address", etc. Still got outbound mail for "unknown command !quack".

Reviewers: amckinley

Maniphest Tasks: T13053, T12491

Differential Revision: https://secure.phabricator.com/D19018
2018-02-08 06:26:16 -08:00
epriestley
5792032dc9 Support Postmark inbound mail via webhook
Summary: Depends on D19016. Ref T13053. Adds a listener for the Postmark webhook.

Test Plan:
Processed some test mail locally, at least:

{F5416053}

Reviewers: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19017
2018-02-08 06:25:26 -08:00
epriestley
0986c7f673 Add a "View Object" button on the web mail view page
Summary:
Depends on D19015. Ref T13053. Currently, we don't link up hyperlinks in the body of mail viewed in the web UI. We should, but this is a little tricky (see T13053#235074).

As a general improvement to make working with "Must Encrypt" mail less painful, add a big button to jump to the related object.

Test Plan: {F5415990}

Reviewers: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19016
2018-02-08 06:24:19 -08:00
epriestley
085221b0d6 In HTML mail, make the text for mail stamps in mail bodies smaller and lighter
Summary: Depends on D19014. Ref T13053.

Test Plan: Used `./bin/mail show-outbound --id <id> --dump-html > out.html && open out.html` to look at HTML mail, saw smaller, lighter stamp text with better spacing.

Reviewers: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19015
2018-02-08 06:23:37 -08:00
epriestley
6e5df2dd71 Document that disabling "metamta.one-mail-per-recipient" leaks recipients for "Must Encrypt"
Summary:
Depends on D19013. Ref T13053. When mail is marked "Must Encrypt", we normally do not include recipient information.

However, when `metamta.one-mail-per-recipient` is disabled, the recipient list will leak in the "To" and "Cc" headers. This interaction is probably not very surprising, but document it explicitly for completeness.

(Also use "Mail messages" instead of "Mails".)

Test Plan: Read documentation in the "Config" application.

Reviewers: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19014
2018-02-08 06:23:08 -08:00
epriestley
aa74af1983 Remove all "originalTitle"/"originalName" fields from objects
Summary:
Depends on D19012. Ref T13053. In D19012, I've changed "Thread-Topic" to always use PHIDs.

This change drops the selective on-object storage we have to track the original, human-readable title for objects.

Even if we end up backing out the "Thread-Topic" change, we'd be better off storing this in a table in the Mail app which just has `<objectPHID, first subject we used when sending mail for that object>`, since then we get the right behavior without needing every object to have this separate field.

Test Plan: Grepped for `original`, `originalName`, `originalTitle`, etc.

Reviewers: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19013
2018-02-08 06:22:03 -08:00
epriestley
f090fa7426 Use object PHIDs for "Thread-Topic" headers in mail
Summary:
Depends on D19009. Ref T13053. For "Must Encrypt" mail, we must currently strip the "Thread-Topic" header because it sometimes contains sensitive information about the object.

I don't actually know if this header is useful or anyting uses it. My understanding is that it's an Outlook/Exchange thing, but we also implement "Thread-Index" which I think is what Outlook/Exchange actually look at. This header may have done something before we implemented "Thread-Index", or maybe never done anything. Or maybe older versions of Excel/Outlook did something with it and newer versions don't, or do less. So it's possible that an even better fix here would be to simply remove this, but I wasn't able to convince myself of that after Googling for 10 minutes and I don't think it's worth hours of installing Exchange/Outlook to figure out. Instead, I'm just trying to simplify our handling of this header for now, and maybe some day we'll learn more about Exchange/Outlook and can remove it.

In a number of cases we already use the object monogram or PHID as a "Thread-Topic" without users ever complaining, so I think that if this header is useful it probably isn't shown to users, or isn't shown very often (e.g., only in a specific "conversation" sub-view?). Just use the object PHID (which should be unique and stable) as a thread-topic, everywhere, automatically.

Then allow this header through for "Must Encrypt" mail.

Test Plan: Processed some local mail, saw object PHIDs for "Thread-Topic" headers.

Reviewers: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19012
2018-02-08 06:21:00 -08:00
epriestley
19b3fb8863 Add a Postmark mail adapter so it can be configured as an outbound mailer
Summary: Depends on D19007. Ref T12677.

Test Plan: Used `bin/mail send-test ... --mailer postmark` to deliver some mail via Postmark.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12677

Differential Revision: https://secure.phabricator.com/D19009
2018-02-08 06:18:23 -08:00
epriestley
1f53aa27e4 Add unit tests for mail failover behaviors when multiple mailers are configured
Summary: Depends on D19006. Ref T13053. Ref T12677. When multiple mailers are configured but one or more fail, test that we recover (or don't) appropriately.

Test Plan: Ran unit tests.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053, T12677

Differential Revision: https://secure.phabricator.com/D19007
2018-02-08 06:17:49 -08:00
epriestley
9947eee182 Add some test coverage for mailers configuration
Summary: Depends on D19005. Ref T12677. Ref T13053. Tests that turning `cluster.mailers` into an actual list of mailers more or less works as expected.

Test Plan: Ran unit tests.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053, T12677

Differential Revision: https://secure.phabricator.com/D19006
2018-02-08 06:17:19 -08:00
epriestley
994d2e8e15 Use "cluster.mailers" if it is configured
Summary:
Depends on D19004. Ref T13053. Ref T12677. If the new `cluster.mailers` is configured, make use of it. Also use it in the Sengrid/Mailgun inbound stuff.

Also fix a bug where "Must Encrypt" mail to no recipients could fatal because no `$mail` was returned.

Test Plan: Processed some mail locally. The testing on this is still pretty flimsy, but I plan to solidify it in an upcoming change.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053, T12677

Differential Revision: https://secure.phabricator.com/D19005
2018-02-08 06:12:54 -08:00
epriestley
4236952cdb Add a bin/config set <key> --stdin < value.json flag to make CLI configuration of complex values easier
Summary:
Depends on D19003. Ref T12677. Ref T13053. For the first time, we're requiring CLI configuration of a complex value (not just a string, integer, bool, etc) to do something fairly standard (send mail).

Users sometimes have very reasonable difficulty figuring out how to `./bin/config set key <some big JSON mess>`. Provide an easy way to handle this and make sure it gets appropriate callouts in the documentation.

(Also, hide the `cluster.mailers` value rather than just locking it, since it may have API keys or SMTP passwords.)

Test Plan: Read documentation, used old and new flags to set configuration.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053, T12677

Differential Revision: https://secure.phabricator.com/D19004
2018-02-08 06:09:09 -08:00
epriestley
c868ee9c07 Introduce and document a new cluster.mailers option for configuring multiple mailers
Summary:
Depends on D19002. Ref T13053. Ref T12677. Adds a new option to allow configuration of multiple mailers.

Nothing actually uses this yet.

Test Plan: Tried to set it to various bad values, got reasonable error messages. Read documentation.

Reviewers: amckinley

Maniphest Tasks: T13053, T12677

Differential Revision: https://secure.phabricator.com/D19003
2018-02-08 06:08:34 -08:00
epriestley
7f2c90fbd1 Prepare for multiple mailers of the same type
Summary:
Depends on D19000. Ref T13053. Ref T12677. Currently, most mailers are configured with a bunch of `<mailer>.setting-name` global config options.

This means that you can't configure two different SMTP servers, which is a reasonable thing to want to do in the brave new world of mail failover.

It also means you can't configure two Mailgun accounts or two SES accounts. Although this might seem a little silly, we've had more service disruptions because of policy issues / administrative error (where a particular account was disabled) than actual downtime, so maybe it's not completely ridiculous.

Realign mailers so they can take configuration directly in an explicit way. A later change will add new configuration to take advantage of this and let us move away from having ~10 global options for this stuff eventually.

(This also makes writing third-party mailers easier.)

Test Plan:
Processed some mail, ran existing unit tests. But I wasn't especially thorough.

I expect later changes to provide some tools to make this more testable, so I'll vet each provider more thoroughly and add coverage for multiple mailers after that stuff is ready.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053, T12677

Differential Revision: https://secure.phabricator.com/D19002
2018-02-08 06:00:59 -08:00
epriestley
7765299f83 Mask the sender for "Must Encrypt" mail
Summary:
Depends on D18998. Ref T13053. When we send "Must Encrypt" mail, we currently send it with a normal "From" address.

This discloses a little information about the object (for example, if the Director of Silly Walks is interacting with a "must encrypt" object, the vulnerability is probably related to Silly Walks), so anonymize who is interacting with the object.

Test Plan: Processed some mail. (The actual final "From" is ephemeral and a little tricky to examine and I didn't actually transmit mail over the network, but it should be obvious if this works or not on `secure`.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D19000
2018-02-08 05:59:35 -08:00
epriestley
1485debcbd Prepare mail transmission to support failover across multiple mailers
Summary:
Ref T13053. Ref T12677. This restructures the calls and error handling logic so that we can pass in a list of multiple mailers and get retry logic.

This doesn't actually ever use multiple mailers yet, and shouldn't change any behavior. I'll add multiple-mailer coverage a little further in, since there's currently no way to effectively test which of several mailers ended up transmitting a message.

Test Plan:
  - This has test coverage; tests still pass.
  - Poked around locally doing things that send mail, saw mail appear to send. I'm not attached to a real mailer though so my confidence in local testing is only so-so.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053, T12677

Differential Revision: https://secure.phabricator.com/D18998
2018-02-08 05:49:06 -08:00
epriestley
150a04791c Fix bad NUX link in Legalpad search view
Summary:
See <https://discourse.phabricator-community.org/t/clicking-the-create-a-document-button-on-fresly-installed-phabricators-legalpad-results-in-404/1088>.

This URI isn't correct.

Test Plan: Visited {nav Use Results > New User State} in developer mode, clicked green button. Before: 404. After: taken to the edit screen.

Differential Revision: https://secure.phabricator.com/D19024
2018-02-08 05:47:02 -08:00
epriestley
a5bbadbaba Fix another Git 2.16.0 CLI compatibility issue
Summary:
This command also needs a "." instead of an empty string now.

(This powers the file browser typeahead in Diffusion.)

Test Plan: Will test in production since there's still no easy 2.16 installer for macOS.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19010
2018-02-07 17:54:39 -08:00
epriestley
56bf069080 Try running Herald when performing inverse edge edits
Summary:
Ref T13053. When you mention one object on another (or link two objects together with an action like "Edit Parent Revisions"), we write a transaction on each side to add the "alice added subtask X" and "alice added parent task Y" items to the timeline.

This behavior now causes problems in T13053 with the "Must Encrypt" flag because it prevents the flag from being applied to the corresponding "inverse edge" mail.

This was added in rP5050389f as a quick workaround for a fatal related to Editors not having enough data to apply Herald on mentions. However, that was in 2014, and since then:

  - Herald got a significant rewrite to modularize all the rules and adapters.
  - Editing got a significant upgrade in EditEngine and most edit workflows now operate through EditEngine.
  - We generally do more editing on more pathways, everything is more modular, and we have standardized how data is loaded to a greater degree.

I suspect there's no longer a problem with just running Herald here, and can't reproduce one. If anything does crop up, it's probably easy (and desirable) to fix it.

This makes Herald fire a little more often: if someone writes a rule, mentioning or creating a relationship to old tasks will now make the rule act. Offhand, that seems fine. If it turns out to be weird, we can probably tailor Herald's behavior.

Test Plan:
I wasn't able to break anything:

  - Mentioned a task on another task (original issue).
  - Linked tasks with commits, mocks, revisions.
  - Linked revisions with commits, tasks.
  - Mentioned users, revisions, and commits.
  - Verified that mail generated by creating links (e.g., Revision > Edit Tasks) now gets the "Must Encrypt" flag properly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D18999
2018-02-06 12:18:56 -08:00
epriestley
1bf64e5cbc Add Differential and Herald mail stamps and some refinements
Summary:
Ref T13053. Adds revision stamps (status, reviewers, etc). Adds Herald rule stamps, like the existing X-Herald-Rules header.

Removes the "self" stamps, since you can just write a rule against `whatever(@epriestley)` equivalently. If there's routing logic around this, it can live in the routing layer. This avoids tons of self-actor, self-mention, self-reviewer, self-blocking-reviewer, self-resigned-reviewer, etc., stamps.

Use `natcasesort()` instead of `sort()` so that numeric values (like monograms) sort `9, 80, 700` instead of `700, 80, 9`.

Remove the commas from rendering since they don't really add anything.

Test Plan: Edited tasks and revisions, looked at mail stamps, saw stamps that looked pretty reasonable (with no more self stuff, no more commas, sorting numbers, and Herald stamps).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D18997
2018-02-06 04:06:07 -08:00
epriestley
7d475eb09a Add more mail stamps: tasks, subscribers, projects, spaces
Summary: Ref T13053. Adds task stamps plus the major infrastructure applications.

Test Plan: {F5413058}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D18996
2018-02-06 04:05:46 -08:00
epriestley
3131e733a8 Add Editor-based mail stamps: actor, via, silent, encrypted, new, mention, self-actor, self-mention
Summary: Ref T13053. Adds more mail tags with information available on the Editor object.

Test Plan: Banged around in Maniphest, viewed the resulting mail, all the stamps seemed to align with reality.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D18995
2018-02-06 04:04:52 -08:00
epriestley
9de54aedb5 Remove inconsistent and confusing use of the term "multiplex" in mail
Summary:
Ref T13053. Because I previously misunderstood what "multiplex" means, I used it in various contradictory and inconsistent ways.

We can send mail in two ways: either one mail to everyone with a big "To" and a big "Cc" (not default; better for mailing lists) or one mail to each recipient with just them in "To" (default; better for almost everything else).

"Multiplexing" is combining multiple signals over a single channel, so it more accurately describes the big to/cc. However, it is sometimes used to descibe the other approach. Since it's ambiguous and I've tainted it through misuse, get rid of it and use more clear language.

(There's still some likely misuse in the SMS stuff, and a couple of legitimate uses in other contexts.)

Test Plan: Grepped for `multiplex`, saw less of it.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D18994
2018-02-06 04:04:34 -08:00
epriestley
c3f95bc410 Add basic support for mail "stamps" to improve client mail routing
Summary:
Ref T10448. Currently, we use "mail tags" (in {nav Settings > Email Preferences})  to give users some ability to route mail. There are a number of major issues with this:

  - It isn't modular and can't be extended by third-party applications.
  - The UI is a giant mess of 5,000 individual settings.
  - Settings don't map clearly to actual edits.
  - A lot of stuff isn't covered by any setting.

This adds a new system, called "mail stamps", which is similar to "mail tags" but tries to fix all these problems.

I called these "stamps" because: stamps make sense with mail; we can't throw away the old system just yet and need to keep it around for a bit; we don't use this term for anything else; it avoids confusion with project tags.

(Conceptually, imagine these as ink stamps like "RETURN TO SENDER" or "FRAGILE", not actual postage stamps.)

The only real "trick" here is that later versions of this will need to enumerate possible stamps for an object and maybe all possible stamps for all objects in the system. This is why stamp generation is separated into a "template" phase and a "value" phase. In future changes, the "template" phase can be used on its own to generate documentation and typeaheads and let users build rules. This may need some more refinement before it really works since I haven't built any of that yet.

Also adds a preference for getting stamps in the header only (default) or header and body (better for Gmail, which can't route based on headers).

Test Plan:
Fiddled with preference, sent some mail and saw a "STAMPS" setting in the body and an "X-Phabricator-Stamps" header.

{F5411694}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T10448

Differential Revision: https://secure.phabricator.com/D18991
2018-02-06 04:04:13 -08:00
epriestley
ef121b3e17 Fix a Herald repetition policy selection error for rule types which support only one policy
Summary:
Ref T13048. See <https://discourse.phabricator-community.org/t/configuring-commit-hook-commit-content-rules-fail-with-exception/1077/3>.

When a rule supports only one repetition policy (always "every time") like "Commit Hook" rules, we don't render a control for `repetition_policy` and fail to update it when saving.

Before the changes to support the new "if the rule did not match the last time" policy, this workflow just defaulted to "every time" if the input was invalid, but this was changed by accident in D18926 when I removed some of the toInt/toString juggling code.

(This patch also prevents users from fiddling with the form to create a rule which evaluates with an invalid policy; this wasn't validated before.)

Test Plan:
  - Created new "Commit Hook" (only one policy available) rule.
  - Saved existing "Commit Hook" rule.
  - Created new "Task" (multiple policies) rule.
  - Saved existing Task rule.
  - Set task rule to each repetition policy, saved, verified the save worked.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13048

Differential Revision: https://secure.phabricator.com/D18992
2018-02-05 13:35:36 -08:00
epriestley
956c4058e6 Add a bin/conduit call support binary
Summary:
Ref T13060. See PHI343. Triaging this bug required figuring out where in the pipeline UTF8 was being dropped, and bisecting the pipeline required making calls to Conduit.

Currently, there's no easy way to debug/inspect arbitrary Conduit calls, especially when they are `diffusion.*` calls which route to a different host (even if you have a real session and use the web console for these, you just see an HTTP service call to the target host in DarkConsole).

Add a `bin/conduit` utility to make this kind of debugging easier, with an eye toward the Phacility production cluster (or other similar clusters) specifically.

Test Plan:
  - Ran `echo '{}' | bin/conduit call --method conduit.ping --input -` and similar.
  - Used a similar approach to successfully diagnose the UTF8 issue in T13060.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13060

Differential Revision: https://secure.phabricator.com/D18987
2018-02-05 12:20:49 -08:00
epriestley
55f7cdb99b Fix a bad classname reference in the "Must Encrypt" action 2018-02-02 14:57:25 -08:00
epriestley
6d90c7ad92 Save mail attachments in Files, not on the actual objects
Summary:
Depends on D18985. Ref T13053. See PHI125. Currently, mail attachments are just encoded onto the actual objects in the `MetaMTAMail` table.

This fails if attachments can't be encoded in JSON -- e.g., they aren't UTF8. This happens most often when revisions or commits attach patches to mail and those patches contain source code changes for files that are not encoded in UTF8.

Instead, save attachments in (and load attachments from) Files.

Test Plan: Enabled patches for mail, created a revision, saw it attach a patch. Viewed mail in web UI, saw link to download patch. Followed link, saw sensible file. Checked database, saw a `filePHID`. Destroyed mail with `bin/remove destroy`, saw attached files also destroyed.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D18986
2018-02-02 14:38:08 -08:00
epriestley
eb06aca951 Support DestructionEngine in MetaMTAMail
Summary:
Depends on D18984. Ref T13053. See D13408 for the original change and why this doesn't use DestructionEngine right now. The quick version is:

  - It causes us to write a destruction log, which is slightly silly (we're deleting one thing and creating another).
  - It's a little bit slower than not using DestructionEngine.

However, it gets us some stuff for free that's likely relevant now (e.g., Herald Transcript cleanup) and I'm planning to move attachments to Files, but want to be able to delete them when mail is destroyed.

The destruction log is a touch silly, but those records are very small and that log gets GC'd later without generating new logs. We could silence the log from the GC if it's ever an issue.

Test Plan: Used `bin/remove destroy` and `bin/garbage collect --collector mail.sent` to destroy mail and collect garbage.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D18985
2018-02-02 14:37:33 -08:00
epriestley
cbe4e68c07 Add a Herald action to trigger "Must Encrypt" for mail
Summary: Depends on D18983. Ref T13053. Adds a new Herald action to activate the "must encrypt" flag and drop mail content.

Test Plan:
  - Created a new Herald rule:

{F5407075}

  - Created a "dog task" (woof woof, unsecure) and a "duck task" (quack quack, secure).
  - Viewed mail for both in `bin/mail` and web UI, saw appropriate security/encryption behavior.
  - Viewed "Must Encrypt" in "Headers" tab for the duck mail, saw why the mail was encrypted (link to Herald rule).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D18984
2018-02-02 14:35:26 -08:00
epriestley
7b2b5cd91e Add basic support for a "Must Encrypt" mail flag which prevents unsecured content transmission
Summary:
Ref T13053. See PHI291. For particularly sensitive objects (like security issues), installs may reasonably wish to prevent details from being sent in plaintext over email.

This adds a "Must Encrypt" mail behavior, which discards mail content and all identifying details, replacing it with a link to the `/mail/` application. Users can follow the link to view the message over HTTPS.

The flag discards body content, attachments, and headers which imply things about the content of the object. It retains threading headers and headers which may uniquely identify the object as long as they don't disclose anyting about the content.

The `bin/mail list-outbound` command now flags these messages with a `#` mark.

The `bin/mail show-outbound` command now shows sent/suppressed headers and the body content as delivered (if it differs from the original body content).

The `/mail/` web UI now shows a tag for messages marked with this flag.

For now, there is no way to actually set this flag on mail.

Test Plan:
  - Forced this flag on, made comments and took actions to send mail.
  - Reviewed mail with `bin/mail` and `/mail/` in the web UI, saw all content information omitted.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13053

Differential Revision: https://secure.phabricator.com/D18983
2018-02-02 14:34:34 -08:00
epriestley
032f5b2294 Allow revisions to revert commits and one another, and commits to revert revisions
Summary:
Ref T13057. This makes "reverts" syntax more visible and useful. In particular, you can now `Reverts Dxx` in a revision or commit, and `Reverts <hash>` from a revision.

When you do, the corresponding object will get a more-visible cross-reference marker in its timeline:

{F5405517}

From here, we can look at surfacing revert information more heavily, since we can now query it on revision/commit pages via edges.

Test Plan: Used "reverts <hash>" and "reverts <revision>" in Differential and Diffusion, got sensible results in the timeline.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13057

Differential Revision: https://secure.phabricator.com/D18978
2018-02-02 08:25:58 -08:00
epriestley
f535981c0d Fix a missing getSSHUser() callsite
Summary:
See <https://discourse.phabricator-community.org/t/after-upgrade-git-lfs-push-ends-up-in-call-to-undefined-method-on-diffusion-git-lfs-authenticate-workflow/1047/1>.

I renamed this method in D18912 but missed this callsite since the workflow doesn't live alongside the other ones.

Test Plan: Ran `git push` in an LFS repository over SSH. Before: fatal; after: clean push.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18977
2018-01-31 15:34:12 -08:00
epriestley
6d5f265a57 Accept null via conduit.edit to unassign a task
Summary:
See <https://discourse.phabricator-community.org/t/maniphest-edit-to-unassign-owner-documentation-is-wrong/1053>. This unusual field doesn't actually accept `null`, although the documentation says it does and that was the intent.

Accept `null`, and show `phid|null` in the docs.

Test Plan: Viewed docs, saw `phid|null`. Unassigned with `null`.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18976
2018-01-31 15:33:52 -08:00
epriestley
c9df8f77c8 Fix transcription of single-value bulk edit fields ("Assign to")
Summary: See PHI333. Some of the cleanup at the tail end of the bulk edit changes made "Assign To" stop working properly, since we don't strip the `array(...)` off the `array(PHID)` value we receive.

Test Plan:
  - Used bulk editor to assign and unassign tasks (single value datasource).
  - Used bulk editor to change projects (multi-value datasource).

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18975
2018-01-31 10:56:16 -08:00
epriestley
1e3d1271ad Make push log "flags", "reject code" human readable; add crumbs to pull/push logs
Summary:
Depends on D18972. Ref T13049.

Currently, the "flags" columns renders an inscrutible bitmask which you have to go hunt down in the code. Show a list of flags in human-readable text instead.

The "code" column renders a meaningless integer code. Show a text description instead.

The pull logs and push logs pages don't have a crumb to go back up out of the current query. Add one.

Test Plan: Viewed push logs, no more arcane numbers. Saw and clicked crumbs on each log page.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18973
2018-01-30 15:45:58 -08:00
epriestley
ff98f6f522 Make the remote address rules for Settings > Activity Logs more consistent
Summary:
Depends on D18971. Ref T13049. The rule is currently "you can see IP addresses for actions which affect your account".

There's some legitimate motivation for this, since it's good if you can see that someone you don't recognize has been trying to log into your account.

However, this includes cases where an administrator disables/enables your account, or promotes/demotes you to administrator. In these cases, //their// IP is shown!

Make the rule:

  - Administrators can see it (consistent with everything else).
  - You can see your own actions.
  - You can see actions which affected you that have no actor (these are things like login attempts).
  - You can't see other stuff: usually, administrators changing your account settings.

Test Plan: Viewed activity log as a non-admin, no longer saw administrator's IP address disclosed in "Demote from Admin" log.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18972
2018-01-30 15:45:39 -08:00
epriestley
8a2863e3f7 Change the "can see remote address?" policy to "is administrator?" everywhere
Summary:
Depends on D18970. Ref T13049. Currently, the policy for viewing remote addresses is:

  - In activity logs: administrators.
  - In push and pull logs: users who can edit the corresponding repository.

This sort of makes sense, but is also sort of weird. Particularly, I think it's kind of hard to understand and predict, and hard to guess that this is the behavior we implement. The actual implementation is complex, too.

Instead, just use the rule "administrators can see remote addresses" consistently across all applications. This should generally be more strict than the old rule, because administrators could usually have seen everyone's address in the activity logs anyway. It's also simpler and more expected, and I don't really know of any legit use cases for the "repository editor" rule.

Test Plan: Viewed pull/push/activity logs as non-admin. Saw remote addresses as an admin, and none as a non-admin.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18971
2018-01-30 15:45:23 -08:00
epriestley
75bc86589f Add date range filtering for activity, push, and pull logs
Summary: Ref T13049. This is just a general nice-to-have so you don't have to export a 300MB file if you want to check the last month of data or whatever.

Test Plan: Applied filters to all three logs, got appropriate date-range result sets.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18970
2018-01-30 15:36:22 -08:00
epriestley
0d5379ee17 Fix an export bug where queries specified in the URI ("?param=value") were ignored when filtering the result set
Summary:
Depends on D18968. Ref T13049. Currently, if you visit `/query/?param=value`, there is no `queryKey` for the page but we build a query later on.

Right now, we incorrectly link to `/query/all/export/` in this case (and export too many results), but we should actually link to `/query/<constructed query key>/export/` to export only the desired/previewed results.

Swap the logic around a little bit so we look at the query we're actually executing, not the original URI, to figure out the query key we should use when building the link.

Test Plan: Visited a `/?param=value` page, exported data, got a subset of the full data instead of everything.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18969
2018-01-30 11:19:37 -08:00
epriestley
5b22412f24 Support data export on push logs
Summary: Depends on D18967. Ref T13049. Nothing too fancy going on here.

Test Plan: Exported push logs, looked at the export, seemed sensible.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18968
2018-01-30 11:19:20 -08:00
epriestley
a5b8be0316 Support export of user activity logs
Summary:
Depends on D18966. Ref T13049. Adds export support to user activity logs.

These don't have PHIDs. We could add them, but just make the "phid" column test if the objects have PHIDs or not for now.

Test Plan:
  - Exported user activity logs, got sensible output (with no PHIDs).
  - Exported some users to make sure I didn't break PHIDs, got an export with PHIDs.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18967
2018-01-30 11:12:32 -08:00
epriestley
91108cf838 Upgrade user account activity logs to modern construction
Summary: Depends on D18965. Ref T13049. Move this Query and SearchEngine to be a little more modern, to prepare for Export support.

Test Plan:
  - Used all the query fields, viewed activity logs via People and Settings.
  - I'm not sure the "Session" query is used/useful and may remove it before I'm done here, but I just left it in place for now.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18966
2018-01-30 11:11:47 -08:00
epriestley
b27fd05eef Add a bin/bulk export CLI tool to make debugging and profiling large exports easier
Summary:
Ref T13049. When stuff executes asynchronously on the bulk workflow it can be hard to inspect directly, and/or a pain to test because you have to go through a bunch of steps to run it again.

Make future work here easier by making export triggerable from the CLI. This makes it easy to repeat, inspect with `--trace`, profile with `--xprofile`, etc.

Test Plan:
  - Ran several invalid commands, got sensible error messages.
  - Ran some valid commands, got exported data.
  - Used `--xprofile` to look at the profile for a 300MB dump of 100K tasks which took about 40 seconds to export. Nothing jumped out as sketchy to me -- CustomField wrangling is a little slow but most of the time looked like it was being spent legitimately.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18965
2018-01-30 11:11:13 -08:00
epriestley
84df122085 When exporting more than 1,000 records, export in the background
Summary:
Depends on D18961. Ref T13049. Currently, longer exports don't give the user any feedback, and exports that take longer than 30 seconds are likely to timeout.

For small exports (up to 1,000 rows) continue doing the export in the web process.

For large exports, queue a bulk job and do them in the workers instead. This sends the user through the bulk operation UI and is similar to bulk edits. It's a little clunky for now, but you get your data at the end, which is far better than hanging for 30 seconds and then fataling.

Test Plan: Exported small result sets, got the same workflow as before. Exported very large result sets, went through the bulk flow, got reasonable results out.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18962
2018-01-29 16:08:02 -08:00
epriestley
ea58b6acea Remove the old, non-modular Excel export workflow from Maniphest
Summary:
Depends on D18960. Ref T13049. Now that Maniphest fully supports "Export Data", remove the old hard-coded version.

This is a backward compatibility break with the handful of installs that might have defined a custom export by subclassing `ManiphestExcelFormat`. I suspect this is almost zero installs, and that the additional data in the new format may serve most of the needs of this tiny number of installs. They can upgrade to `ExportEngineExtensions` fairly easily if this isn't true.

Test Plan:
  - Viewed Maniphest, no longer saw the old export workflow.
  - Grepped for `export` and similar strings to try to hunt everything down.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18961
2018-01-29 16:06:13 -08:00
epriestley
c00838878a Implement common infrastructure fields as export extensions
Summary:
Depends on D18959. Ref T13049. Provide tags, subscribers, spaces, and created/modified as automatic extensions for all objects which support them.

(Also, for JSON export, be a little more consistent about exporting `null` instead of empty string when there's no value in a text field.)

Test Plan: Exported users and tasks, saw relevant fields in the export.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18960
2018-01-29 16:05:32 -08:00
epriestley
2ac4e1991b Support new data export infrastructure in Maniphest
Summary: Depends on D18958. Ref T13049. Support the new stuff. There are a couple more fields this needs to strictly improve on the old export, but I'll add them as extensions shortly.

Test Plan: Exported tasks to Excel, saw reasonble-looking data in the export.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18959
2018-01-29 16:04:39 -08:00
epriestley
00b4eae1f4 When PHPExcel is not installed, detect it and provide install instructions
Summary:
Depends on D18957. Ref T13049. To do Excel exports, PHPExcel needs to be installed on the system somewhere.

This library is enormous (1K files, ~100K SLOC), which is why we don't just include it in `externals/`. This install process is a little weird and we could improve it, but users don't seem to have too much difficulty with it. This shouldn't be worse than the existing workflow in Maniphest, and I tried to make it at least slightly more clear.

Test Plan: Uninstalled PHPExcel, got it marked "Unavailable" and got reasonably-helpful-ish guidance on how to get it to work. Reinstalled, exported, got a sheet.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18958
2018-01-29 16:03:34 -08:00
epriestley
61b8c12970 Make the data export format selector remember your last setting
Summary:
Depends on D18956. Ref T13049. Make the "Export Format" selector sticky.

This is partly selfish, since it makes testing format changes a bit easier.

It also seems like it's probably a good behavior in general: if you export to Excel once, that's probably what you're going to pick next time.

Test Plan: Exported to excel. Exported again, got excel as the default option.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18957
2018-01-29 16:01:54 -08:00
epriestley
0409279595 Support Excel as a data export format
Summary:
Depends on D18954. Ref T13049. This brings over the existing Maniphest Excel export pipeline in a generic way.

The `<Type>ExportField` classes know directly that `PHPExcel` exists, which is a little sketchy, but writing an Excel indirection layer sounds like a lot of work and I don't anticipate us changing Excel backends anytime soon, so trying to abstract this feels YAGNI.

This doesn't bring over the install instructions for PHPExcel or the detection of whether or not it exists. I'll bring that over in a future change.

Test Plan: Exported users as Excel, opened them up, got a sensible-looking Excel sheet.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18955
2018-01-29 16:00:41 -08:00
epriestley
a067f64ebb Support export engine extensions and implement an extension for custom fields
Summary:
Depends on D18953. Ref T13049. Allow applications and infrastructure to supplement exportable fields for objects.

Then, implement an extension for custom fields. Only a couple field types (int, string) are supported for now.

Test Plan: Added some custom fields to Users, populated them, exported users. Saw custom fields in the export.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18954
2018-01-29 15:59:58 -08:00
epriestley
8b8a3142b3 Support export of data in files larger than 8MB
Summary:
Depends on D18952. Ref T13049. For files larger than 8MB, we need to engage the chunk storage engine. `PhabricatorFile::newFromFileData()` always writes a single chunk, and can't handle files larger than the mandatory chunk threshold (8MB).

Use `IteratorUploadSource`, which can, and "stream" the data into it. This should raise the limit from 8MB to 2GB (maximum size of a string in PHP).

If we need to go above 2GB we could stream CSV and text pretty easily, and JSON without too much trouble, but Excel might be trickier. Hopefully no one is trying to export 2GB+ datafiles, though.

Test Plan:
  - Changed the JSON exporter to just export 8MB of the letter "q": `return str_repeat('q', 1024 * 1024 * 9);`.
  - Before change: fatal, "no storage engine can store this file".
  - After change: export works cleanly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18953
2018-01-29 15:58:34 -08:00
epriestley
0de6210808 Give data exporters a header row
Summary:
Depends on D18951. Ref T13049. When we export to CSV or plain text, add a header row in the first line of the file to explain what each column means. This often isn't obvious with PHIDs, etc.

JSON has keys and is essentially self-labeling, so don't do anything special.

Test Plan: Exported CSV and text, saw new headers. Exported JSON, no changes.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18952
2018-01-29 15:17:30 -08:00
epriestley
213eb8e93d Define common ID and PHID export fields in SearchEngine
Summary:
Ref T13049. All exportable objects should always have these fields, so make them builtins.

This also sets things up for extensions (like custom fields).

Test Plan: Exported user data, got the same export as before.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18951
2018-01-29 15:17:00 -08:00
epriestley
d8f51dff6e Use the configured viewer more consistently in the Herald commit adapter
Summary: See PHI276. Ref T13048. The fix in D18933 got one callsite, but missed the one in the `callConduit()` method, so the issue isn't fully fixed in production. Convert this adapter to use a real viewer (if one is available) more thoroughly.

Test Plan: Ran rules in test console, saw field values. Will test in production again.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13048

Differential Revision: https://secure.phabricator.com/D18950
2018-01-29 15:00:26 -08:00
epriestley
e5639a8ed9 Write edge transactions in a more compact way
Summary: Depends on D18946. Ref T13051. Begins writing edge transactions as just a list of changed PHIDs.

Test Plan: Added, edited, and removed projects. Reviewed transaction record and database. Saw no user-facing changes but a far more compact database representation.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13051

Differential Revision: https://secure.phabricator.com/D18947
2018-01-29 11:33:58 -08:00
epriestley
de7f836f03 Wrap edge transaction readers in a translation layer
Summary:
Ref T13051. This puts a translation layer between the raw edge data in the transaction table and the UI that uses it.

The intent is to start writing new, more compact data soon. This class give us a consistent API for interacting with either the new or old data format, so we don't have to migrate everything upfront.

Test Plan: Browsed around, saw existing edge transactions render properly in transactions and feed. Added and removed subscribers and projects, saw good transaction rendering.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13051

Differential Revision: https://secure.phabricator.com/D18946
2018-01-29 11:33:41 -08:00
epriestley
162563d40b Move the fix for Git 2.16.0 from the "Mercurial" part of the code to the "Git" part of the code
Summary: Ref T13050. Oh boy. Both of them run `grep`!

Test Plan: Will push again.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13050

Differential Revision: https://secure.phabricator.com/D18945
2018-01-26 13:48:35 -08:00
epriestley
a80d1e7e7d Pass "." to git grep to satisfy "all paths" for Git 2.16.0
Summary:
Ref T13050. See <https://discourse.phabricator-community.org/t/issues-with-git-2-16-0/1004/2>.

`secure` picked up 2.16.0 so this reproduces now: <https://secure.phabricator.com/source/phabricator/browse/master/?grep=dog>

Test Plan: Will push.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13050

Differential Revision: https://secure.phabricator.com/D18944
2018-01-26 13:38:18 -08:00
epriestley
4b5a78e343 Add "you can only enter one value" UI limits to Herald "set status" and "set priority" actions
Summary:
Ref T13048. Fixes T11112. I mostly fixed this before in D18887, but missed that these other actions are also affected. T11112 had a more complete list of missing limits.

(It's possible there are some others somewhere, but this is everything we know about, I think.)

Test Plan: Created rules using these actions, typed stuff into the box, was only able to enter one value.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13048, T11112

Differential Revision: https://secure.phabricator.com/D18943
2018-01-26 13:22:26 -08:00
epriestley
7e7720803c Set GIT_SSH_VARIANT alongside GIT_SSH
A recent version of Git has changed some piece of behavior here and we
now get "fatal: ssh variant 'simple' does not support setting port"
when using a port. Explicitly setting GIT_SSH_VARIANT to `ssh` likely
fixes this.
2018-01-26 13:21:10 -08:00
epriestley
9d69118664 Add a discovery format hint for date fields in SearchEngine UIs
Summary:
See PHI316. Maniphest and other applications currently have controls like `Created After: [_____]` where you just get an empty text field.

Although most formats work -- including relative formats like "3 days ago" -- and we validate inputs so you get an error if you enter something nonsensical, this still isn't very user friendly.

T8060 or some other approach is likely the long term of this control.

In the meantime, add placeholder text to suggest that `YYYY-MM-DD` or `X days ago` will work.

Test Plan: Viewed date inputs, saw placeholder text.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18942
2018-01-26 13:11:10 -08:00
epriestley
fd66ab5579 Always use the same set of transactions to generate mail and mail tags
Summary:
See PHI307. Currently, when reviews undraft, we retroactively add in older activity to the mail ("alice created this revision...").

However, we don't add that activity to the mail tags, so the relevant tags (like "revision created") are dropped forever.

Instead, use the same set of transactions for both mail body and mail tag construction.

This should be obsoleted in the relatively near future by T10448, but it's a better/more correct behavior in general and we probably can't get rid of tags completely for a while.

Test Plan:
Applied patch, created a revision with builds, saw it auto-undraft after builds finished. Used `bin/mail list-outbound` and `bin/mail show-outbound` to see the mail. Verified that it included retroactive text ("created this revision") AND retroactive tags.

Note that the tag for "A new revision is created" is `DifferentialTransaction::MAILTAG_REVIEW_REQUEST` with literal value `differential-review-request`.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18941
2018-01-26 13:09:04 -08:00
epriestley
ad7755d9a9 Fix an issue with symbol lookup identifying path names in Diffusion
Summary:
Depends on D18939. Ref T13047. Symbol lookup can be activated from a diff (in Differential or Diffusion) or from the static view of a file at a particular commit.

In the latter case, we need to figure out the path a little differently. The character and line number approaches still work as written.

Test Plan:
  - Command-clicked symbols in the Diffusion browse view with blame on and off; saw path, line and char populate properly.
  - Command-clicked symbols in Differential diff view to check I didn't break anything.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13047

Differential Revision: https://secure.phabricator.com/D18940
2018-01-26 13:02:20 -08:00
epriestley
fdc36677ba Provide character position information to symbol queries
Summary: Depends on D18937. Ref T13047. When available, provide character positions so external indexers can return more accurate results.

Test Plan: Clicked symbols in Safari, Firefox and Chrome, got sensible-looking character positions.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13047

Differential Revision: https://secure.phabricator.com/D18939
2018-01-26 13:01:57 -08:00
epriestley
3a2d337679 Add a "Revision status" field to Herald for Differential revisions
Summary: See PHI280. We have a similar field for tasks already, this is generally a reasonable sort of thing to support, and the addition of "draft" states means there are some pretty reasonable use cases.

Test Plan:
  - Wrote a status-based ("status is needs revision") Herald rule.
  - Tested it against a "Needs Revision" revision (passed) and a "Changes Planned" revision (failed).

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18938
2018-01-26 13:01:36 -08:00
epriestley
d606eb1c38 When available, pass path, line and repository hints to external symbol queries
Summary:
Depends on D18936. Ref T13047. Third parties can define external symbol sources that let users jump to PHP or Python documentation or query some server.

Give these queries more information so they can try to get better results: the path and line where the symbol appeared, and any known repository scope.

Test Plan: Wrote a fake external source that used this data, command-clicked a symbol in Differential, saw a fake external symbol result.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13047

Differential Revision: https://secure.phabricator.com/D18937
2018-01-26 12:00:44 -08:00
epriestley
c37b6c6633 When users click a symbol in Differential to jump to the definition, include path/line context
Summary:
Ref T13047. In some reasonable cases, knowing the path and line number where a symbol appears is useful in ranking or filtering the set of matching symbols.

Giving symbol sources more information can't hurt, and it's generally free for us to include this context since we just need to grab it out of the document and pass it along.

We can't always get this data (for example, if a user types `s idx` into global search, we have no clue) but this is similar to other types of context which are only available sometimes (like which repository a symbol appears in).

Test Plan: Command-clicked some symbols in 1-up (unified) and 2-up (side-by-side) diff views with symbol indexes configured. Got accurate path and line information in the URI I was redirected to.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13047

Differential Revision: https://secure.phabricator.com/D18936
2018-01-26 11:59:48 -08:00
epriestley
0ec83132a8 Support basic export of user accounts
Summary:
Depends on D18934. Ref T13046. Add support for the new export flow to a second application.

My goal here is mostly just to make sure that this is general enough to work in more than one place, and exporting user accounts seems plausible as a useful feature, although we do see occasional requests for this feature exactly (like <https://discourse.phabricator-community.org/t/users-export-to-csv/968>).

The exported data may not truly be useful for much (no disabled/admin/verified/MFA flags, no external account data, no email addresses for policy reasons) but we can expand it as use cases arise.

Test Plan: Exported user accounts in several formats.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13046

Differential Revision: https://secure.phabricator.com/D18935
2018-01-26 11:17:44 -08:00
epriestley
a79bb55f3f Support CSV, JSON, and tab-separated text as export formats
Summary: Depends on D18919. Ref T13046. Adds some simple modular exporters.

Test Plan: Exported pull logs in each format.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13046

Differential Revision: https://secure.phabricator.com/D18934
2018-01-26 11:16:52 -08:00
epriestley
c0b8e4784b Add a basic, general-purpose export workflow for all objects with SearchEngine support
Summary:
Depends on D18918. Ref T13046. Ref T5954. Pull logs can currently be browsed in the web UI, but this isn't very powerful, especially if you have thousands of them.

Allow SearchEngine implementations to define exportable fields so that users can "Use Results > Export Data" on any query. In particular, they can use this workflow to download a file with pull logs.

In the future, this can replace the existing "Export to Excel" feature in Maniphest.

For now, we hard-code JSON as the only supported datatype and don't actually make any effort to format the data properly, but this leaves room to add more exporters (CSV, Excel) and data type awareness (integer casting, date formatting, etc) in the future.

For sufficiently large result sets, this will probably time out. At some point, I'll make this use the job queue (like bulk editing) when the export is "large" (affects more than 1K rows?).

Test Plan: Downloaded pull logs in JSON format.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13046, T5954

Differential Revision: https://secure.phabricator.com/D18919
2018-01-26 11:15:59 -08:00
epriestley
5058cfb972 Pass a real viewer to HeraldAdapter when doing test console runs
Summary:
Depends on D18932. Ref T13048. See PHI276. In the cluster, we don't have device keys on `web` nodes. This is generally good, since they don't need them, and it means that we aren't putting more credentials than we need on those hosts.

However, it means that when we pull diff content to test "Commit" rules via the Herald test console, we use the omnipotent user and try to use device credentials, and this fails since we don't have any.

Instead, pass the real viewer in this case so we just sign the request as them, like we do for normal Diffusion requests.

Test Plan:
Wrote and ran a commit content rule locally, no issues.

This isn't completely convincing since my local setup does have device credentials, but I'll double-check in production once this deploys.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13048

Differential Revision: https://secure.phabricator.com/D18933
2018-01-26 11:08:12 -08:00
epriestley
a9f87857af Mark the "Reviewer" field for Commits as deprecated
Summary:
Depends on D18931. Ref T13048. Ref T13041. This field means "the first accepting reviewer, where order is mostly arbitrary". Modern rules should almost certainly use "Accepting Reviewers" instead.

Getting rid of this completely is a pain, but we can at least reduce confusion by marking it as not-the-new-hotness. Add a "Deprecated" group, move it there, and mark it for exile.

Test Plan:
Edited a commit rule, saw it in "Deprecated" group at the bottom of the list:

{F5395001}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13048, T13041

Differential Revision: https://secure.phabricator.com/D18932
2018-01-26 11:07:02 -08:00
epriestley
a34b6bdd06 Implement an "only if the rule did not match last time" policy for Herald rules
Summary: Depends on D18927. Ref T13048. This implements a new policy which allows Herald rules to fire on some kinds of state changes.

Test Plan:
Wrote and tested rules with the new policy:

{F5394971}

{F5394972}

Also wrote and tested rules with the old policies:

{F5394973}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13048

Differential Revision: https://secure.phabricator.com/D18930
2018-01-26 11:06:13 -08:00
epriestley
204d1de683 Convert storage for Herald repetition policy to "text32"
Summary:
Depends on D18926. Ref T6203. Ref T13048. Herald rule repetition policies are stored as integers but treated as strings in most contexts.

After D18926, the integer stuff is almost totally hidden inside `HeraldRule` and getting rid of it completely isn't too tricky.

Do so now.

Test Plan:
  - Created "only the first time" and "every time" rules. Did a SELECT on their rows in the database.
  - Ran migrations, got a clean bill of health from `storage adjust`.
  - Did another SELECT on the rows, saw a faithful conversion to strings "every" and "first".
  - Edited and reviewed rules, swapping them between "every" and "first".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13048, T6203

Differential Revision: https://secure.phabricator.com/D18927
2018-01-26 11:05:37 -08:00
epriestley
1dd1237c92 Remove "HeraldRepetitionPolicyConfig" and hide storage details inside HeraldRule
Summary:
Depends on D18925. Ref T13048. Currently, HeraldRule stores policies as integers (0 or 1) in the database.

The application tries to mostly use strings ("first", "every"), but doesn't do a good job of hiding the fact that the values are integers deeper in the stack. So we end up with a lot of code like this:

```lang=php
$stored_int_value = $rule->getRepetitionPolicy();
$equivalent_string = HeraldRepetitionPolicyConfig::getString($stored_int_value);
$is_first = ($equivalent_string === HeraldRepetitionPolicyConfig::FIRST);
```

This happens in several places and is generally awful. Replace it with:

```lang=php
$is_first = $rule->isRepeatFirst();
```

To do this, merge `HeraldRepetitionPolicyConfig` into `HeraldRule` and hide all the mess inside the methods.

(This may let us just get rid of the integers in a later change, although I'm not sure I want to commit to that.)

Test Plan:
  - Grepped for `HeraldRepetitionPolicyConfig`, no more hits.
  - Grepped for `setRepetitionPolicy(...)` and `getRepetitionPolicy(...)`. There are no remaining callers outside of `HeraldRule`.
  - Browed and edited several rules. I'll vet this more convincingly after adding the new repetition rule.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13048

Differential Revision: https://secure.phabricator.com/D18926
2018-01-26 11:03:29 -08:00
epriestley
a90b16e83a Define available Herald rule repetition options in terms of "isSingleEventAdapter()"
Summary:
Depends on D18924. Ref T13048. Each adapter defines which repetition options ("every time", "only the first time") users may select for rules.

Currently, this is all explicit and hard-coded. However, every adapter really just implements this rule (except for some bugs, see below):

> You can pick "only the first time" if this adapter fires more than once on the same object.

Since we already have a `isSingleEventAdapter()` method which lets us tell if an adapter fires more than once, just write this rule in the base class and delete all the copy/pasting.

This also fixes two bugs because of the copy/pasting: Pholio Mocks and Phriction Documents did not allow you to write "only the first time" rules. There's no reason for this, they just didn't copy/paste enough methods when they were implemented.

This will make a future diff (which introduces an "if the rule did not match last time" policy) cleaner.

Test Plan:
  - Checked several different types of rules, saw appropriate options in the dropdown (pre-commit: no options; tasks: first or every).
  - Checked mocks and wiki docs, saw that you can now write "only the first time" rules.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13048

Differential Revision: https://secure.phabricator.com/D18925
2018-01-26 11:02:35 -08:00
epriestley
5529458e14 Add test coverage for SSH key revocation
Summary: Depends on D18928. Ref T13043. Add some automated test coverage for SSH revocation rules.

Test Plan: Ran tests, got a clean bill of health.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18929
2018-01-25 19:47:20 -08:00
epriestley
deb754dfe1 Make SSH key revocation actually prevent adding the same key back
Summary:
Ref T13043. In an earlier change I updated this langauge from "Deactivate" to "Revoke", but the behavior doesn't quite match.

This table has a unique key on `<isActive, keyBody>`, which enforces the rule that "a key can only be active for one unique user".

However, we set `isActive` to `null` when we revoke a key, and multiple rows are allowed to have the value `<null, "asdf">` (since a `null` column in a unique key basically means "don't enforce this unique key").

This is intentional, to support this workflow:

  - You add key X to bot A.
  - Whoops, wrong account.
  - You revoke key X from bot A.
  - You add key X to bot B.

This isn't necessarily a great workflow -- ideally, you'd throw key X away and go generate a new key after you realize you made a mistake -- but it's the sort of practical workflow that users are likely to expect and want to see work ("I don't want to generate a new key, it's already being used by 5 other services and cycling it is a ton of work and this is just a test install for my dog anyway."), and there's no technical reason we can't support it.

To prevent users from adding keys on the revocation list back to their account, just check explicitly.

(This is probably better in general anyway, because "cert-authority" support from PHI269 may mean that two keys are "equivalent" even if their text differs, and we may not be able to rely on a database test anyway.)

Test Plan:
  - Added the key `ssh-rsa asdf` to my account.
  - Revoked it.
  - Tried to add it again.
    - Before patch: worked.
    - After patch: error, "this key has been revoked".
  - Added it to a different account (the "I put it on the wrong bot" workflow).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18928
2018-01-25 19:43:37 -08:00
epriestley
1059fae6c9 Set an explicit default value for the bulk action control
Summary:
Ref T13025. See <https://discourse.phabricator-community.org/t/bulk-edit-no-actions-available/1011/1>.

I'm not sure if this is what the user is seeing, but in Chrome, the `<select />` does not automatically get set to the first valid value like it does in Safari.

Set it to the first valid value explicitly.

Test Plan: In Chrome, bulk editor previously hit a JS error when trying to read a bad action off the `<select />`. After patch, bulk edits go through cleanly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025

Differential Revision: https://secure.phabricator.com/D18923
2018-01-25 10:42:42 -08:00
epriestley
d28ddc21a5 Don't error when trying to mirror or observe an empty repository
Summary:
Fixes T5965.

Fixes two issues:

  - Observing an empty repository could write a warning to the log.
  - Mirroring an empty repository to a remote could fail.

For observing:

If newly-created with `git init --bare`, `git ls-remote` will
return the empty string.  Properly return an empty set of refs, rather
than attempting to parse the single "line" that is produced by
splitting that on newlines:

```
[2018-01-23 18:47:00] ERROR 8: Undefined offset: 1 at [/phab_path/phabricator/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:405]
arcanist(head=master, ref.master=5634f8410176), phabricator(head=master, ref.master=12551a1055ce), phutil(head=master, ref.master=4755785517cf)
  #0 PhabricatorRepositoryPullEngine::loadGitRemoteRefs(PhabricatorRepository) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:343]
  #1 PhabricatorRepositoryPullEngine::executeGitUpdate() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:126]
  #2 PhabricatorRepositoryPullEngine::pullRepositoryWithLock() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:40]
  #3 PhabricatorRepositoryPullEngine::pullRepository() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementUpdateWorkflow.php:59]
  ...
```

For mirroring:

`git` treats `git push --mirror` specially when a repository is empty. Detect this case by seeing if `git for-each-ref --count 1` does anything. If the repository is empty, just bail.

Test Plan:
  - Observed an empty and non-empty repository.
  - Mirrored an empty and non-empty repository.

Reviewers: alexmv, amckinley

Reviewed By: alexmv

Subscribers: Korvin, epriestley

Maniphest Tasks: T5965

Differential Revision: https://secure.phabricator.com/D18920
2018-01-24 15:50:30 -08:00
Mike Nicholson
c68a78360e Include extension name for GDSetupCheck.
Summary: Help for GD SetupChcek was missing the "how to install extension" content.

Test Plan: Uninstalled gd, validated extension installation instructions were present in Setup Issue.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D18922
2018-01-24 16:02:28 -05:00
epriestley
dabd3f0b41 Fix a race between Harbormaster and reviewers (often bots) to publish drafts for review
Summary:
See PHI309. There is a window of time between when all builds pass and when Harbormaster actually publishes a revision out of draft.

If any other user tries to interact with the revision during that window, they'll pick up the undraft transaction as a side effect. However, they won't have permission to apply it and will be stopped by a validation error.

Instead, only automatically publish a revision if the actor is the revision author or some system/application user (essentially always Harbormaster).

Test Plan:
  - Added a `echo ...; sleep(30);` to `HarbormasterBuildEngine->updateBuildable()` before the `applyTransactions()` at the bottom.
  - Wrote an "Always, run an HTTP request" Herald rule and Harbormaster build plan.
  - Ran daemons with `bin/phd debug task`.
  - Created a new revision with `arc diff`, as user A.
  - Waited for `phd` to enter the race window.
  - In a separate browser, as user B, submitted a comment via `differential.revision.edit`.
    - Before patch: edits during the race window were rejected with a validation error, "you don't have permission to request review".
    - After patch: edits go through cleanly.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18921
2018-01-24 11:16:06 -08:00
epriestley
167e7932ef Modernize "PhabricatorRepositoryPushLogSearchEngine"
Summary: Depends on D18917. Ref T13046. While I'm in here, update this to use more modern construction.

Test Plan: Browed and queried for push logs.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13046

Differential Revision: https://secure.phabricator.com/D18918
2018-01-23 14:14:44 -08:00
epriestley
778dfff277 Make minor correctness and display improvements to pull logs
Summary:
Depends on D18915. Ref T13046.

  - Distinguish between HTTP and HTTPS.
  - Use more constants and fewer magical strings.
  - For HTTP responses, give them better type information and more helpful UI behaviors.

Test Plan: Pulled over SSH and HTTP. Reviewed resulting logs from the web UI. Hit errors like missing/invalid credentials.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13046

Differential Revision: https://secure.phabricator.com/D18917
2018-01-23 14:13:18 -08:00
epriestley
a6fbde784c Modernize "PhabricatorRepositoryPushEventQuery"
Summary: Depends on D18914. Updates this Query to use slightly more modern construction while I'm working in adjacent code.

Test Plan: Viewed push logs in web UI.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18915
2018-01-23 14:12:14 -08:00
epriestley
e6a9db56a9 Add a basic view for repository pull logs
Summary:
Depends on D18912. Ref T13046. Add a UI to browse the existing pull log table.

The actual log still has some significant flaws, but get the basics working.

Test Plan: {F5391909}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13046

Differential Revision: https://secure.phabricator.com/D18914
2018-01-23 14:10:10 -08:00
epriestley
2914613444 Fix failure to record pullerPHID in repository pull logs
Summary:
See PHI305. Ref T13046.

The SSH workflows currently extend `PhabricatorManagementWorkflow` to benefit from sharing all the standard argument parsing code. Sharing the parsing code is good, but it also means they inherit a `getViewer()` method which returns the ommnipotent viewer.

This is appropriate for everything else which extends `ManagementWorkflow` (like `bin/storage`, `bin/auth`, etc.) but not appropriate for SSH workflows, which have a real user.

This caused a bug with the pull logs where `pullerPHID` was not recorded properly. We used `$this->getViewer()->getPHID()` but the correct code was `$this->getUser()->getPHID()`.

To harden this against future mistakes:

  - Don't extend `ManagementWorkflow`. Extend `PhutilArgumentWorkflow` instead. We **only** want the argument parsing code.
  - Rename `get/setUser()` to `get/setSSHUser()` to make them explicit.

Then, fix the pull log bug by calling `getSSHUser()` instead of `getViewer()`.

Test Plan:
  - Pulled and pushed to a repository over SSH.
  - Grepped all the SSH stuff for the altered symbols.
  -  Saw pulls record a valid `pullerPHID` in the pull log.
  - Used `echo {} | ssh ... conduit conduit.ping` to test conduit over SSH.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13046

Differential Revision: https://secure.phabricator.com/D18912
2018-01-23 14:09:42 -08:00
epriestley
bf2868c070 Rename "PhabricatorPasswordHashInterface" to "PhabricatorAuthPasswordHashInterface"
Summary: Depends on D18911. Ref T13043. Improves consistency with other "PhabricatorAuthPassword..." classes.

Test Plan: Unit tests, `grep`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18916
2018-01-23 14:06:05 -08:00
epriestley
3becd5a57c Add "bin/auth revoke --list" to explain what can be revoked
Summary:
Depends on D18908. Ref T13043. Allow users to get information about what revokers do with a new `--list` flag.

You can use `--list --type <key>` to get information about a specfic revoker.

Test Plan: Ran `bin/auth revoke --list`, saw a list of revokers with useful information.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18910
2018-01-23 14:01:39 -08:00
epriestley
21e415299f Mark all existing password hashes as "legacy" and start upgrading digest formats
Summary:
Depends on D18907. Ref T13043. Ref T12509. We have some weird old password digest behavior that isn't terribly concerning, but also isn't great.

Specifically, old passwords were digested in weird ways before being hashed. Notably, account passwords were digested with usernames, so your password stops working if your username is chagned. Not the end of the world, but silly.

Mark all existing hashes as "v1", and automatically upgrade then when they're used or changed. Some day, far in the future, we could stop supporting these legacy digests and delete the code and passwords and just issue upgrade advice ("Passwords which haven't been used in more than two years no longer work."). But at least get things on a path toward sane, modern behavior.

Test Plan: Ran migration. Spot-checked that everthing in the database got marked as "v1". Used an existing password to login successfully. Verified that it was upgraded to a `null` (modern) digest. Logged in with it again.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043, T12509

Differential Revision: https://secure.phabricator.com/D18908
2018-01-23 14:01:09 -08:00
epriestley
13ef5c6f23 When administrators revoke SSH keys, don't include a "security warning" in the mail
Summary:
Depends on D18906. Ref T13043. When SSH keys are edited, we normally include a warning that if you don't recognize the activity you might have problems in the mail body.

Currently, this warning is also shown for revocations with `bin/auth revoke --type ssh`. However, these revocations are safe (revocations are generally not dangerous anyway) and almost certainly legitimate and administrative, so don't warn users about them.

Test Plan:
  - Created and revoked a key.
  - Creation mail still had warning; revocation mail no longer did.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18907
2018-01-23 14:00:13 -08:00
epriestley
026ec11b9d Add a rate limit for guessing old passwords when changing passwords
Summary:
Depends on D18904. Ref T13043. If an attacker compromises a victim's session and bypasses their MFA, they can try to guess the user's current account password by making repeated requests to change it: if they guess the right "Old Password", they get a different error than if they don't.

I don't think this is really a very serious concern (the attacker already got a session and MFA, if configured, somehow; many installs don't use passwords anyway) but we get occasional reports about it from HackerOne. Technically, it's better policy to rate limit it, and this should reduce the reports we receive.

Test Plan: Tried to change password over and over again, eventually got rated limited. Used `bin/auth unlimit` to clear the limit, changed password normally without issues.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18906
2018-01-23 13:46:06 -08:00
epriestley
cab2bba6f2 Remove "passwordHash" and "passwordSalt" from User objects
Summary:
Ref T13043. After D18903, this data has migrated to shared infrastructure and has no remaining readers or writers.

Just delete it now, since the cost of a mistake here is very small (users need to "Forgot Password?" and pick a new password).

Test Plan: Grepped for `passwordHash`, `passwordSalt`, and variations.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18904
2018-01-23 13:44:26 -08:00
epriestley
abc030fa00 Move account passwords to shared infrastructure
Summary:
Ref T13043. This moves user account passwords to the new shared infrastructure.

There's a lot of code changes here, but essentially all of it is the same as the VCS password logic in D18898.

Test Plan:
- Ran migration.
- Spot checked table for general sanity.
- Logged in with an existing password.
- Hit all error conditions on "change password", "set password", "register new account" flows.
- Verified that changing password logs out other sessions.
- Verified that revoked passwords of a different type can't be selected.
- Changed passwords a bunch.
- Verified that salt regenerates properly after password change.
- Tried to login with the wrong password, which didn't work.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18903
2018-01-23 13:43:07 -08:00
epriestley
6b99aac49d Digest changeset anchors into purely alphanumeric strings
Summary:
Ref T13045. See that task for discussion.

This replaces `digestForIndex()` with a "clever" algorithm in `digestForAnchor()`. The new digest is the same as `digestForIndex()` except when the original output was "." or "_". In those cases, a replacement character is selected based on entropy accumulated by the digest function as it iterates through the string.

Test Plan: Added unit tests.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13045

Differential Revision: https://secure.phabricator.com/D18909
2018-01-23 13:42:08 -08:00
epriestley
b8a515cb29 Bring new password validation into AuthPasswordEngine
Summary:
Ref T13043. We have ~4 copies of this logic (registration, lost password recovery, set password, set VCS password).

Currently it varies a bit from case to case, but since it's all going to be basically identical once account passwords swap to the new infrastructure, bring it into the Engine so it can live in one place.

This also fixes VCS passwords not being affected by `account.minimum-password-length`.

Test Plan: Hit all errors in "VCS Password" panel. Successfully changed password.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18902
2018-01-23 10:58:37 -08:00
epriestley
aa3b582c7b Remove "set password" from bin/accountadmin and let bin/auth recover recover anyone
Summary:
Ref T13043. This cleans some things up to prepare for moving account passwords to shared infrastructure.

Currently, the (very old, fairly unusual) `bin/accountadmin` tool can set account passwords. This is a bit weird, generally not great, and makes upgrading to shared infrastructure more difficult. Just get rid of this to simplify things. Many installs don't have passwords and this is pointless and unhelpful in those cases.

Instead, let `bin/auth recover` recover any account, not just administrator accounts. This was a guardrail against administrative abuse, but it has always seemed especially flimsy (since anyone who can run the tool can easily comment out the checks) and I use this tool in cluster support with some frequency, occasionally just commenting out the checks. This is generally a better solution than actually setting a password on accounts anyway. Just get rid of the check and give users enough rope to shoot themselves in the foot with if they truly desire.

Test Plan:
  - Ran `bin/accountadmin`, didn't get prompted to swap passwords anymore.
  - Ran `bin/auth recover` to recover a non-admin account.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18901
2018-01-23 10:58:11 -08:00
epriestley
5a8a56f414 Prepare the new AuthPassword infrastructure for storing account passwords
Summary:
Ref T13043. In D18898 I moved VCS passwords to the new shared infrastructure.

Before account passwords can move, we need to make two changes:

  - For legacy reasons, VCS passwords and Account passwords have different "digest" algorithms. Both are more complicated than they should be, but we can't easily fix it without breaking existing passwords. Add a `PasswordHashInterface` so that objects which can have passwords hashes can implement custom digest logic for each password type.
  - Account passwords have a dedicated external salt (`PhabricatorUser->passwordSalt`). This is a generally reasonable thing to support (since not all hashers are self-salting) and we need to keep it around so existing passwords still work. Add salt support to `AuthPassword` and make it generate/regenerate when passwords are updated.

Then add a nice story about password digestion.

Test Plan: Ran migrations. Used an existing VCS password; changed VCS password. Tried to use a revoked password. Unit tests still pass. Grepped for callers to legacy `PhabricatorHash::digestPassword()`, found none.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18900
2018-01-23 10:57:40 -08:00
epriestley
753c4c5ff1 Remove the "PhabricatorRepositoryVCSPassword" class and table
Summary:
Ref T13043. After D18898, this has been migrated to new, more modern storage and no longer has any readers or writers.

One migration from long ago (early 2014) is affected. Since this is ancient and the cost of dropping this is small (see inline), I just dropped it.

I'll note this in the changelog.

Test Plan: Ran migrations, got a clean bill of health from `storage status`. Grepped for removed symbol.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18899
2018-01-23 10:56:37 -08:00
epriestley
dd8f588ac5 Migrate VCS passwords to new shared password infrastructure
Summary:
Ref T13043. Migrate VCS passwords away from their dedicated table to new the new shared infrastructure.

Future changes will migrate account passwords and remove the old table.

Test Plan:
- Ran migrations.
  - Cloned with the same password that was configured before the migrations (worked).
  - Cloned with a different, invalid password (failed).
- Changed password.
  - Cloned with old password (failed).
  - Cloned with new password (worked).
- Deleted password in web UI.
  - Cloned with old password (failed).
- Set password to the same password as it currently is set to (worked, no "unique" collision).
- Set password to account password. !!This (incorrectly) works for now until account passwords migrate, since the uniqueness check can't see them yet.!!
- Set password to a new unique password.
  - Cloned (worked).
  - Revoked the password with `bin/auth revoke`.
  - Verified web UI shows "no password set".
  - Verified that pull no longer works.
  - Verified that I can no longer select the revoked password.
- Verified that accounts do not interact:
  - Tried to set account B to account A's password (worked).
  - Tried to set account B to a password revoked on account A (worked).
- Spot checked the `password` and `passwordtransaction` tables for saniity.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18898
2018-01-23 10:56:13 -08:00
epriestley
bb12f4bab7 Add test coverage to the PasswordEngine upgrade workflow and fix a few bugs
Summary:
Ref T13043. When we verify a password and a better hasher is available, we automatically upgrade the stored hash to the stronger hasher.

Add test coverage for this workflow and fix a few bugs and issues, mostly related to shuffling the old hasher name into the transaction.

This doesn't touch anything user-visible yet.

Test Plan: Ran unit tests.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18897
2018-01-23 10:55:35 -08:00
epriestley
c280c85772 Consolidate password verification/revocation logic in a new PhabricatorAuthPasswordEngine
Summary:
Ref T13043. This provides a new piece of shared infrastructure that VCS passwords and account passwords can use to validate passwords that users enter.

This isn't reachable by anything yet.

The test coverage of the "upgrade" flow (where we rehash a password to use a stronger hasher) isn't great in this diff, I'll expand that in the next change and then start migrating things.

Test Plan: Added a bunch of unit tests.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18896
2018-01-23 10:54:49 -08:00
epriestley
42e2cd9af0 Add a "--force" flag to bin/auth revoke
Summary: Ref T13043. I'd like to replace the manual credential revocation in the Phacility export workflow with shared code in `bin/auth revoke`, but we need it to run non-interactively. Add a `--force` flag purely to make our lives easier.

Test Plan: Ran `bin/auth revoke --everywhere ...` with and without `--force`. Got prompted without, got total annihilation with.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18895
2018-01-22 20:12:36 -08:00
epriestley
9c00a43784 Add a more modern object for storing password hashes
Summary:
Ref T13043. Currently:

  - Passwords are stored separately in the "VCS Passwords" and "User" tables and don't share as much code as they could.
  - Because User objects are all over the place in the code, password hashes are all over the place too (i.e., often somewhere in process memory). This is a very low-severity, theoretical sort of issue, but it could make leaving a stray `var_dump()` in the code somewhere a lot more dangerous than it otherwise is. Even if we never do this, third-party developers might. So it "feels nice" to imagine separating this data into a different table that we rarely load.
  - Passwords can not be //revoked//. They can be //deleted//, but users can set the same password again. If you believe or suspect that a password may have been compromised, you might reasonably prefer to revoke it and force the user to select a //different// password.

This change prepares to remedy these issues by adding a new, more modern dedicated password storage table which supports storing multiple password types (account vs VCS), gives passwords real PHIDs and transactions, supports DestructionEngine, supports revocation, and supports `bin/auth revoke`.

It doesn't actually make anything use this new table yet. Future changes will migrate VCS passwords and account passwords to this table.

(This also gives third party applications a reasonable place to store password hashes in a consistent way if they have some need for it.)

Test Plan: Added some basic unit tests to cover general behavior. This is just skeleton code for now and will get more thorough testing when applications move.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18894
2018-01-22 15:35:28 -08:00
epriestley
fa1ecb7f66 Add a bin/auth revoke revoker for SSH keys
Summary: Ref T13043. Adds CLI support for revoking SSH keys. Also retargets UI language from "Deactivate" to "Revoke" to make it more clear that this is a one-way operation. This operation is already correctly implemented as a "Revoke" operation.

Test Plan: Used `bin/auth revoke --type ssh` to revoke keys, verified they became revoked (with proper transactions) in the UI. Revoked keys from the web UI flow.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18893
2018-01-22 15:35:07 -08:00
epriestley
39c3b10a2f Add a bin/auth revoke revoker for sessions
Summary: Ref T13043. Allows CLI revocation of login sessions.

Test Plan: Used `bin/auth revoke --type session` with `--from` and `--everywhere` to revoke sessions. Saw accounts get logged out in web UI.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18892
2018-01-22 12:01:14 -08:00
epriestley
7970cf0585 Add a bin/auth revoke revoker for temporary tokens
Summary: Ref T13043. Allows CLI revocation of temporary ("forgot password", "one-time login") tokens.

Test Plan: Used "Forgot Password?" to generate tokens, used `bin/auth revoke --type temporary` with `--from` and `--everywhere` to revoke them.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18891
2018-01-22 12:00:33 -08:00
epriestley
a9d7b4f0ff Support bulk edit of "points" for Maniphest tasks
Summary: Ref T13025. Fixes T10973. Fairly straightforward. The "points" type is just an alias for "text" today.

Test Plan: Bulk edited points.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025, T10973

Differential Revision: https://secure.phabricator.com/D18889
2018-01-22 11:59:52 -08:00
epriestley
d9b6513a21 Respect tokenizer limits in the bulk editor
Summary: Ref T13025. This makes limits (for fields like "Assign To") work in the bulk editor, so you can't type "Assign to: x, y, z" anymore.

Test Plan: Hit limit for "Assign to" and a custom project field. No limit for "Add subscribers".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025

Differential Revision: https://secure.phabricator.com/D18888
2018-01-22 11:55:55 -08:00
epriestley
fbfcc37531 Respect token limits for "Assign to" and custom datasource fields in Herald
Summary:
See PHI173. Currently, Herald has an "Assign to" action for tasks, and you can specify custom fields with datasource values (like users or projects) that have a limit (like 1 "Owner", or 12 "Jury Members").

Herald doesn't support these limits right now, so you can write `[ Assign to ][ X, Y, Z ]`. This just means "Assign to X", but make it more clear by actually enforcing the limit in the UI.

Test Plan:
  - Created a "projects" custom field with limit 1.
  - Tried to create actions that 'assign to' or 'set custom field to' more than one thing, got helpfully rebuffed by the UI.
  - Created an "add subscribers" action with more than one value.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18887
2018-01-22 11:54:12 -08:00
epriestley
6a62797056 Fix some issues with Diffusion file data limits
Summary:
See <https://discourse.phabricator-community.org/t/files-created-from-repository-contents-slightly-over-one-chunk-in-size-are-truncated-to-exactly-one-chunk-in-size/988/1>. Three issues here:

  - When we finish reading `git cat-file ...` or whatever, we can end up with more than one chunk worth of bytes left in the internal buffer if the read is fast. Use `while` instead of `if` to make sure we write the whole buffer.
  - Limiting output with `setStdoutSizeLimit()` isn't really a reliable way to limit the size if we're also reading from the buffer. It's also pretty indirect and confusing. Instead, just let the `FileUploadSource` explicitly implement a byte limit in a straightforward way.
  - We weren't setting the time limit correctly on the main path.

Overall, this could cause >4MB files to "write" as 4MB files, with the rest of the file left in the UploadSource buffer. Since these files were technically under the limit, they could return as valid. This was intermittent.

Test Plan:
  - Pushed a ~4.2MB file.
  - Reloaded Diffusion a bunch, sometimes saw the `while/if` buffer race and produce a 4MB file with a prompt to download it. (Other times, the buffer worked right and the page just says "this file is too big, sorry").
  - Applied patches.
  - Reloaded Diffusion a bunch, no longer saw bad behavior or truncated files.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18885
2018-01-22 11:52:37 -08:00
epriestley
c637680445 Show related objects on daemon worker task queue detail pages
Summary:
See PHI287. Currently, you can't get very much information about a task in the worker queue from the web UI.

This is largely intentional (e.g., it's bad if we let you inspect the content of a "send an administrator's password reset email" task), and a lot of the data is task-class specific so it would be a lot of work to expose it, but we can add one useful piece of information pretty easily: for tasks with an `objectPHID` that points at a valid object that's visible to the user, tell the user which object the task is associated with.

Test Plan: {F5386562}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18890
2018-01-20 06:52:15 -08:00
epriestley
3b7547e726 Fix an issue where certain configurations could fail to publish revision feed stories
Summary: See PHI292. This is just a generalization of D18851: feed stories have the same issue as mail. Don't hide "requested a review" in either mail or feed.

Test Plan:
  - Enable prototypes.
  - No harbormaster builds.
  - Create a revision.
    - Pre-patch: no feed story.
    - Post-patch: feed story.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18886
2018-01-19 15:39:31 -08:00
epriestley
86a0c7daa2 Fix a fatal in blame when the viewer can't see a revision because of a permission issue
Summary:
Fixes T13040. To reproduce:

  - View a file with blame enabled, where some line has an associated revision (say, `D123`).
  - Edit `D123` so it exists and is a valid revision, but the viewer can't see it.
  - Reload the page.

Instead, only add revisions to the map if we actually managed to load them.

Test Plan: Page no longer fatals.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13040

Differential Revision: https://secure.phabricator.com/D18884
2018-01-19 14:19:06 -08:00
epriestley
3038d564a6 Allow bulk edits to be made silently if you have CLI access
Summary:
Fixes T13042. This hooks up the new "silent" mode from D18882 and makes it actually work.

The UI (where we tell you to go run some command and then reload the page) is pretty clumsy, but should solve some problems for now and can be cleaned up eventually. The actual mechanics (timeline aggregation, Herald interaction,  etc.) are on firmer ground.

Test Plan:
  - Made a normal bulk edit, got mail and feed stories.
  - Made a silent bulk edit, no mail and no feed.
  - Saw "Silent Edit" marker in timeline for silent edits:

{F5386245}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13042

Differential Revision: https://secure.phabricator.com/D18883
2018-01-19 13:24:54 -08:00
epriestley
8b12fa6d6e Prepare TransactionEditor for silent transactions via bulk edit
Summary:
Ref T13042. This adds a "silent" edit mechanism which suppresses feed stories, email, and notifications.

The other behaviors here are:

  - The transactions are marked as "silent" so we can render a hint in the UI in the future to make it clear to users that they aren't missing email.
  - If the editor uses Herald, mail rules are suppressed so they don't fire incorrectly (this mostly affects "the first time this rule matches, send me an email" rules: without this, they'd match "the first time" on the bulk edit, not send email, then never match again since they already matched).
  - If the edit queues additional edits, those are applied silently too.

This doesn't (or, at least, shouldn't) actually change any behavior since you can't apply silent edits yet.

Test Plan:
Somewhat theoretical, since this isn't reachable yet. Should get meaningful testing in an upcoming change.

Did a bit of var_dump() / debug poking to attempt to verify that nothing too crazy is happening.

Viewed and edited objects, no changes in behavior.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13042

Differential Revision: https://secure.phabricator.com/D18882
2018-01-19 13:23:38 -08:00
epriestley
8a5def8e0a Remove legacy transaction editor "getDisableEmail()" method
Summary:
Ref T13042. This is a very, very old policy-violating option from yesteryear which supported build systems publishing updates by adding comments to revisions, without sending email about it.

Harbormaster has served this role for a long time and this is policy-violating in the general case (it allows attackers to act in secret).

Test Plan: Grepped for affected symbols.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13042

Differential Revision: https://secure.phabricator.com/D18881
2018-01-19 13:23:06 -08:00
epriestley
7a43181337 Organize bulk edit actions into nice groups
Summary: Ref T13025. We're getting kind of a lot of actions, so put them in nice groups so they're easier to work with.

Test Plan: {F5386038}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025

Differential Revision: https://secure.phabricator.com/D18880
2018-01-19 13:22:25 -08:00
epriestley
f8113aecdc Make bulk edit <select /> fields a little more natrual and set options for subtype transactions
Summary:
Ref T13025. This is some minor technical stuff: make the "select" bulk edit type a little more consistent with other types by passing data down instead of having it reach up the stack. This simplifies the implementation of a custom field "select" in a future change.

Also, provide an option list to the "select" edit field for object subtypes. This is only accessible via Conduit so it currently never actually renders anything in the UI, but with the bulk edit stuff we get some initialization order issues if we don't set anything. This will also make any future changes which expose subtypes more broadly more straightforward.

Test Plan:
  - Bulk edited "select" fields, like "Status" and "Priority".
  - No more fatal when trying to `getOptions()` internally on the subtype field.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025

Differential Revision: https://secure.phabricator.com/D18878
2018-01-19 13:17:28 -08:00
epriestley
a26cf20dd1 Fix a bug with setting custom PHID list field values via Conduit and prepare for bulk edits
Summary:
Ref T13025. Custom field transactions work somewhat unusually: the values sometimes need to be encoded. We currently do not apply this encoding correctly via Conduit.

For example, setting some custom PHID field to `["PHID-X-Y"]` fails with a bunch of JSON errors.

Add an extra hook callback so that EditTypes can apply processing to transaction values, then apply the correct CustomField processing.

This only affects Conduit. In a future diff, this also allows bulk edit of custom fields to work correctly.

Test Plan: Added a custom field to Maniphest with a list of projects. Used Conduit to bulk edit it (which now works, but did not before). Used the web UI to bulk edit it.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025

Differential Revision: https://secure.phabricator.com/D18876
2018-01-19 12:51:35 -08:00
epriestley
1e51f1d0dc Reuse more transaction construction code in bulk editor
Summary:
Ref T13025. Currently, the bulk editor takes an HTTP request and emits a list of "raw" transactions (simple dictionaries). This goes into the job queue, and the background job builds a real transaction.

However, the logic to turn an HTTP request into a raw transaction is ending up with some duplication, since we generally already have logic to turn an HTTP request into a full object.

Instead: build real objects first, then serialize them to dictionaries. Send those to the job queue, rebuild them into objects again, and we end up in the same spot with a little less code duplication.

Finally, delete the mostly-copied code.

Test Plan: Used bulk editor to add comments, projects, and rename tasks.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025

Differential Revision: https://secure.phabricator.com/D18875
2018-01-19 12:49:17 -08:00
epriestley
91a78db99b Support "Assign To" in Maniphest bulk editor
Summary:
Ref T13025. See PHI173. This supports the "Assign to" field in the new editor.

This is very slightly funky: to unassign tasks, you need to leave the field blank. I have half a diff to fix this, but the way the `none()` token works in the default datasource is odd so it needs a separate datasource. I'm punting on this for now since it works, at least, and isn't //completely// unreasonable.

This also simplifies some EditEngine stuff a little. Notably:

  - I reorganized EditType construction slightly so subclasses can copy/paste a little bit less.
  - EditType had `field` and `editField` properties which had the same values. I canonicalized on `editField` and made this value set a little more automatically.

Test Plan: Used bulk editor to reassign some tasks. By leaving the field blank, unassigned tasks.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025

Differential Revision: https://secure.phabricator.com/D18874
2018-01-19 12:48:41 -08:00
epriestley
0cad6021b6 Restore "Tags" and "Subscribers" edit capabilities to Maniphest bulk editor
Summary: Depends on D18867. Ref T13025. Fixes T8740. Rebuilds the tag/subscriber actions (add, remove, set) into the bulk editor.

Test Plan: Added, removed and set these values via bulk edit.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025, T8740

Differential Revision: https://secure.phabricator.com/D18868
2018-01-19 12:47:10 -08:00
epriestley
687fada5af Restore bulk edit support for remarkup fields (description, add comment)
Summary:
Depends on D18866. Ref T13025. Fixes T12415. This makes the old "Add Comment" action work, and adds support for a new "Set description to" action (possibly, I could imagine "append description" being useful some day, maybe).

The implementation is just a `<textarea />`, not a whole fancy remarkup box with `[Bold] [Italic] ...` buttons, preview, typeaheads, etc. It would be nice to enrich this eventually but doing the rendering in pure JS is currently very involved.

This requires a little bit of gymnastics to get the transaction populated properly, and adds some extra validation since we need some code there anyway.

Test Plan:
  - Changed the description of a task via bulk editor.
  - Added a comment to a task via bulk editor.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025, T12415

Differential Revision: https://secure.phabricator.com/D18867
2018-01-19 12:45:34 -08:00
epriestley
bf1ac701c3 Support "select" types in bulk editor (status, priority)
Summary: Depends on D18864. Ref T13025. Adds bulk edit support back for "status" and "priority" using `<select />` controls.

Test Plan:
Used bulk editor to change status and priority for tasks.

{F5374436}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025

Differential Revision: https://secure.phabricator.com/D18866
2018-01-19 12:44:48 -08:00
epriestley
a251db4618 Remove the Maniphest-specific bulk job type
Summary: Depends on D18863. Ref PHI173. Ref T13025. After D18863, this job type is no longer used: the workflow uses a genric worker instead which can apply transactions to any object.

Test Plan: Grepped for callsites, found none.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025

Differential Revision: https://secure.phabricator.com/D18864
2018-01-19 12:44:16 -08:00
epriestley
09e71a4082 Define bulk edits in terms of EditEngine, not hard-coded ad-hoc definitions
Summary:
Depends on D18862. See PHI173. Ref T13025. Fixes T10005. This redefines bulk edits in terms of EditEngine fields, rather than hard-coding the whole thing.

Only text fields -- and, specifically, only the "Title" field -- are supported after this change. Followup changes will add more bulk edit parameter types and broader field support.

However, the title field now works without any Maniphest-specific code, outside of the small amount of binding code in the `ManiphestBulkEditor` subclass.

Test Plan: Used the bulk edit workflow to change the titles of tasks.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025, T10005

Differential Revision: https://secure.phabricator.com/D18863
2018-01-19 12:43:47 -08:00
epriestley
6ef45d8245 Provide a generic transaction-oriented bulk job worker
Summary:
Depends on D18806. Ref T13025. See PHI173. Currently, Maniphest bulk edits are processed by a Maniphest-specific worker. I want to replace this with a generic worker which can apply transactional edits to any object.

This implements a generic worker, although it has no callers yet. Future changes give it callers, and later remove the Maniphest-specific worker.

Test Plan: See next changes.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025

Differential Revision: https://secure.phabricator.com/D18862
2018-01-19 12:41:56 -08:00
epriestley
7f91c8c4ac Rebuild the bulk editor on SearchEngine
Summary:
Depends on D18805. Ref T13025. Fixes T10268.

Instead of using a list of IDs for the bulk editor, power it with SearchEngine queries. This gives us the full power of SearchEngine and lets us use a query key instead of a list of 20,000 IDs to avoid issues with URL lengths.

Also, split it into a base `BulkEngine` and per-application subclasses. This moves us toward T10005 and universal support for bulk operations.

Also:

  - Renames most of "batch" to "bulk": we're curently inconsitent about this, I like "bulk" better since I think it's more clear if you don't regularly interact with `.bat` files, and newer stuff mostly uses "bulk".
  - When objects in the result set can't be edited because you don't have permission, show the status more clearly.

This probably breaks some stuff a bit since I refactored so heavily, but it seems mostly OK from poking around. I'll clean up anything I missed in followups to deal with remaining items on T13025.

Test Plan:
{F5302300}

  - Bulk edited from Maniphest.
  - Bulk edited from a workboard (no more giant `?ids=....` in the URL).
  - Hit most of the error conditions, I think?
  - Clicked the "Cancel" button.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025, T10268

Differential Revision: https://secure.phabricator.com/D18806
2018-01-19 12:40:08 -08:00
epriestley
ad659627b3 Make bulk editor working set editable and more homogenous
Summary:
Ref T13025. See PHI50. Fixes T11286. Ref T10005. Begin modernizing the bulk editor.

For T10005 ("move the bulk editor to modern infrastructure"), rewrite the rendering of the editable set so that it is application-agnostic and can work with any kind of object.

For T11286 ("let users de-select items in the working set"), make the working set editable.

Test Plan:
{F5302158}

  - Deselected some objects, applied an edit, saw the edit apply to only selected objects.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13025, T11286, T10005

Differential Revision: https://secure.phabricator.com/D18805
2018-01-19 12:39:27 -08:00
epriestley
3e983b583d Fix a transposed feed story in Badges
Summary: See <https://discourse.phabricator-community.org/t/badge-quality-from-and-to-interchanged-in-activity-log/987>. These are swapped.

Test Plan: Changed a badge quality, viewed feed story, saw it position the strings correctly.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18870
2018-01-16 13:57:01 -08:00
epriestley
82bfb98179 Fix a copy/paste error on the burnup chart
Summary: See PHI286, maybe. See PHI273. Strongly considering just deleting this.

Test Plan: ~.~

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18865
2018-01-11 07:07:46 -08:00
Alex Vandiver
2140741e25 Fix typo in new setting description
Summary:
Noticed by @amckinley in
https://secure.phabricator.com/D18850#inline-57246 but not fixed
before landing.

Test Plan: ispell

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, amckinley, epriestley

Differential Revision: https://secure.phabricator.com/D18861
2018-01-06 07:25:27 -08:00
epriestley
adbd7d4fd8 Make the new synthetic burnup chart data respect the "Project" filter
Summary: See PHI273. Third time's the charm? This page has a "Project" filter which lets you view data for only one project, but the synthetic data currently ignores it.

Test Plan: Filtered burnup chart by various projects, saw sensible-looking data.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18860
2018-01-05 10:34:45 -08:00
epriestley
5543592034 Add a couple of clarifying comments to the Mercurial protocol parser
Summary: See D18857. Ref T13036. See PHI275. Explain what's going on here a little better since it isn't entirely obvious and debugging these stream parsers is a gigantic pain.

Test Plan: Read text.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13036

Differential Revision: https://secure.phabricator.com/D18859
2018-01-04 14:23:28 -08:00
epriestley
94db95a165 Sort burnup data chronologically after merging synthetic and "real" data
Summary:
Ref T13020. See PHI273. See D18853. On `secure`, the chart looks less promising than it did locally, and is full of discontinuities:

{F5356544}

I think this is a sorting issue. But if I can't fake my way through this soon I'll maybe get the Fact engine running and use it to provide the data here, as a sort of half-step toward T1562?

Test Plan: Chart looks the same locally, will push and see if `secure` improves?

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13020

Differential Revision: https://secure.phabricator.com/D18854
2018-01-04 14:08:03 -08:00
epriestley
13c8963dab Fix a Mercurial wire protocol parser issue when we receive a length frame before any data
Summary:
Depends on D18856. Ref T13036. See PHI275. When we receive a length frame but the buffer doesn't have any data yet, we currently emit a pointless 0-length data frame on the channel.

For normal chatter this is harmless/valid, but it causes problems when a channel has transitioned into bundle2 mode (probably it indicates "end of stream")?

In any case, it's never helpful, so if we're about to read a data block and don't have any data, just bail out until we see some more data.

Note that we can't end up here //expecting// a 0-length data block: both the `data-length` and `data-bytes` states already handle that properly.

Test Plan: Pushed 4MB changes to a Mercurial repository with Mercurial 4.1.1, was no longer able to hit channel errors.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13036

Differential Revision: https://secure.phabricator.com/D18857
2018-01-04 14:06:52 -08:00
epriestley
3a4e14431f Remove an obsolete comment about Mercurial SSH error behavior
Summary:
Depends on D18855. Ref T13036. This comment no longer seems to be accurate: anything we send over `stderr` is faithfully shown to the user with recent clients.

From [[ https://www.mercurial-scm.org/repo/hg/file/default/mercurial/help/internals/wireprotocol.txt | this document ]], the missing sauce may have been:

```
A generic error response type is also supported. It consists of a an error
message written to ``stderr`` followed by ``\n-\n``. In addition, ``\n`` is
written to ``stdout``.
```

That is, writing "\n" to stdout in addition to writing the error to stderr. However, this no longer appears to be necessary.

I think the modern client behavior is generally sensible (and consistent with the behavior of Git and Subversion) so this //probably// isn't a bug or me making a mistake.

Test Plan: With a modern client, threw some arbitrary exception during execution. Observed a helpful message on the client with no additional steps.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13036

Differential Revision: https://secure.phabricator.com/D18856
2018-01-04 14:05:44 -08:00
epriestley
0f02d79ffa Remove nonfunctional Mercurial "bundle2" capability filtering from SSH pathway
Summary:
Ref T13036. This code attempts to filter the "capabilities" message to remove "bundle2", but I think this has never worked.

Specifically, the //write// pathway is hooked, and "write" here means "client is writing a message to the server". However, the "capabilities" frame is part of the response, not part of the request. Thus, this code never fires, at least on recent versions of Mercurial.

Since I plan to support bundle2 and don't want to decode response frames, just get rid of this, assuming we'll achieve those goals.

I think this was just overlooked in D14241, which probably focused on the HTTP version. This code does (at least, potentially) do something for HTTP.

I'm leaving the actual "strip stuff" code in place for now since I think it's still used on the HTTP pathway.

Test Plan:
  - Added debug logging, saw this code never hit even though `hg push --debug` shows the client believing bundle2 is supported.
  - Logged both halves of the wire protocol and saw this come from the server, not the client.
  - Ran the failing `hg push` of a 4MB file under hg 4.4.1, got the same error as before.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: cspeckmim

Maniphest Tasks: T13036

Differential Revision: https://secure.phabricator.com/D18855
2018-01-04 14:05:13 -08:00
epriestley
f3f1f9dc57 Allow "drydock.blueprint.edit" to create blueprints
Summary:
Depends on D18848. Ref PHI243. This puts a bit of logic up front to figure out the blueprint type before we actually start editing it.

This implementation is a little messy but it keeps the API clean. Eventually, the implementation could probably go in the TransactionTypes so more code is shared, but I'd like to wait for a couple more of these first.

This capability probably isn't too useful, but just pays down a bit of technical debt from the caveat introduced in D18822.

Test Plan:
  - Created a new blueprint with the API.
  - Tried to create a blueprint without a "type" (got a helpful error).
  - Created and edited blueprints via the web UI.
  - Tried to change the "type" of an existing blueprint (got a helpful error).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D18849
2018-01-04 10:08:07 -08:00
epriestley
6d9776fa89 Give EditEngine a Conduit-specific initialization pathway for objects
Summary:
Depends on D18845. See PHI243 for context and more details.

Briefly, some objects need a "type" transaction (or something similar) very early on, and we can't generate their fields until we know the object type. Drydock blueprints are an example: a blueprint's fields depend on the blueprint's type.

In web interfaces, the workflow just forces the user to select a type first. For Conduit workflows, I think the cleanest approach is to proactively extract and apply type information before processing the request. This will make the implementation a little messier, but the API cleaner.

An alternative is to add more fields to the API, like a "type" field. This makes the implementation cleaner, but the API messier. I think we're better off favoring a cleaner API here.

This change just makes it possible for `DrydockBlueprintEditEngine` to look at the incoming transactions and extract a "type"; it doesn't actually change any behavior.

Test Plan: Performed edits via API, but this change doesn't alter any behavior.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18847
2018-01-04 10:07:07 -08:00
epriestley
83c528c464 Modularize transactions for Drydock Blueprints
Summary: Ref PHI243. This is a followup to D18822, which added an edit-only `drydock.blueprint.edit`. By modularizing transactions (here) and then adding a "type" transaction (next change) I intend to remove the "edit-only" limitation and make this API method fully functional.

Test Plan: Created and edited blueprints via the web UI. Edited blueprints via the API. Disabled/enabled blueprints (currently web UI only).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D18845
2018-01-04 10:03:44 -08:00
epriestley
53b25db918 Prevent enormous changes from being pushed to repositoires by default
Summary:
Fixes T13031. "Enormous" changes are basically changes which are too large to hold in memory, although the actual definition we use today is "more than 1GB of change text or `git diff` runs for more than 15 minutes".

If an install configures a Herald content rule like "when content matches /XYZ/, do something" and then a user pushes a 30 GB source file, we can't put it into memory to `preg_match()` it. Currently, the way to handle this case is to write a separate Herald rule that rejects enormous changes. However, this isn't obvious and means the default behavior is unsafe.

Make the default behavior safe by rejecting these changes with a message, similar to how we reject "dangerous" changes (which permanently delete or overwrite history) by default.

Also, change a couple of UI strings from "Enormous" to "Very Large" to reduce ambiguity. See <https://discourse.phabricator-community.org/t/herald-enormous-check/822>.

Test Plan: Changed the definition of "enormous" from 1GB to 1 byte. Pushed a change; got rejected. Allowed enormous changes, pushed, got rejected by a Herald rule. Disabled the Herald rule, pushed, got a clean push. Prevented enormous changes again. Grepped for "enormous" elsewhere in the UI.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: joshuaspence

Maniphest Tasks: T13031

Differential Revision: https://secure.phabricator.com/D18850
2018-01-04 10:02:29 -08:00
epriestley
cb957f8d62 Pile more atrocities onto the Maniphest burnup report
Summary:
See PHI273. Ref T13020. After D18777, tasks created directly into the default status (which is common) via the web UI no longer write a "status" transaction.

This is consistent with other applications, and consistent with the API/email behavior for tasks since early 2016. It also improves the consistency of //reading// tasks via the API.

However, it impacted the "Burnup Report" which relies on directly reading these rows to detect task creation. Until this is fixed properly (T1562), synthetically generate the "missing" transactions which this page expects by looking at task creation dates instead.

Specifically, we:

  - Generate a fake `status: null -> "open"` transaction for every task by looking at the Task table.
  - Go through the transaction list and remove all the legacy `status: null -> "any open status"` transactions. These will only exist for older tasks.
  - Merge all our new fake transactions into the list of transactions.
  - Continue on as though nothing happened, letting the rendering code continue to operate on legacy-looking data.

I think this will slightly miscount tasks which were created directly into a closed status, but this is very rare, and does not significantly impact the accuracy of this report relative to other known issues (notably, merging closed tasks).

This will also get the wrong result if the default status has changed from an "open" status to a "closed" status at any point, but this is exceptionally bizarre/rare.

Ultimately, T1562 will let us delete all this stuff and disavow its existence.

Test Plan:
  - Created some tasks, loaded burnup before/after this patch.
  - My local chart looks more accurate afterwards, but the data is super weird (I used `bin/lipsum` to create a huge number of tasks a couple months ago). I'll vet this on `secure`, which has more reasonable data.

Here's my local chart:

{F5356499}

That's what it //should// look like, it's just hard to be confident that nothing else is hiding there.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13020

Differential Revision: https://secure.phabricator.com/D18853
2018-01-04 10:02:15 -08:00
epriestley
129e3a1208 Fix a minor/harmless race with feed publishers in certain draft states
Summary:
Depends on D18851. Ref T13035. After D18819, revision creation transactions may be split into two groups (if prototypes are enabled).

This split means we have two workers. The first worker doesn't publish feed stories or mail; the second one does.

Currently, both workers call `shouldPublishFeedStory()` before they queue, and then again after the daemons pull them out of the queue. However, the answer to this question can change.

Specifically, this happens:

  - `arc` creates a revision.
  - The first transaction group applies, creating the revision as a draft, and returns `false` from `shouldPublishFeedStory()`, and does not generate related PHIDs. It queues a daemon to send mail, expecting it not to publish a feed story.
  - The second transaction group applies, promoting the revision to "needs review". Since the revision has promoted, `shouldPublishFeedStory()` now returns true. This editor generates related PHIDs and queues a daemon task, expecting it to send mail / publish feed.
  - A few milliseconds pass.
  - The first job gets pulled out of the daemon queue and worked on. It does not have any feed metadata because the object wasn't publishable when the job was queued -- but `shouldPublishFeedStory()` now returns true, so it tries to publish a story without any metadata available. Slightly bad stuff happens (see below).
  - The second job gets pulled out of the daemon queue and worked on. This one has metadata and works fine.

The "slightly bad stuff" is that we publish an empty feed story with no references to any objects, then try to push it to hooks and other listeners. Since it doesn't have any references, it fails to load during the "push to external listeners" phase.

This is harmless but clutters the log and doesn't help anything.

Instead, cache the state of "are we publishing a feed story for this object?" when we queue the worker so it can't race.

Test Plan:
  - Enabled prototypes.
  - Disabled all Herald triggers for Harbormaster build plans.
  - Ran `bin/phd debug task` in one window.
  - Created a revision in a separate window.
  - Before patch: saw "unable to load feed story" errors in the daemon log.
  - After patch: no more "unable to load feed story" errors.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13035

Differential Revision: https://secure.phabricator.com/D18852
2018-01-04 08:14:55 -08:00
epriestley
de6c68b91e Always show "X requested review" in mail to stop some undraft mail from being dropped
Summary: Ref T13035. See that task for a description of the issue.

Test Plan:
  - Enabled prototypes.
  - Disabled all Herald rules that trigger Harbormaster builds.
  - Created a new revision.
  - Before patch: initial review request email was dropped.
  - After patch: initial review request email is sent.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13035

Differential Revision: https://secure.phabricator.com/D18851
2018-01-04 08:14:31 -08:00
epriestley
ead5f4fd9c Add an "Accepting reviewers" Herald field for commits
Summary:
See PHI262. Fixes T12578. Although this is a bit niche and probably better accomplished through advisory/soft measures ("Add blocking reviewers") in most cases, it isn't difficult to implement and doesn't create any technical or product tension.

If installs write a rule that blocks commits, that will probably also naturally lead them to an "add reviewers" rule anyway.

Also, allow packages to be hit with the typeahead. They're valid reviewers but previously you couldn't write rules against them, for no actual reason.

Test Plan: Used test console to run this against commits, got sensible results for the field value.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12578

Differential Revision: https://secure.phabricator.com/D18839
2017-12-26 15:59:36 -08:00
epriestley
ad4db9b2f3 Separate "Set/Reset Password" from "Change Password"
Summary:
See PHI223. Ref T13024. There's a remaining registration/login order issue after the other changes in T13024: we lose track of the current URI when we go through the MFA flow, so we can lose "Set Password" at the end of the flow.

Specifically, the flow goes like this today:

  - User clicks the welcome link in email.
  - They get redirected to the "set password" settings panel.
  - This gets pre-empted by Legalpad (although we'll potentially survive this with the URI intact).
  - This also gets pre-empted by the "Set MFA" workflow. If the user completes this flow, they get redirected to a `/auth/multifactor/?id=123` sort of URI to highlight the factor they added. This causes us to lose the `/settings/panel/password/blah/blah?key=xyz` URI.

The ordering on this is also not ideal; it's preferable to start with a password, then do the other steps, so the user can return to the flow more easily if they are interrupted.

Resolve this by separating the "change your password" and "set/reset your password" flows onto two different pages. This copy/pastes a bit of code, but both flows end up simpler so it feels reasonable to me overall.

We don't require a full session for "set/reset password" (so you can do it if you don't have MFA/legalpad yet) and do it first.

This works better and is broadly simpler for users.

Test Plan:
  - Required MFA + legalpad, invited a user via email, registered.
    - Before: password set flow got lost when setting MFA.
    - After: prompted to set password, then sign documents, then set up MFA.
  - Reset password (with MFA confgiured, was required to MFA first).
  - Tried to reset password without a valid reset key, wasn't successful.
  - Changed password using existing flow.
  - Hit various (all?) error cases (short password, common password, mismatch, missing password, etc).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13024

Differential Revision: https://secure.phabricator.com/D18840
2017-12-26 08:34:14 -08:00
epriestley
66b74073be Provide ANSI color information for Harbormaster build status via API
Summary:
Ref PHI261. This moves Harbormaster build status to work more similarly to other modern status types, like Differential revision status, where we try to specify as much behavior on the server as possible so that the client and server can vary independently.

(I don't have any specific plans to make Harbormaster build status configurable on the server side, but it isn't out of the question.)

Test Plan: Ran `harbormaster.querybuilds` (saw same data as before) and `harbormaster.build.search` (saw same data as before, plus new ANSI color data).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D18838
2017-12-23 11:39:05 -08:00
Mukunda Modell
bd5aa0c90f Prevent hiding the PhabricatorProjectDetailsProfileMenuItem
Summary: This probably isn't the best solution, however, it conveniently avoids the bug from T13033. It would probably be more user-friendly (but more difficult to implement) if we allowed either Project Details //or// Workboard to be hidden but not both.

Test Plan: Tested locally, indeed this prevents hiding the menu item.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18836
2017-12-23 11:38:05 -08:00
epriestley
84cf493879 Allow "Manage" to be the default menu item for projects
Summary:
Fixes T13033. Currently (prior to D18836) all the default items for projects can be hidden. When this occurs, the main project page fatals.

If we fix the fatal narrowly (don't try to call `null->getBuiltinKey()` when `$default` is `null`), it would 404, which is a little better but not by much.

After D18836 you can't hide "Profile", which is pretty sensible, and effectively fixes this. This change doubles down: let "Manage" be a default, and send the user there if we can't find a different default.

Ideally, the MenuEngine itself will do more rendering eventually (as it does for some of the newer Home stuff) and could handle this defaulting behavior with less special casing, but we'd still end up in a similar situation if a project had only one "Link" item to "coolcats.com" or something: redirecting the user to "coolcats.com" is probably better than fataling, but not by a huge margin, and not likely to be what they expect.

Test Plan:
Before D18836, disabled both "Profile" and "Workboard" items on a project. Visited project page.

Before patch: fatal. After patch: manage page.

After D18836, you can't do this and just get the profile, so this is sort of moot and mostly future-proofing/for-completeness.

Reviewers: 20after4, amckinley

Reviewed By: amckinley

Maniphest Tasks: T13033

Differential Revision: https://secure.phabricator.com/D18843
2017-12-23 11:37:12 -08:00
Dmitri Iouchtchenko
393824656f Don't notify without notifiable attendees
Summary: Events with no attendees (e.g. fresh instances of recurring events) would trigger an exception when sending notifications, because `$attendee_map` would still get populated.

Test Plan: Declined event, restarted daemons. Did not see exception. Accepted event, restarted daemons. Saw "[Calendar] [Reminder]" email.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D18835
2017-12-21 12:46:46 -08:00
epriestley
e411d75964 Fix an issue where blame could fatal for unrecognized authors
Summary: See PHI255. See <https://discourse.phabricator-community.org/t/error-generating-blame-data/766>.

Test Plan:
  - Viewed a file contributed to by users with no Phabricator user accounts, in Diffusion.
  - Enabled blame.
  - Before patch: blame failed, fatal in logs.
  - After patch: blame worked.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18833
2017-12-20 11:20:23 -08:00
epriestley
7cbbe2ccf7 When users browse to a submodule path in Diffusion explicitly, don't fatal
Summary: Ref T13030. See PHI254. This behavior could be cleaner than I've made it, but it fixes the "this is totally broken" issue, replacing a fatal/exception with an informative (just not terribly useful) page.

Test Plan:
  - Added a submodule to a repository.
  - In Diffusion, clicked some other file next to the submodule, then edited the URI to the submodule path instead.
    - Before patch: fatal.
    - After patch: relatively useful message about this being a submodule.

Note that it's normally hard to hit this URI directly. In the browse view, submodules are marked up as directories and linked to a separate submodule resolution flow.

{F5321524}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13030

Differential Revision: https://secure.phabricator.com/D18831
2017-12-18 09:18:22 -08:00
epriestley
a1ad184ddd Denormalize added and removed line counts for the current diff onto revisions
Summary:
See PHI230. Currently, we denormalize raw line counts onto diffs and revisions, but not added/removed line counts.

I'd like to try a `[---+  ]` sort of size hint element (see D16322 for more) as a general approach to conveying size information at a glance and see how it feels, since I think the raw size number isn't very scannable/useful and it may be a significant improvement to hint about how much of a change is throwing stuff out vs adding new stuff.

This just makes the data available without any subquerying and doesn't actually change the UI.

Test Plan:
Created a revision, saw detailed change information populate in the database.

```
mysql> select * from differential_revision where id = 292\G
*************************** 1. row ***************************
              id: 292
           title: WIP
   originalTitle: WIP
            phid: PHID-DREV-ux3cxptibn3l5pxsug3z
          status: draft
         summary: asdf
        testPlan: asdf
      authorPHID: PHID-USER-cvfydnwadpdj7vdon36z
lastReviewerPHID: NULL
       lineCount: 41
     dateCreated: 1513179418
    dateModified: 1513179418
        attached: []
         mailKey: h4mn6perdio47o4beomyvu75zezwvredx3mbrlgz
      branchName: NULL
      viewPolicy: users
      editPolicy: users
  repositoryPHID: PHID-REPO-wif5lutk5gn3y6ursk4p
      properties: {"lines.added":40,"lines.removed":1}
  activeDiffPHID: PHID-DIFF-ixjphpunpkenqgukpmce
1 row in set (0.00 sec)
```

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18832
2017-12-18 09:17:55 -08:00
epriestley
e1d6bad864 Stop trying to assess the image dimensions of large files and file chunks
Summary:
Depends on D18828. Ref T7789. See <https://discourse.phabricator-community.org/t/git-lfs-fails-with-large-images/584>.

Currently, when you upload a large (>4MB) image, we may try to assess the dimensions for the image and for each individual chunk.

At best, this is slow and not useful. At worst, it fatals or consumes a ton of memory and I/O we don't need to be using.

Instead:

  - Don't try to assess dimensions for chunked files.
  - Don't try to assess dimensions for the chunks themselves.
  - Squelch errors for bad data, etc., that `gd` can't actually read, since we recover sensibly.

Test Plan:
  - Created a 2048x2048 PNG in Photoshop using the "Random Noise" filter which weighs 8.5MB.
  - Uploaded it.
  - Before patch: got complaints in log about `imagecreatefromstring()` failing, although the actual upload went OK in my environment.
  - After patch: clean log, no attempt to detect the size of a big image.
  - Also uploaded a small image, got dimensions detected properly still.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T7789

Differential Revision: https://secure.phabricator.com/D18830
2017-12-18 09:17:32 -08:00
epriestley
5295840a4c Restore the "Download from Git LFS" UI button to Diffusion
Summary: Depends on D18827. Ref T7789. See PHI204. See PHI131. This button got accidentally removed in Diffusion refactoring (`$data` is no longer used).

Test Plan: {F5321459}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T7789

Differential Revision: https://secure.phabricator.com/D18828
2017-12-18 09:13:19 -08:00
epriestley
8e416474c0 Add a Herald pre-commit field for detecting LFS usage
Summary: Depends on D18825. Ref T7789. See PHI131. Allows installs to selectively disable LFS by adding Herald rules to block commits that use LFS.

Test Plan:
  - Wrote an LFS rule ("When commit uses git lfs, block commit").
  - Pushed an LFS commit: rejected.
  - Pushed a non-lFS commit: success.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T7789

Differential Revision: https://secure.phabricator.com/D18827
2017-12-18 09:12:52 -08:00
epriestley
e34b4bbd90 Move the Git LFS gate to dedicated (non-prototype) config
Summary: See PHI131. Ref T7789. Although this probably isn't 100% complete, there don't seem to be any actual, known, practical blocking issues remaining (everything is either heresay or not reproducible).

Test Plan: Tried to push LFS locally, got blocked with a helpful message. Enabled setting, tried to push LFS locally, got a successful push.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T7789

Differential Revision: https://secure.phabricator.com/D18825
2017-12-18 09:12:22 -08:00
epriestley
c9a0d68340 Allow Herald rules to add comments
Summary:
See PHI242. All use cases for this that I know of are pretty hacky, but they don't seem perilous, and it's easier than webhooks.

See P1895, T10183, and T9853 for me previously refusing to implement this since all those use cases were also pretty bad.

Test Plan:
  - Wrote a rule to add comments, saw it add comments.
  - Reviewed summary, re-edited rule, reviewed transcript to check that all the strings worked OK.
  - Wrote a new rule for a non-commentable object (a blog) to make sure I wasn't offered the "Add a comment" action.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18823
2017-12-18 09:10:57 -08:00
Tim Hirsh
60e5c0ec1b Add drydock.blueprint.edit Conduit method
Summary:
Ref: https://admin.phacility.com/PHI243

Since our use case primarily focuses on transaction editing, this patch implements the `drydock.blueprint.edit` api method with the understanding that:
a) this is a work in progress
b) object editing is supported, but object creation is not yet implemented

Test Plan:
* updated existing blueprints via Conduit UI
* regression tested `maniphest.edit` by creating new and updating existing tasks

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, yelirekim, jcox

Differential Revision: https://secure.phabricator.com/D18822
2017-12-08 11:55:08 -05:00
epriestley
6d3baa92f9 Execute Herald again after promoting revisions out of the "Draft" state
Summary:
Fixes T13027. Ref T2543. When revisions promote from "Draft" because builds finish or no builds are configured, the status currently switches from "Draft" to "Needs Review" without re-running Herald.

This means that some rules -- notably, "Send me an email" rules -- don't fire as soon as they should.

Instead of applying this promotion in a hacky way inline, queue it and apply it normally in a second edit, after the current group finishes.

Test Plan:
  - Created a revision, reviewed Herald transcripts.
  - Saw three Herald passes:
    - First pass (revision creation) triggered builds and no email.
    - Second pass (builds finished) did not trigger builds (no update) and did not trigger email (revision still a draft).
    - Third pass (after promotion out of 'draft') did not trigger builds (no update) but did trigger email (revision no longer a draft).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13027, T2543

Differential Revision: https://secure.phabricator.com/D18819
2017-12-05 12:13:56 -08:00
epriestley
4f8340c05f Restore the "Log In" menubar action
Summary:
See <https://discourse.phabricator-community.org/t/activation-link-in-welcome-mail-only-works-if-new-user-isnt-semi-logged-in/740/7>.

In T13024, I rewrote the main menu bar to hide potentially sensitive items (like notification and message counts and saved search filters) until users fully log in.

However, the "Log In" item got caught in this too. For clarity, rename `shouldAllowPartialSessions()` to `shouldRequireFullSession()` (since logged-out users don't have any session at all, so it would be a bit misleading to say that "Log In" "allows" a partial session). Then let "Log In" work again for logged-out users.

(In most cases, users are prompted to log in when they take an action which requires them to be logged in -- like creating or editing an object, or adding comments -- so this item doesn't really need to exist. However, it aligns better with user expectations in many cases to have it present, and some reasonable operations like "Check if I have notifications/messages" don't have an obvious thing to click otherwise.)

Test Plan: Viewed site in an incognito window, saw "Log In" button again. Browsed normally, saw normal menu.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18818
2017-12-05 12:13:10 -08:00
epriestley
a989dd181d Fix Mercurial commit history ordering
Summary:
See <https://discourse.phabricator-community.org/t/diffusion-observed-mercurial-repository-history-broken/825>.

In D18769, I rewrote this from using the `--branch` flag (which is unsafe and does not function on branches named `--config=x.y` and such).

However, this rewrite accidentally changed the result order, which impacted Mercurial commit hisotry lists and graphs. Swap the order of the constraints so we get newest-to-oldest again, as expected.

Test Plan: Viewed a Mercurial repository's history graph, saw sensible chronology after the patch.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18817
2017-12-05 12:12:44 -08:00
lkassianik
46d496b8cc Fix error for URL's that could mean several commits
Summary: Ref T13001, URLs that return multiple commits should show a list of those commits. Not sure if the actual list looks very pretty this way, but was wondering if this approach was vaguely correct.

Test Plan:
- Navigate to `install/rPbd3c23`
- User should see a list view providing links to `install/rPbd3c2355e8e2b220ae5e3cbfe4a057c8088c6a38` and `install/rPbd3c239d5aada68a31db5742bbb8ec099074a561`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Maniphest Tasks: T13001

Differential Revision: https://secure.phabricator.com/D18816
2017-12-05 19:24:57 +00:00
epriestley
c924351a58 Mark sessions as "signed all documents" when Legalpad has been uninstalled
See PHI238. When an install uninstalls "Legalpad", we were incorrectly failing
to mark sessions as "Signed All Required Documents" by bailing early.

Test Plan: Uninstalled Legalpad, logged in.
2017-12-02 16:15:59 -08:00
lkassianik
42034e6739 Property list view on Diffusion commits should show build status but not Subscriptions, Projects, or Tokens
Summary: Ref T13019, adds build status back to Diffusion commits

Test Plan: Open a Diffusion commit that has a build status, property list view should show the build status, but not Subscriptions, Projects, or Tokens.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Maniphest Tasks: T13019

Differential Revision: https://secure.phabricator.com/D18813
2017-12-01 18:16:26 +00:00
epriestley
5240cffd9c Fix an issue where Diffusion could fatal if the default branch was deleted
Summary: See PHI234. In T12931 we improved the behavior of Diffusion when a repository's default branch is set to a branch that does not exist, but in T11823 the way refcursors work changed, and we can now get a cursor (just with no positions) back for a deleted branch. When we did, we didn't handle things gracefully.

Test Plan:
  - Set default branch to a deleted branch, saw nice error instead of fatal.
  - Set default branch to a nonexistent branch which never existed, saw nice error.
  - Set default branch to existing "master", saw repository normally.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18811
2017-11-30 14:06:41 -08:00
epriestley
14cc0abeb3 Fix several safety issues with repository URIs
Summary:
See PHI234. Several issues here:

  - The warning about observing a repository in Read/Write mode checks the raw I/O type, not the effective I/O type. That means we can fail to warn if other URIs are set to "Default", and "Default" is "Read/Write" in practice.
  - There's just an actual typo which prevents the "Observe" version of this error from triggering properly.

Additionally, add more forceful warnings that "Observe" and "Mirror" mean that you want to //replace// a repository with another one, not that we somehow merge branches selectively. It isn't necessarily obvious that "Observe" doesn't mean "merge/union", since the reasons it can't in the general case are somewhat subtle (conflicts between refs with the same names, detecting ref deletion).

Test Plan:
Read documentation. Hit the error locally by trying to "Observe" while in Read/Write mode:

{F5302655}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18810
2017-11-30 14:06:21 -08:00
epriestley
f786c86a6a Don't require the "gd" extension be installed in order to run unit tests
Summary:
Ref T10405. If `gd` is not installed, a number of unit tests currently fail because they generate synthetic users and try to composite profile pictures for them.

Instead, just fall back to a static default if `gd` is not available.

Test Plan: Faked this pathway, created a new user, saw a default profile picture.

Reviewers: amckinley, lpriestley, avivey

Reviewed By: avivey

Maniphest Tasks: T10405

Differential Revision: https://secure.phabricator.com/D18812
2017-11-30 13:51:31 -08:00
epriestley
0807b70ea1 Add an explicit warning in the Differential transaction log when users skip review
Summary:
Ref T10233. See PHI231. When users ignore the `arc land` prompt about bad revision states, make it explicitly clear in the transaction log that they broke the rules.

You can currently figure this out by noticing that there's no "This revision is accepted and ready to land." message, but it's unrealistic to expect non-expert users to look for the //absence// of a message to indicate something, and this state change is often relevant.

Test Plan: {F5302351}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T10233

Differential Revision: https://secure.phabricator.com/D18808
2017-11-30 11:03:55 -08:00
Aviv Eyal
d8f2630d5c Modernize QuickSearch typeahead
Summary:
Use ClassQuery to find datasources for the quick-search.

Mostly, this allows extensions to add quicksearches.

Test Plan:
using `/typeahead/class/`, tested several search terms that make sense.
Removed the tag interface from a datasource, which removed it from results.

Reviewers: epriestley, amckinley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18760
2017-11-30 15:07:49 +00:00
epriestley
1b76250f89 Disallow "Accept" on drafts, allow "Resign"
Summary:
Depends on D18801. Ref T2543. See PHI229. I missed "Accept" before, but intended to disallow it (like "Reject") since I don't want drafts to be reviewable.

However, "Resign" seems fine to allow? So let's allow that for now.

Test Plan: Was no longer offered "Accept" on draft revisions.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18802
2017-11-28 13:46:26 -08:00
epriestley
54f131dc4b Make "first broadcast" rules for Differential drafts more general
Summary:
See PHI228. Ref T2543. The current logic gets this slightly wrong: prototypes are off, you create a draft with `--draft`, then promote it with "Request Review". This misses both branches.

Instead, test these conditions a little more broadly. We also need to store broadcast state since `getIsNewObject()` isn't good enough with this workflow.

Test Plan:
  - With prototypes on and autopromotion, got a rich email after builds finished.
  - With prototypes off, got a rich email immediately.
  - With prototypes off and `--draft`, got a rich email after "Request Review".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18801
2017-11-28 13:42:10 -08:00
epriestley
e919233b31 Don't show personalized menu items until users establish a full session
Summary:
Depends on D18792. Fixes T13024. Fixes T89198. Currently, when users are logging in initially (for example, need to enter MFA) we show more menu items than we should.

Notably, we may show some personalized/private account details, like the number of unread notifications (probably not relevant) or a user's saved queries (possibly sensitive). At best these are misleading (they won't work yet) and there's an outside possibility they leak a little bit of private data.

Instead, nuke everything except "Log Out" when users have partial sessions.

Test Plan:
Hit a partial session (MFA required, email verification required) and looked at the menu. Only saw "Log Out".

{F5297713}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13024

Differential Revision: https://secure.phabricator.com/D18793
2017-11-28 10:01:58 -08:00
epriestley
dc62d18b47 Allow MFA enrollment before email verification
Summary: Depends on D18791. Ref T13024. This clears up another initialization order issue, where an unverified address could prevent MFA enrollment.

Test Plan: Configured both verification required and MFA required, clicked "Add Factor", got a dialog for the workflow.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13024

Differential Revision: https://secure.phabricator.com/D18792
2017-11-28 10:01:09 -08:00
epriestley
ab0f61aa32 Tell users to "Wait Patiently" for admin account verification later in the registration process
Summary:
Depends on D18790. Ref T13024. Fixes T8335. Currently, "unapproved" and "disabled" users are bundled together. This prevents users from completing some registration steps (verification, legalpad documents, MFA enrollment) before approval.

Separate approval out and move it to the end so users can do all the required enrollment stuff on their end before we roadblock them.

Test Plan: Required approval, email verification, signatures, and MFA. Registered an account. Verified email, signed documents, enrolled in MFA, and then got prompted to wait for approval.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13024, T8335

Differential Revision: https://secure.phabricator.com/D18791
2017-11-28 10:00:03 -08:00
epriestley
c7718d3a21 Ask users to sign Legalpad documents before requiring they enroll in MFA
Summary:
Depends on D18789. Ref T13024. See PHI223. Currently, if `security.require-multi-factor-auth` and Legalpad "Signature Required" documents are //both// set, it's not possible to survive account registration, since MFA is requiried to sign and signatures are required to add MFA.

Instead, check for signatures before requiring MFA enrollment. This makes logical sense, since it's silly to add MFA if you don't agree to a Terms of Service or whatever.

(Note that if you already have MFA, we prompt for that first, before either of these steps, which also makes sense.)

Test Plan: Configured `security.require-multi-factor-auth`. Added a signature-required document. Loaded a page as a new user. Went through signature workflow, then through the MFA enrollment workflow.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13024

Differential Revision: https://secure.phabricator.com/D18790
2017-11-28 09:59:39 -08:00
epriestley
e850bc6b95 When requiring login signatures, order documents from oldest to newest
Summary: Depends on D18788. Ref T13024. Currently, we prompt users to sign from newest to oldest. This seems less intuitive than oldest to newest.

Test Plan: Dumped document order, saw it swap to oldest-first.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13024

Differential Revision: https://secure.phabricator.com/D18789
2017-11-28 09:59:22 -08:00
epriestley
ba4b9f7184 Refactor on-login Legalpad document signature requirement
Summary: Depends on D18786. Ref T13024. I'm going to change the order this occurs in, but move it to a separate method and clean it up a little first.

Test Plan: Added a new document as required, reloaded, signed it, got logged into a session.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13024

Differential Revision: https://secure.phabricator.com/D18788
2017-11-28 09:58:53 -08:00
epriestley
71406ca93b Lightly modernize LegalpadDocumentSearchEngine
Summary: Depends on D18785. Ref T13024. While I'm in here, update this a bit to use the newer stuff.

Test Plan: Searched for Legalpad documents, saw more modern support (subscribers, order by, viewer()).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13024

Differential Revision: https://secure.phabricator.com/D18786
2017-11-28 09:56:49 -08:00
epriestley
3d742eb50b Lightly modernize LegalpadDocumentQuery
Summary: Ref T13024. Updates LegalpadDocumentQuery to use slightly more modern constructions.

Test Plan: Queried for Legalpad documents, got the same results.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13024

Differential Revision: https://secure.phabricator.com/D18785
2017-11-28 09:56:33 -08:00
epriestley
00baa3c1dd Hide archived projects only on workboards, not hovercards
Summary:
See PHI225. Previously, see D15335 / T10413. On workboards, we hide archived project tags since they aren't terribly useful in that context, at least most of the time. Originally, see T10349#159916 and D15297.

However, hovercards reuse this display logic, and it's inconsistent/confusing to hide them there, since the actual "Tags" elements on task pages show them. Narrow the scope of this rule.

Test Plan:
  - Viewed a hovercard for a task with an archived project tagged, saw archived project.
  - Viewed a workboard for the same task, saw only unarchived projects other than the current board tagged (this behavior is unchanged).

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18783
2017-11-27 14:37:51 -08:00
epriestley
49b57eae7d Revert partial/nonfunctional OpenGraph support
Summary:
Ref T13018. See that task and the Discourse thread for discussion.

This doesn't work as-is and we need to `og:description` everything to make it work. I don't want to sink any more time into this so just back all the changes out for now.

(The `<html>` change is unnecessary anyway.)

Test Plan: Strict revert.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13018

Differential Revision: https://secure.phabricator.com/D18782
2017-11-22 15:21:10 -08:00
epriestley
2c72c2b924 Add basic support for OpenGraph header tags for public installs
Summary: Ref T13018. This is easy to get working roughly, at least, and seems reasonable.

Test Plan: Viewed page source, saw tags. Custom header logo still worked. Pretty hard to debug against a local install since Disqus / debugger tools can't hit it, but I'll see what it looks like in production and tweak it if I got anything horribly wrong.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13018

Differential Revision: https://secure.phabricator.com/D18780
2017-11-22 11:16:56 -08:00
epriestley
2d4a158356 Fix a bad link target in Diffusion content search results
Summary: See <https://discourse.phabricator-community.org/t/broken-links-in-diffusion-pattern-search-results-page/757>. This links to the display path, which is incorrect.

Test Plan:
  - In any repository, browsed into a directory.
  - Used pattern search to search for something that hits results.
  - Clicked the title (filename/path) of a result table.
    - Before patch: URL omits path context, 404 or wrong result.
    - After patch: taken to proper page.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18779
2017-11-22 11:16:14 -08:00
epriestley
d321cc810a Freeze "maniphest.gettasktransactions" and make status/priority transactions more consistent
Summary:
Ref T13020. See PHI221.

Freeze legacy method `maniphest.gettasktransactions` in favor of modern method `transaction.search`.

Remove legacy "null on create" behavior from Maniphest status and priority transactions. This behavior is obsolete with EditEngine, and leads to inconsistent transaction sets in the transaction record.

The desired behavior is that transactions which don't do anything (e.g., default value was not changed) don't appear in the transaction log.

Test Plan:
  - Viewed API UI and saw `maniphest.gettasktransactions` marked as "Frozen".
  - Created a new task via web UI (without changing status/priority), queried transactions with `maniphest.gettasktransacitons`/`transaction.search`, no longer saw "null on create" no-op transactions in record.
    - Web UI is unchanged, since these transactions were hidden before and now do not exist.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13020

Differential Revision: https://secure.phabricator.com/D18777
2017-11-22 11:13:53 -08:00
Austin McKinley
bea45e90d3 Add yaml files to differential.whitespace-matters
Summary: Whitespace has semantic meaning for yaml files, so we shouldn't suppress whitespace-only lines of diff by default.

Test Plan: Edited local config to include yaml files, saw expected whitespace changes.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18775
2017-11-15 11:57:11 -08:00
epriestley
3700bcb638 Warn and prevent 1-up/2-up switch in Differential if the user is editing an inline
Summary:
See PHI180. Currently, if you begin creating or editing an inline and then swap display modes (for example, with "View Unified"), your edit is lost.

Persisting the editor state is complicated and this is very rare, so just prevent the action and warn the user instead.

Also make the warning persist for a little longer since a few of the messages, including this one, take a couple seconds to read now.

Test Plan:
  - Edited a comment, tried to swap display modes, got a warning.
  - Swapped display modes normally with no comment being edited.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18774
2017-11-15 10:02:48 -08:00
Mukunda Modell
a1f12b4ac7 Specify a null behavior for the callsign sort column.
Summary:
Fixes paging on the Diffusion Repository List.

PhabricatorRepositoryQuery needs to specify a behavior for `null` on the OderableColumns definition for the `callsign` column.

See https://phabricator.wikimedia.org/T180457

Test Plan:
1. On an instance with more than 100 repositories
 * some of which are missing a callsign
2. Attempt to sort by callsign.
3. See the sorted results

Previously:

3. Exception: "Column "0" has null value, but does not specify a null behavior."

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18773
2017-11-14 17:16:01 -06:00
epriestley
95a5b41b0b Allow "arc diff --draft" to create revisions as drafts more forcefully
Summary:
Depends on D18771. See PHI206. Currently, `arc diff --draft` only holds revisions in draft mode: it doesn't put them into draft mode if the install isn't configured to use draft mode.

Instead, make it a bit more forceful so that `arc diff --draft` can create into draft mode explicitly even if protoypes are off. This aligns with expection a little more clearly.

Test Plan: Ran `arc diff --draft` with prototypes off, got a revision held in draft mode.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18772
2017-11-14 12:30:47 -08:00
epriestley
314e7266c3 Restore "Summary" and "Test Plan" to initial mail for non-draft configurations
Summary: See PHI210. Ref T2543. Currently, we don't set this flag if you have prototypes off and don't get any of the new draft stuff, so the mail drops some of the details it is supposed to have.

Test Plan: Disabled prototypes, created a revision, saw summary / test plan in the initial mail.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18771
2017-11-14 12:23:15 -08:00
epriestley
a7921a4448 Filter and reject "--config" and "--debugger" flags to Mercurial in any position
Summary:
Ref T13012. These flags can be exploited by attackers to execute code remotely. See T13012 for discussion and context.

Additionally, harden some Mercurial commands where possible (by using additional quoting or embedding arguments in other constructs) so they resist these flags and behave properly when passed arguments with these values.

Test Plan:
  - Added unit tests.
  - Verified "--config" and "--debugger" commands are rejected.
  - Verified more commands now work properly even with branches and files named `--debugger`, although not all of them do.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13012

Differential Revision: https://secure.phabricator.com/D18769
2017-11-10 08:42:07 -08:00
epriestley
759c757264 Include "Draft" revisions in Differential legacy status queries
Summary:
See PHI199. Ref T2543. When you run a RevisionQuery with a legacy status constraint (via `differential.query`), we currently don't match "Draft" revisions.

Use the actual complete map from `DifferentialRevisionStatus` instead of hard coding the status list so "Draft" is included.

Test Plan:
  - Ran `differential.query` with `ids` and `status` for a draft revision.
  - Before patch: revision not returned in results.
  - After patch: revision returned in results.

(Note that it returns as "Needs Review", for compatibility.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18765
2017-11-07 15:35:33 -08:00
epriestley
12e6106a59 Refine bucketing and display rules for voided "Accepts" in Differential
Summary:
See PHI190. This clarifies the ruleset a bit:

  - If you accepted, then the author used "Request Review" explicitly, we now show "Accepted Earlier" instead of "Accepted" in the "Reviewers" list on the main revision page. This makes it sligthly more clear why the revision is back in your review queue without picking through the transaction log.
  - Instead of moving all non-current accepts into "Ready to Review", move only voided accepts into "Ready to Review". This stops us from pulling older accepts which haven't been voided (which could have been incorrectly pulled) and correctly pulls older, voided accepts from before an update (for example: accept, then request review, then update) and generally aligns better with intent/expectation.

Test Plan:
  - Accepted, requested review.
  - Saw reviewer as "Accepted Earlier".
  - Saw review in "Ready to Review" bucket.
  - Accepted, updated (with sticky accept).
  - Saw reviewer as "Accepted Prior Diff".
  - Saw review as "Waiting on Authors".

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18764
2017-11-07 15:35:11 -08:00
epriestley
587faa6f67 Remove some defunct old-style transaction policy checks
Summary:
Ref PHI193. This method of enforcing policy checks is now (mostly) obsolete, and they're generally checked at the Controller/API level instead.

Notably, this method does not call `adjustObjectForPolicyChecks(...)` properly, so it can not handle special cases like "creating a project and taking its newly created members into account" for object policies like "Project Members".

Just remove these checks, which are redundant with checks elsewhere.

Test Plan:
  - Set Project application default edit policy to "Administrators and Project Members".
  - Tried to create a project as a non-administrator, adding myself.
  - Before patch: policy fatal on a VOID object (the project with no PHID generated yet).
  - After patch: object created properly. Got a sensible policy error if I didn't include myself as a member.
  - Also verified that other edit rules are still enforced/respected (I can't edit stuff I shouldn't be able to edit).
  - There's at least a bit of unit test coverage of this, too, which I updated to work via API (which hits the new broad capability checks) instead of via low-level transactions (which enforce only a subset of policy operations now).

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18763
2017-11-07 15:34:51 -08:00
epriestley
cc865e549b Provide revision parent/child edges in edge.search, and more information in differential.revision.search
Summary: See PHI195. This bulks out these API methods since all the requests are pretty straightforward.

Test Plan: Ran `edge.search` and `differential.revision.search`.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18762
2017-11-07 15:34:31 -08:00
epriestley
cb98b60033 Fill in some straightforward Maniphest transactions for transaction.search
Summary:
See PHI197. Populates "status" transactions and a few other obvious types where there's no security/performance/payload/formatting issue I can come up with.

The names here are the same as the names for editing with `maniphest.edit`.

Test Plan: Used `transaction.search` to retrieve transactions of all new types.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18761
2017-11-07 15:33:55 -08:00
epriestley
03d059dd26 Don't include resigned reviewers in the Differential "To" list
Summary: Ref T12689. See PHI178. This isn't a complete solution (you may still get mailed via packages/projects) but should fix the obvious issue, where "Resigned" reviewers are incorrectly always sent mail directly.

Test Plan: Had Alice resign, interacted as Bailey, no mail to Alice.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12689

Differential Revision: https://secure.phabricator.com/D18758
2017-11-01 17:22:45 -07:00
epriestley
6ecdadb76a After "Request Review", move revisions with voided "Accepts" into "Ready to Review", not "Waiting on Other Reviewers"
Summary:
Depends on D18756. Fixes T12539. See PHI190. Currently, when this occurs:

  - Alice accepts.
  - Bailey requests review.
  - Alice views her dashboard.

...the revision appears in "Waiting on Other Reviewers" (regardless of whether other reviewers actually exist or not).

Instead, ignore these voided/non-current accepts and let the revisions appear in "Ready to Review", which is more natural.

Test Plan: Went through the steps above. On `master`, saw revision in "Waiting on Other Reviewers". After patch, saw it in "Ready to Review".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12539

Differential Revision: https://secure.phabricator.com/D18757
2017-11-01 17:19:54 -07:00
epriestley
6d36eb9113 Denormalize Diff PHIDs onto Revisions
Summary:
Ref T12539. See PHI190. Currently, each Diff has a `revisionID`, but Revisions do not point at the current active diff. To find the active diff for a given revision, we need to issue a separate query.

Furthermore, this query is inefficient for bulk loads: if we have a lot of revisions, we end up querying for all diff IDs for all those revisions first, then selecting the largest ones and querying again to get the actual diff objects. This strategy could likely be optimized but the query is a mess in any case.

In several cases, it's useful to have the active diff PHID without needing to do a second query -- sometimes for convenience, and sometimes for performance.

T12539 is an example of such a case: it would be nice to refine the bucketing logic (which only depends on active diff PHIDs), but it feels bad to make the page heavier to do it.

For now, this is unused. I'll start using it to fix the bucketing issue, and then we can expand it gradually to address other performance/convenience issues.

Test Plan:
  - Ran migrations, inspected database, saw sensible values.
  - Created a new revision, saw a sensible database value.
  - Updated an existing revision, saw database update properly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12539

Differential Revision: https://secure.phabricator.com/D18756
2017-11-01 17:19:38 -07:00
epriestley
80ebe401e5 Tweak padding/spacing on Diffusion blame view for profile pictures
Summary: Give profile images a little more space, fix "/" spacing, add a tooltip.

Test Plan: {F5251205}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18749
2017-10-31 12:56:36 -07:00
epriestley
a4b934cad2 Clean up Differential draft mail behaviors
Summary:
Ref T2543. Fixes two relatively minor things:

  - When builds finish in Harbormaster, send mail "From" the author.
  - Set the `firstBroadcast` flag so that initial mail picks up earlier history (notably, the "reviewers" line).

For now, I'm not setting `firstBroadcast` on explicit "Request Review" (but maybe we should), and not trying to deal with weird cases where you leave a bunch of comments on a draft. Those might be fine as-is or may get tweaked later.

Test Plan: Created a revision with Harbormaster builds, ran builds, saw initial email come "From" the right user with more metadata.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18748
2017-10-31 12:56:03 -07:00
epriestley
bde71324f8 Show small author portraits in Diffusion blame view
Summary: Depends on D18746. See PHI174. Adds small author portraits next to each blame line (this is similar to GitHub).

Test Plan:
My local test data isn't that great since I don't have commits from a lot of accounts, but looks functional:

{F5251056}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18747
2017-10-31 12:10:45 -07:00
epriestley
90d0f8ac6c Revert changes to Diffusion blame view
Summary:
Ref PHI174. This reverts most of these changes:

- 37843127e9 / D18481
- 94cad30ac3 / D18474
- 12ae08b6b1 / D18473
- 0a01334172 / D18462
- ac91ab1ef9 / D18452

These changes made the Diffusion blame view very similar to GitHub's blame view. See D18452 for a before/after of the bulk of these changes; the other revisions are bugfixes.

I think this was generally a step backward, and not motivated by solving a specific problem. I've found the new UI less usable than the old one, and at least one install (see PHI174) also has.

In particular, the revision/commit titles are very bulky and not terribly useful; the date column also isn't terribly useful; the "age" color actually IS pretty useful and was heavily de-emphasized.

I've kept one bugfix here (missing `'a'` tag type) and kept the upgraded icon for "Skip Past This Commit".

I'm going to follow this up with some additional changes:

  - Show a small author profile icon, similar to GitHub, to address PHI174 more directly.
  - Try a zebra-stripe on blocks of rows to make it more clear where changes affected by a particular commit begin and end.
  - Try a hue shift, not just a brightness/saturation shift, to make the "age" color more distinct.
  - Try computing colors as even steps, not based purely on age. Currently, if a file has one long-distant commit and several recent commits, all the recent ones show up as very bright green. I think this would probably be more useful if they were distributed more evenly across the available color bands.

Test Plan:
Viewed blame views in Diffusion, saw a more compact UI similar to the old UI.

{F5251019}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18746
2017-10-31 11:54:47 -07:00
epriestley
7fa0d066bc Don't run Herald build rules when Differential revisions are updated automatically
Summary: Ref T2543. After D18731, Herald build rules run more often, but now incorrectly try to run builds when Diffusion closes a revision because a commit landed.

Test Plan: Made some mundane updates locally; this is tricky to test comprehensively locally so I'm mostly planning to just push it to `secure`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18745
2017-10-31 11:36:04 -07:00
epriestley
28cec2f8a2 Allow revisions to be held as drafts, even after builds finish
Summary:
Ref T2543. Instead of autosubmitting revisions to "Needs Review" when builds finish, allow them to be held in "Draft" indefinitely.

There's currently no UI for this. I plan to just expose it as `arc diff --draft` for now, in a followup change.

Test Plan:
  - Created a revision (via Conduit) with "hold as draft", saw it hold as draft after builds finished.
  - Created a revision (normally), saw it autosubmit after builds finished.
  - Requested review of a "hold as draft" revision to kick it out of draft state.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18737
2017-10-31 09:39:32 -07:00
epriestley
f5336cd6e7 Return transactions from "differential.parsecommitmessage"
Summary:
Depends on D18740. Prepares `arc` to receive a `--draft` flag by letting us switch to "differential.revision.edit" instead of "differential.createrevision".

To "differential.revision.edit", we need a transaction list, but we can't automatically construct this list from a field map. Return the transaction list alongside the field map.

The next change uses this list (if available) to switch us to the modern API method.

Test Plan: Ran `arc diff` on the experiemntal branch with followup changes, got a new revision.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18741
2017-10-30 15:15:42 -07:00
epriestley
0da3f34728 Provide "differential.diff.search"
Summary: See PHI90. For now, this only provides a limited amount of information, but should satisfy the use case in PHI90 and build toward a more complete version in the future.

Test Plan: Used new Conduit method to retrieve information about diffs.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18744
2017-10-30 15:06:10 -07:00
epriestley
f7f3dd5b20 Don't run Herald build and mail rules when they don't make sense
Summary:
Ref T2543. Fixes T10109.

Currently, Herald only runs in Differential when a change updates the diff. This is partly for historical reasons, and partly because we don't want to restart builds every time someone makes a comment. However, this behavior is inconsistent with other applications (which always trigger on any change), and occasionally confusing to users (in T10109, for example) or otherwise undesirable.

A similar issue is that T2543 has introduced a "Draft" state, where revisions don't send normal mail until builds finish. This interacts poorly with "Send me an email" rules (which shouldn't do anything here) and particularly with "Send me an email + only run these actions the first time the rule matches", since that might have an effect like "do nothing when the revision is created, then never anything again since you already did nothing once".

To navigate both of these issues, let objects tell Herald that certain actions (like mail or builds) are currently forbidden. If a rule uses a field or action which is currently forbidden, the whole rule automatically fails before it executes, but doesn't count toward "only the first time" as far as Herald's tracking of rule execution is concerned.

Then, forbid mail for draft revisions, and forbid builds for revisions which didn't just get updated. Forbidding mail fixes the issues with "Send me an email" that were created by the introduction of the draft state.

Finally, make Herald run on every revision update, not just substantive updates to the diff. This resolves T10109.

Test Plan:
Created revisions via the draft -> submit workflow, saw different transcripts. Here's a mail action being forbidden for a draft:

{F5237324}

Here's a build action being forbidden for a "mundane" update:

{F5237326}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T10109, T2543

Differential Revision: https://secure.phabricator.com/D18731
2017-10-27 08:44:12 -07:00
epriestley
d1b4c9f50b Add a missing key on DrydockLease
Summary: Depends on D18734. See PHI176. We run this query on the main Drydock lease web UI, among other places. There is currently no `status` key which can satisfy it.

Test Plan:
Viewed Drydock lease page to get the query.

Ran ##explain SELECT * FROM `drydock_lease` WHERE (status IN ('pending', 'acquired', 'active')) ORDER BY `id` DESC LIMIT 101;## before and after the change.

I don't have a ton of leases locally so the un-key'd EXPLAIN isn't //that// bad, but still shows that we're getting a better key. Before:

```
mysql> explain SELECT * FROM `drydock_lease` WHERE (status IN ('pending', 'acquired', 'active')) ORDER BY `id` DESC LIMIT 101;
+----+-------------+---------------+-------+---------------+---------+---------+------+------+-------------+
| id | select_type | table         | type  | possible_keys | key     | key_len | ref  | rows | Extra       |
+----+-------------+---------------+-------+---------------+---------+---------+------+------+-------------+
|  1 | SIMPLE      | drydock_lease | index | NULL          | PRIMARY | 4       | NULL |  101 | Using where |
+----+-------------+---------------+-------+---------------+---------+---------+------+------+-------------+
1 row in set (0.00 sec)
```

After:

```
mysql> explain SELECT * FROM `drydock_lease` WHERE (status IN ('pending', 'acquired', 'active')) ORDER BY `id` DESC LIMIT 101;
+----+-------------+---------------+-------+---------------+------------+---------+------+------+---------------------------------------+
| id | select_type | table         | type  | possible_keys | key        | key_len | ref  | rows | Extra                                 |
+----+-------------+---------------+-------+---------------+------------+---------+------+------+---------------------------------------+
|  1 | SIMPLE      | drydock_lease | range | key_status    | key_status | 130     | NULL |    5 | Using index condition; Using filesort |
+----+-------------+---------------+-------+---------------+------------+---------+------+------+---------------------------------------+
1 row in set (0.00 sec)
```

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D18735
2017-10-26 18:20:38 -07:00
epriestley
b04d9fdc0b Add a missing key to PhabricatorFile for destroying Files
Summary: See PHI176. Depends on D18733. We issue a query when deleting files that currently doesn't hit any keys.

Test Plan:
Ran `./bin/remove destroy --force --trace F56376` to get the query.

Ran ##SELECT * FROM `file` WHERE storageEngine = 'blob' AND storageHandle = '23366' LIMIT 1## before and after the change.

Before:

```
mysql> explain SELECT * FROM `file` WHERE storageEngine = 'blob' AND storageHandle = '23366' LIMIT 1;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | file  | ALL  | NULL          | NULL | NULL    | NULL | 33866 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (0.01 sec)
```

After:

```
mysql> explain SELECT * FROM `file` WHERE storageEngine = 'blob' AND storageHandle = '23366' LIMIT 1;
+----+-------------+-------+------+---------------+------------+---------+-------------+------+------------------------------------+
| id | select_type | table | type | possible_keys | key        | key_len | ref         | rows | Extra                              |
+----+-------------+-------+------+---------------+------------+---------+-------------+------+------------------------------------+
|  1 | SIMPLE      | file  | ref  | key_engine    | key_engine | 388     | const,const |  190 | Using index condition; Using where |
+----+-------------+-------+------+---------------+------------+---------+-------------+------+------------------------------------+
1 row in set (0.00 sec)
```

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18734
2017-10-26 18:20:12 -07:00
epriestley
fbfed82efd Add a missing DaemonLogEvent key for garbage collection
Summary: See PHI176. This is issued periodically by the garbage collector. Normally this table is relatively small-ish so this missing key isn't hugely noticeable.

Test Plan:
Ran `./bin/garbage collect --collector daemon.processes --trace` to get the query the GC runs.

Ran ##DELETE FROM `daemon_logevent` WHERE epoch < 1508443504 LIMIT 100## before and after the key, saw a much better query plan afterward:

Before:

```
mysql> explain DELETE FROM `daemon_logevent` WHERE epoch < 1508443504 LIMIT 100;
+----+-------------+-----------------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table           | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-----------------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | daemon_logevent | ALL  | NULL          | NULL | NULL    | NULL | 19325 | Using where |
+----+-------------+-----------------+------+---------------+------+---------+------+-------+-------------+
1 row in set (0.00 sec)
```

After:

```
mysql> explain DELETE FROM `daemon_logevent` WHERE epoch < 1508443504 LIMIT 100;
+----+-------------+-----------------+-------+---------------+-----------+---------+-------+------+-------------+
| id | select_type | table           | type  | possible_keys | key       | key_len | ref   | rows | Extra       |
+----+-------------+-----------------+-------+---------------+-----------+---------+-------+------+-------------+
|  1 | SIMPLE      | daemon_logevent | range | key_epoch     | key_epoch | 4       | const |    1 | Using where |
+----+-------------+-----------------+-------+---------------+-----------+---------+-------+------+-------------+
1 row in set (0.00 sec)
```

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18733
2017-10-26 18:19:46 -07:00
epriestley
a03da0c2af Add a missing artifactIndex key to HarbormasterBuildArtifact
Summary: See PHI176. We issue a query with only `artifactIndex` from `BuildTarget`, but don't have an applicable key.

Test Plan: This isn't on the normal Harbormaster execution path so I'm not 100% sure I have a local repro, but will confirm with customer.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D18732
2017-10-26 18:18:24 -07:00
epriestley
f1204c8c45 Convert Ponder Questions to Ferret engine
Summary: See PHI177. Ref T12974. PonderQuestion was overlooked during the Ferret engine conversions.

Test Plan:
Ran migrations, searched for questions, got results:

{F5241185}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12974

Differential Revision: https://secure.phabricator.com/D18736
2017-10-26 18:18:04 -07:00
epriestley
beaf0ad9a6 Attribute revision promotion from "Draft" to "Needs Review" to the author
Summary:
Ref T2543. When Harbormaster finishes builds and promotes a draft revision to review, we currently publish "Harbormaster requested review of...".

Instead, attribute this action to the author, since that's more natural and more useful.

Test Plan: Promoted a diff locally, saw it attributed to me rather than Harbormaster.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18730
2017-10-26 12:57:47 -07:00
epriestley
28a24c333f Fix a couple of other missing getApplicationTransactionCommentObject() implementations
Summary:
See PHI165. See D18715. These objects (projects, blogs) also need implementations now.

(I thought about making this method `abstract` or doing try/catch to maybe make this more robust, but I think this should be the end of it, and those changes have mild complexity/compatibility/risk issues.)

Test Plan: Changed `bin/search index` to index only one document of each type, ran `bin/search index --all --force`, saw no more comment-related errors.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18729
2017-10-24 09:05:23 -07:00
epriestley
683df399e7 Simplify UNION/ORDER query construction in DifferentialRevisionQuery
Summary: Ref T12680. Use the slightly sleeker construction from D18722 in Differential.

Test Plan: Viewed revision list, reordered by date modified.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12680

Differential Revision: https://secure.phabricator.com/D18727
2017-10-23 16:17:47 -07:00
epriestley
e3a48dde1d Correct a method signature in DifferentialDraftField
Summary:
Ref T12190. See <https://discourse.phabricator-community.org/t/exception-preventing-access-to-differential-application/606>.

(I have a followup to fix the root issue.)

Test Plan: Loaded Differential with an eye on the error log in PHP7, no longer saw warnings.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12190

Differential Revision: https://secure.phabricator.com/D18723
2017-10-23 10:33:53 -07:00
epriestley
157f47cd14 Rewrite CommitQuery to use UNION for performance
Summary:
Ref T12680. See PHI167. See that task for discussion.

Rewrite `DiffusionCommitQuery` to work more like `DifferentialRevisionQuery`, and use a UNION to find "all revisions you need to audit OR respond to".

I tried to get this working a little more cleanly than RevisionQuery does, and can probably simplify that now.

Test Plan: Poked at the UI locally without hitting any apparent issues, but my local data is pretty garbage at this point. I'll take a look at how the query plans work on `secure`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12680

Differential Revision: https://secure.phabricator.com/D18722
2017-10-23 10:32:24 -07:00
epriestley
63e6b2553e Simply how Differential drafts ignore Harbormaster autobuilds
Summary:
Ref T2543. When a revision is created, we check if any builds are waiting/failed, and submit it for review immediately if we aren't waiting for anything.

In doing this, we ignore builds with only autotargets, since these are client-side and failures from local `arc lint` / `arc unit` should not count (the user has already chosen to ignore/skip them).

The way we do this has some issues:

  - Herald may have started builds, but they may still be PENDING and not have any targets yet. In this case, we'll see "no non-autotargets" and ignore the build, which is wrong.
  - We have to load targets but don't really care about them, which is more work than we really need to do.
  - And it's kind of complex, too.

Instead, just let `BuildQuery` filter out "autobuilds" (builds generated from autoplans) with a JOIN.

Test Plan: Ran `arc diff` with builds configured, got a clean "Draft" state instead of an incorrect promotion directly to "Needs Review".

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18721
2017-10-23 10:31:48 -07:00
epriestley
672247eff3 Add aural "+" and "-" hints to unified diffs for users who use screenreaders
Summary: See PHI160 for discussion.

Test Plan:
With `?__aural__=1`, saw aural hints:

{F5229986}

Without, saw normal visual diff.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18718
2017-10-20 11:09:46 -07:00
epriestley
65f13b156f Improve "refengine" performance for testing large numbers of Mercurial branches
Summary:
See PHI158. In the RefEngine, we test if any old branch positions have been removed from the repository. This is uncommon (but not impossible) in Mercurial, and corresponds to users deleting branches in Git.

Currently, we end up running `hg log` for each position, in parallel. Because of Python's large startup overhead, this can be resource intensive for repositories with a large number of branches.

We have to do this in the general case because the caller may be asking us to resolve `tip`, `newfeature`, `tip~3`, `9`, etc. However, in the specific case where the refs are 40-digit hashes, we can bulk resolve them if they exist, like this:

```
hg log ... --rev (abcd or def0 or ab12 or ...)
```

In the general case, we could probably do less of this than we currently do (instead of testing all old heads, we could prune the list by removing commits which we know are still pointed to by current heads) but that's a slightly more involved change and the effect here is already dramatic.

Test Plan:
Verified that CPU usage drops from ~110s -> ~0.9s:

Before:

```
epriestley@orbital ~/dev/phabricator $ time ./bin/repository refs nss
Updating refs in "nss"...
Done.

real	0m14.676s
user	1m24.714s
sys	0m21.645s
```

After:

```
epriestley@orbital ~/dev/phabricator $ time ./bin/repository refs nss
Updating refs in "nss"...
Done.

real	0m0.861s
user	0m0.882s
sys	0m0.213s
```

  - Manually resolved `blue`, `tip`, `9`, etc., got expected results.
  - Tried to resolve invalid hashes, got expected result (no resolution).

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18717
2017-10-20 11:09:14 -07:00
epriestley
01b1854fb4 Fix an issue with attempting to index comments on packages
Summary: See rPcd14194a329788d5fff6365bcade278fd18f3612 for a similar change. Implement `getApplicationTransactionCommentObject()` to return `null` explicitly.

Test Plan: Ran `bin/search index --type ownerspackage`, got indexing after change.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18715
2017-10-20 09:38:45 -07:00
epriestley
1755ec2429 Show more detailed hints about draft revisions in the UI
Summary: Ref T2543. When revisions are in the draft state, tell the user what we're waiting for or why they aren't moving forward.

Test Plan: {F5228840}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18714
2017-10-20 08:40:17 -07:00
epriestley
bfabe49c5a Start revisions in "Draft" if prototypes are enabled
Summary: Ref T2543. This is a less ambitious version of the rule in D18628, which I backed off from, since I think this probably still has a fair number of loose ends to tie up.

Test Plan: Created a revision locally.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18713
2017-10-20 08:39:58 -07:00
epriestley
d36f98a15a Clarify acceptable values for --threshold in search ngrams
Summary: See D18710.

Test Plan: o_O

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18712
2017-10-17 14:32:25 -07:00
epriestley
63d1230ade Parameterize the common ngrams threshold
Summary:
Ref T13000. Since other changes have generally made the ngrams table manageable, I'm not planning to enable common ngrams by default at this time.

Instead, make the threshold configurable with "--threshold" so we can guide installs through tuning this if they want (e.g. PHI110), and tune hosted instances.

(This might eventually become automatic, but just smoothing this bit off for now feels reasonable to me.)

Test Plan: Ran with `--reset`, and with various invalid and valid `--threshold` arguments.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18710
2017-10-17 14:13:49 -07:00
epriestley
acb145b3d7 Allow duplicates and merged-in tasks to be queried with edge.search
Summary: See PHI147.

Test Plan: Called the method from the web UI, got sensible results.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18706
2017-10-13 13:12:50 -07:00
epriestley
3f53718d10 Modularize rate/connection limits in Phabricator
Summary:
Depends on D18702. Ref T13008. This replaces the old hard-coded single rate limit with multiple flexible limits, and defines two types of limits:

  - Rate: reject requests if a client has completed too many requests recently.
  - Connection: reject requests if a client has too many more connections than disconnections recently.

The connection limit adds +1 to the score for each connection, then adds -1 for each disconnection. So the overall number is how many open connections they have, at least approximately.

Supporting multiple limits will let us do limiting by Hostname and by remote address (e.g., a specific IP can't exceed a low limit, and all requests to a hostname can't exceed a higher limit).

Configuring the new limits looks something like this:

```
PhabricatorStartup::addRateLimit(new PhabricatorClientRateLimit())
  ->setLimitKey('rate')
  ->setClientKey($_SERVER['REMOTE_ADDR'])
  ->setLimit(5);

PhabricatorStartup::addRateLimit(new PhabricatorClientConnectionLimit())
  ->setLimitKey('conn')
  ->setClientKey($_SERVER['REMOTE_ADDR'])
  ->setLimit(2);
```

Test Plan:
  - Configured limits as above.
  - Made a lot of requests, got cut off by the rate limit.
  - Used `curl --limit-rate -F 'data=@the_letter_m.txt' ...` to upload files really slowly. Got cut off by the connection limit. With `enable_post_data_reading` off, this correctly killed the connections //before// the uploads finished.
  - I'll send this stuff to `secure` before production to give it more of a chance.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13008

Differential Revision: https://secure.phabricator.com/D18703
2017-10-13 13:12:05 -07:00
epriestley
9777c66576 Allow Phabricator to run with "enable_post_data_reading" disabled
Summary:
Ref T13008. Depends on D18701. The overall goal here is to make turning `enable_post_data_reading` off not break things, so we can run rate limiting checks before we read file uploads.

The biggest blocker for this is that turning it off stops `$_FILES` from coming into existence.

This //appears// to mostly work. Specifically:

  - Skip the `max_post_size` check when POST is off, since it's meaningless.
  - Don't read or scrub $_POST at startup when POST is off.
  - When we rebuild REQUEST and POST before processing requests, do multipart parsing if we need to and rebuild FILES.
  - Skip the `is_uploaded_file()` check if we built FILES ourselves.

This probably breaks a couple of small things, like maybe `__profile__` and other DarkConsole triggers over POST, and probably some other weird stuff. The parsers may also need more work than they've received so far.

I also need to verify that this actually works (i.e., lets us run code without reading the request body) but I'll include that in the change where I update the actual rate limiting.

Test Plan:
  - Disabled `enable_post_data_reading`.
  - Uploaded a file with a vanilla upload form (project profile image).
  - Uploaded a file with drag and drop.
  - Used DarkConsole.
  - Submitted comments.
  - Created a task.
  - Browsed around.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13008

Differential Revision: https://secure.phabricator.com/D18702
2017-10-13 13:11:11 -07:00
epriestley
821c7ac833 For backup persitsence, mark the "common ngrams" table as a data table, not an index table
Summary:
Ref T13000. Garbage collecting common ngrams is slow because MySQL isn't all that great at deleting rows quickly. See PHI96, where it looks like it's going to take a week to GC ngrams for a ~million objects at a relatively conservative 0.15 threshold.

In the event of a restore, we can reduce the impact by persisting this table so the ngrams just don't get built when the reindex happens.

Test Plan: Viewed schema in Config, saw common ngrams tables marked as "Data" instead of "Index".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18696
2017-10-11 11:07:43 -07:00
Dmitri Iouchtchenko
5897294fa9 Add spelling TODOs
Summary: Ref T13005. Added reminders not to copy/paste.

Test Plan: None.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13005

Differential Revision: https://secure.phabricator.com/D18695
2017-10-09 11:56:53 -07:00
Dmitri Iouchtchenko
cf3e198b9f Change 'tempate' to 'template'
Summary: Fixed typo. See D18693.

Test Plan: None.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D18694
2017-10-09 11:56:06 -07:00
Dmitri Iouchtchenko
9bd6a37055 Fix spelling
Summary: Noticed a couple of typos in the docs, and then things got out of hand.

Test Plan:
  - Stared at the words until my eyes watered and the letters began to swim on the screen.
  - Consulted a dictionary.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D18693
2017-10-09 10:48:04 -07:00
epriestley
85011a46d0 Bail out of PhabricatorRepositoryGraphCache more aggressively after cache fills
Summary:
Ref PHI109. Ref T11786. We currently test elapsed time every 64 iterations (since iterations are normally very fast), but at least one install is seeing the page timeout after 30 seconds.

One reason could be that cache fills may occur, and are likely to be much slower than normal iterations. In an extreme case, we could do 64 cache fills before checking the time. Tweak thing so that we always check the time after doing a cache fill, regardless of how many iterations have elapsed since the last attempt.

Additionally, this API method currently accepts an arbitrary number of paths, but implicitly limits each cache query to 500ms. If more than 60 paths are passed, this may exceed 30s. Only let the cache churn for a maximum of 10s across all paths.

If this is more the latter issue than the former, this might replace the GraphCache timeouts with `git` timeouts, but at least our understanding of what's going on here will improve.

Test Plan: This is difficult to test convincingly locally, since I can't reproduce the original issue. It still works after these changes, but it worked fine before these changes too.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11786

Differential Revision: https://secure.phabricator.com/D18692
2017-10-06 14:12:58 -07:00
epriestley
17e83b53d5 Add "bin/search query" for debugging query execution
Summary:
Ref T13000. Currently, queries can only be executed from the web UI, which requires logging in as a user. I really want to avoid doing that wherever we can, but being able to execute queries on an instance (and, particularly, see the ngrams and timings on the underlying lookups) would have been helpful in several cases.

Improve tooling a bit in advance of the "common ngrams" stuff going out since it seems likely that it will be useful if issues arise.

Test Plan: Ran `bin/search query --query ...`, got useful minimal output. Ran with `--trace` to get internals.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18690
2017-10-06 08:50:34 -07:00
epriestley
66df5b1493 Add a garbage collector for common ngrams
Summary:
Ref T13000. After an ngram is marked as "common", we can delete it from the storage table.

Currently, the only way to get ngrams marked as "common" is to manually run `bin/search ngrams`, so this has no impact on normal installs.

Test Plan: Ran `bin/garbage collect`, saw it start chewing through my local Maniphest ngrams table and removing common ngrams.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18687
2017-10-05 11:41:18 -07:00
epriestley
c767c971ca Add "persistence" types (data, cache, or index) to tables, and tweak what "storage dump" dumps
Summary:
Ref T13000. This marks each table as either "data" (normal data), "cache" (automatically rebuilt, no need to ever dump) or "index" (can be manually rebuilt).

By default, `bin/storage dump` dumps data and index tables, but not cache tables.

With `--no-indexes`, it dumps only data tables. Indexes can be rebuilt after a restore with `bin/search index --all ...`.

Test Plan:
  - Ran `--no-indexes` and normal dumps with `--trace`, verified that cache and index (former case) or cache only (latter case) tables were dumped with `--no-data`.
  - Verified dump has the same number of `CREATE TABLE` statements as before the changes.
  - Reviewed persistence tags in the web UI (note Ferret engine tables are "Index"):

{F5210886}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18682
2017-10-04 12:09:33 -07:00
epriestley
b9fd526250 Fix a fatal on user email settings when account.editable is disabled
Summary:
If `account.editable` is set to false, we try to add a `null` button and fatal:

> Argument 1 passed to PHUIHeaderView::addActionLink() must be an instance of PHUIButtonView, null given, called in /srv/phabricator/phabricator/src/applications/settings/panel/PhabricatorSettingsPanel.php on line 290

Instead, don't try to render `null` as a button.

Test Plan:
  - Configured `account.editable` false.
  - Viewed email address settings.
  - Before: fatal.
  - After: page works, no button is provided.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18677
2017-10-04 10:16:30 -07:00
epriestley
3e589cdd73 Add a workflow for populating (or depopulating) the common ngrams table
Summary:
Depends on D18672. Ref T13000. This does an on-demand build of the common ngrams table.

Plan here is:

  - Push to `secure`.
  - Build the common ngrams table here.
  - See if stuff breaks?

If it looks okay on this dataset, we can build out the GC support and try it in production.

Test Plan:
  - Locally, my dataset has a bunch of `bin/lipsum` tasks with similar, common words.
  - Verified that ipsum terms now skip ngrams. For "lorem ipsum" search performance actually IMPROVED by skipping the ngrams table (12s to 9s).
  - Queried for normal terms, got very fast results using the ngram table, as normal.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18673
2017-10-03 13:28:19 -07:00
epriestley
1de130c9f5 Allow the Ferret engine to remove "common" ngrams from the index
Summary:
Ref T13000. This adds support for tracking "common" ngrams, which occur in too many documents to be useful as part of the ngram index.

If an ngram is listed in the "common" table, it won't be written when indexing documents, or queried for when searching for them.

In this change, nothing actually writes to the "common" table. I'll start writing to the table in a followup change.

Specifically, I plan to do this:

  - A new GC process updates the "common" table periodically, by writing ngrams which appear in more than X% of documents to it, for some value of X, if there are at least a minimum number of documents (maybe like 4,000).
  - A new GC process deletes ngrams that have been added to the common table from the existing indexes.

Hopefully, this will pare down the ngrams index to something reasonable over time without requiring any manual tuning.

Test Plan:
  - Ran some queries and indexes.
  - Manually inserted ngrams `xxx` and `yyy` into the ngrams table, searched and indexed, saw them ignored as viable ngrams for search/index.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18672
2017-10-03 13:27:42 -07:00
epriestley
89fe84f978 Add a "/source/..." URI for Diffusion commits which redirects
Summary:
See PHI112. The install presumably wants to generate links to Diffusion commits from an external tool, but only knows the short name of the repository.

Provide a `/source/phabricator/commit/abcdef908273` URI which redirects to the canonical URI for the commit.

Test Plan:
  - Visited `/source/` URI for a commit, got a redirect.
  - Visited normal URI for a commit, got a commit page.
  - Visited `/branches/` and `/tags/` for a `/source/` repository, got proper pages.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18676
2017-10-03 13:27:03 -07:00
epriestley
f9110b87ab In "Move Tasks to Column...", show only visible columns
Summary:
See PHI94. I considered this initially but wasn't sure about it. However, PHI94 brings up the good point that we already use a similar rule in Maniphest.

For consistency, only show visible columns here too.

Test Plan: Used "Move tasks to column..." on a board with visible and hidden columns, only saw visbile columns offered in the dropdown.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18668
2017-10-02 11:41:02 -07:00
epriestley
cd14194a32 Fix transaction queries using withComments() for transactions with no comments
Summary:
See <https://discourse.phabricator-community.org/t/daemons-tasks-crashing-in-a-loop-during-reindex/506/1>. Some object types (for example, Passphrase Credentials) support indexing but not commenting.

Make `withComments(...)` work properly if the transaction type does not support comments.

Test Plan:
Indexed a credential (no comments) and a revision (comments) with `bin/search index --trace ...`.

Before, credential fataled.

After, credetial succeeds, and skips the transaction query.

Before and after, the revision queries the transaction table.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18667
2017-10-02 09:09:53 -07:00
epriestley
5c73c169fd Rough cut of "Move tasks to column..."
Summary:
Ref T5523. See PHI50. See PHI40. This isn't perfect, but should improve things.

Add a "Move tasks to column..." action to workboards which moves all visible tasks in a column to another column, either on the same board or on a different board.

This is a two-step process so that I don't have to write Javascript, and because I'm not 100% sure this is what users actually want/need. If it sticks, the UI could be refined later.

  - The first dialog asks you to choose a project, defaulting ot the current project.
  - The second dialog asks you to choose a column on that project's board.

Test Plan:
  - Moved tasks on the same board.
  - Moved tasks to a different board.
  - Tried to move tasks to the starting column, got a sensible error.
  - Tried to move tasks to no project, got a sensible error.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T5523

Differential Revision: https://secure.phabricator.com/D18665
2017-09-29 17:38:52 -07:00
epriestley
fe646ec328 Mark Owners package reviewers which own nothing in the current diff
Summary:
Ref PHI91. When Owners (or Herald, or manual user action) adds package reviewers to a revision, later updates to the revision make some of them less relevant or irrelevant.

Provide a hint when a package reviewer doesn't own any of the paths that a diff changes. Humans can then decide if the reviewer is obsolete/irrelevant or not.

This is a rough cut to get the feature working, design could probably use some tweaking if it sticks.

Test Plan: {F5204309}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: jboning

Differential Revision: https://secure.phabricator.com/D18663
2017-09-29 15:06:00 -07:00
epriestley
a39f5e1113 Correct bad context path when doing pattern search inside a repository
Summary:
Ref PHI101. It looks like this was maybe copy/pasted by mistake in recent design refactoring.

We need to pass the full path, not the `basename()` of the path, to the search form.

Test Plan: Searched inside `scripts/test/`, found results inside `scripts/test/`.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18664
2017-09-29 14:51:49 -07:00
epriestley
a3a6c4ed2e Fix fatal when searching for "r matey prepare to be boarded"
Summary:
See <https://discourse.phabricator-community.org/t/unrecoverable-fatal-error-on-repository-search-in-top-search-bar/503/2>.

The Ferret engine replaced `withNameContains()`, but I missed this obscure callsite.

Test Plan:
  - Searched for `r matey prepare to be boarded`.
  - Before: fatal.
  - After: no fatal.
  - Also searched for `r <actual repository name>`, got repository.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18661
2017-09-29 09:45:39 -07:00
epriestley
b75a4151c8 Allow the fulltext index to select only transactions with comments
Summary:
Ref T12997. Although we can't query by transaction type (since we can't easily enumerate all possible types which may have comments -- inline types may also have comments), we //can// just check if there's a comment row or not.

This reduces the amount of garbage we need to load to rebuild indexes for unusual objects with hundreds and hundreds of mentions.

Test Plan:
  - Used batch editor to mention a task 700 times.
  - Indexed it before and after this change, saw index time drop from {nav 1600ms > 160ms}.
  - Made some new comments on it, verified that they still indexed/queried properly.
  - Browsed around, made normal transactions, made inline comments.
  - Added a unique word to an inline comment, indexed revision, searched for word, found revision.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12997

Differential Revision: https://secure.phabricator.com/D18660
2017-09-28 12:55:23 -07:00
epriestley
9875af739f When we purge the request cache, also force PHP to collect cycles
Summary:
Ref T12997. See that task for more details. Briefly, an unusual dataset (where commits are mentioned hundreds of times by other commits) is causing some weird memory behavior in the daemons.

Forcing PHP to GC cycles explicitly after each task completes seems to help with this, by cleaning up some of the memory between tasks. A more thorough fix might be to untangle the `$xactions` structure, but that's significantly more involved.

Test Plan:
  - Did this locally in a controlled environment, saw an immediate collection of a 500MB `$xactions` cycle.
  - Put a similar change in production, memory usage seemed less to improve. It's hard to tell for sure that this does anything, but it shouldn't hurt.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12997

Differential Revision: https://secure.phabricator.com/D18657
2017-09-28 12:37:22 -07:00
epriestley
086a125ad5 Improve performance of Ferret engine ngram extraction, particularly for large input strings
Summary:
See PHI87. Ref T12974. The `array_slice()` method of splitting the string apart can perform poorly for large input strings. I think this is mostly just the large number of calls plus building and returning an array being not entirely trivial.

We can just use `substr()` instead, as long as we're a little bit careful about keeping track of where we're slicing the string if it has UTF8 characters.

Test Plan:
  - Created a task with a single, unbroken blob of base64 encoded data as the description, roughly 100KB long.
  - Saw indexing performance improve from ~6s to ~1.5s after patch.
  - Before: https://secure.phabricator.com/xhprof/profile/PHID-FILE-nrxs4lwdvupbve5lhl6u/
  - After: https://secure.phabricator.com/xhprof/profile/PHID-FILE-6vs2akgjj5nbqt7yo7ul/

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12974

Differential Revision: https://secure.phabricator.com/D18649
2017-09-27 10:41:39 -07:00
epriestley
a1d9a2389d Improve Ferret engine indexing performance for large blocks of text
Summary:
See PHI87. Ref T12974. Currently, we do a lot more work here than we need to: we call `phutil_utf8_strtolower()` on each token, but can do it once at the beginning on the whole block.

Additionally, since ngrams don't care about order, we only need to convert unique tokens into ngrams. This saves us some `phutil_utf8v()`. These calls can be slow for large inputs.

Test Plan:
  - Created a ~4MB task description.
  - Ran `bin/search index Txxx --profile ...` to profile indexing performance before and after the change.
  - Saw total runtime drop form 38s to 9s.
  - Before: <https://secure.phabricator.com/xhprof/profile/PHID-FILE-wiht5d7lkyazaywwxovw/>
  - After: <https://secure.phabricator.com/xhprof/profile/PHID-FILE-efxv56q2hulr6kjrxbx6/>

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12974

Differential Revision: https://secure.phabricator.com/D18647
2017-09-27 08:15:40 -07:00
epriestley
1ac52c09e7 Improve search highlighting for CJK and substring queries
Summary:
Fixes T12995. Currently, the result highlighter (which shows //where// terms matched) only works in "term" mode, not in "substring" mode.

Provide better feedback and behvaior:

  - When a term is a substring term, color it a little differently and add a tooltip. (This is partly to make it easier to debug/diagnose things, probably not enormously valuable to users.)
  - When a term is a substring term, highlight it anywhere in the results.

Test Plan:
Queried for latin and CJK terms.

Here is CJK being highlighted:

{F5192195}

Here is substring vs non-substring implicit behavior:

{F5192196}

Here's ONLY terms being highlighted:

{F5192198}

Here's terms and substrings, since the query now has a substring:

{F5192201}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12995

Differential Revision: https://secure.phabricator.com/D18635
2017-09-22 11:34:46 -07:00
epriestley
36df39761e Create revisions into "Draft", publish them when builds finish
Summary:
Ref T2543. This doesn't stand alone since mail still goes out normally, but gets this piece working: new revisions start as "Draft", then after updates if there are no builds they go into "Needs Review".

This should work in general because builds update revisions when they complete, to publish a "Harbormaster finished build yada yada" transaction. So either we'll un-draft immediately, or un-draft after the last build finishes.

I'll hold this until the mail and some other stuff (like UI hints) are in slightly better shape since I think it's probably too rough on its own.

Test Plan: Created revisions locally, saw them un-draft after builds.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18628
2017-09-21 07:21:21 -07:00
epriestley
fca553f142 Prepare revision mail for the "Draft" status
Summary:
Ref T2543. Currently, we always do some special things when a revision is created, mostly adding more stuff to the mail.

With drafts, we want to suppress initial mail and send this big, rich mail only when the revision actually moves out of "draft".

Prepare the code for this, with the actual methods hard-coded to the current behavior. This will probably take some tweaking but I think I got most of it.

Test Plan: Banged around in Differential so it sent some mail, saw normal mail without anything new.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18627
2017-09-21 07:21:07 -07:00
epriestley
c7af663523 Align most revision actions to the new "Draft" state
Summary:
Ref T2543. Most actions are not available for drafts.

Authors can "Request Review" (move out of draft to become a normal revision) or "Abandon".

Non-authors can't do anything (maybe we'll let them do something later -- like "Commandeer"? -- if there's a good reason).

Test Plan: Viewed a draft revision as an author and non-author, saw fewer actions available.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18626
2017-09-21 07:20:48 -07:00
epriestley
5112dac491 Update an old SSH redirect URI when editing a bot's SSH keys
Summary: See PHI79. When you edit another user's SSH keys (normally, for a bot account) we currently redirect you to an older URI.

Test Plan:
  - Viewed a bot's profile page.
  - Clicked "Edit Settings" on the Manage page.
  - Went to "SSH Keys".
  - Uploaded an SSH key.
  - Before: redirected to a 404 after finishing the workflow.
  - After: redirected to the same page after the workflow.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18633
2017-09-20 14:46:31 -07:00
epriestley
3fbad684c1 More completely explain why we're refusing to send reset mail to an unverified address
Summary:
See PHI78. The user was getting this message and (reasonably) interpreted it to mean "reset mail can never be sent to unverified addresses".

Reword it to be more clear, albeit an entire paragraph long. I don't really have a good solution in these cases where we'd need a whole page to explain what's happening (this, plus "we can't tell you which address you should use because an attacker could get information if we did" and "this rule defuses the risk that an opportunistic attacker may try to compromise your account after you add an email you don't own by mistake"). We could write it up separately and link to it, but I feel like that stuff tends to get out of date.

Just land somewhere in the middle.

Test Plan: {F5189105}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18630
2017-09-20 10:46:22 -07:00
epriestley
03e5d69817 Fix an error in Diffusion when the Owners application is uninstalled
Summary:
See <https://discourse.phabricator-community.org/t/undefined-view-when-owners-is-uninstalled/451>.

When Owners is not installed, Diffusion can fatal with a bad `$view`.

Test Plan:
  - Uninstall Owners.
  - View the content of any file in Diffusion.
  - Before: fatal on `$view` undefined.
  - After: Valid page with no owners information.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18629
2017-09-19 09:42:42 -07:00
epriestley
23867c1487 Add a "Draft" state for revisions, and action bucket support
Summary:
Ref T2543. There's no way to put revisions into this state yet, but start adding support for when there is.

Adds the status constant, plus support for bucketing them.

Test Plan:
  - Manually put a revision in "Draft" state by updating the database directly.
  - Verified my drafts showed up in a "Drafts" section on the bucket view.
  - Verified others' drafts did not appear on the action bucket view.
  - Viewed revisions, queried for "Draft" revisions, etc (stuff we get for free).

{F5186781}

{F5186782}

{F5186783}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18625
2017-09-18 14:01:00 -07:00
epriestley
156adccef0 Fix an issue where "bin/differential migrate-hunk" could decompress data
Summary:
Fixes T12986. I caught this bug in the changes from D18584: when we moved a large hunk to file storage, we would decompress it but keep the "deflated" flag. This could cause confusion when loading it later. I missed this in testing since I wasn't exhaustive enough in checking hunks and didn't run into a compressed one.

Instead of compressing on `save()`, compress during the normal workflow.

We currently never advise users to run this workflow so I didn't bother trying to clean up possible existing migrations.

Test Plan:
  - Ran `bin/differential migrate-hunk` on compressed hunks, moving them to and from file storage. Saw them work correctly and remain compressed.
  - Created new small (uncompressed) and large (compressed) hunks, verified they work properly and get compressed (if applicable).
  - Used `bin/cache purge --caches changeset` to clear changeset caches and make sure the actual table was being hit.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12986

Differential Revision: https://secure.phabricator.com/D18624
2017-09-18 14:00:41 -07:00
epriestley
51b810b0eb Fix "Author's projects" Herald rules for revisions and diffs
Summary:
See PHI71. These didn't get properly updated when we wrote Subprojects and Milestones, and should use materialized members, not raw members. Swap the query so projects you are an indirect member of (e.g., milestones you are a member of the parent for, and parent projects you are a member of a subproject of) are included in the result list.

Also fix a bad typeahead datasource.

Test Plan:
  - Ran a dry run with the test console, saw project PHIDs for milestones and parent projects in the raw field value.
  - Tried to set "Author's projects" to a user, no longer could.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18619
2017-09-15 17:59:49 -07:00
epriestley
b352cacdd9 Swap "-R" and "serve" argument order for Mercurial
Summary: See <https://discourse.phabricator-community.org/t/unable-to-use-current-mercurial-on-debian-stretch/391/13>. I missed that `-R` is order-sensitive.

Test Plan: Verified both orders work on 3.5.2.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18616
2017-09-15 13:07:43 -07:00
epriestley
49b7181780 Update utility "bin/repository parents" workflow to work with RefPosition
Summary:
Ref T11823. I think this is the last callsite which relies on the old data format: `bin/repository parents` rebuilds a cache which we don't currently use very heavily.

Update it to work with the new data.

Test Plan: Ran `bin/repository parents <repository> --trace`, saw successful script execution and reasonable-looking output.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11823

Differential Revision: https://secure.phabricator.com/D18615
2017-09-15 10:21:51 -07:00
epriestley
8982e3e52d Update major RefCursor callsites to work properly with RefPosition
Summary:
Ref T11823. This is the meaty part of the change, and updates `RefEngine` to use separate RefCursor (for names) and RefPosition (for actual commit positions) tables.

I'll hold this whole series until after the release cut so it has some time to bake on `secure` to look for issues. It's also not a huge problem if there are bugs here since these tables are just caches anyway, although they do feed into some other things, and obviously it's never good to have bugs.

Test Plan:
  - This logic can be invoked directly with `bin/repository refs <repository> --trace --verbose`.
  - Ran that on unchanged repositories, new branches, removed branches, and modified branches. Saw appropriate output and cursor positions.
  - Ran on a mercurial repository to test the close/open logic, saw it correct open/closed state of incorrect positions.
  - Browed around Diffusion in various repositories.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11823

Differential Revision: https://secure.phabricator.com/D18614
2017-09-15 10:21:32 -07:00
epriestley
5cf62f86d7 Remove obsolete columns from RefCursor table
Summary:
Ref T11823. This change isn't standalone, but prepares for the more involved code change by dropping obsolete columns from the RefCursor table and adding the unique key we need to prevent the ambiguous/duplicate refs issue.

This data was moved to the RefPosition table in D18612.

Test Plan: Ran storage upgrade. See next revision for more substantial testing of this change series.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11823

Differential Revision: https://secure.phabricator.com/D18613
2017-09-15 10:21:12 -07:00
epriestley
9d5a2b3b4f Add a RefPosition table to hold branch/tag positions once the RefCursor table is split
Summary:
Ref T11823. Currently, we have a "RefCursor" table which stores rows like `<branch or tag name, commit it is pointing at>` with some more data.

Because Mercurial can have a single branch pointing at several different places, this table must allow multiple rows with the same branch or tag name.

Among other things, this means there isn't a single PHID which can be used to identify a branch name in a stable way. However, we have several UIs where we want to be able to do this.

Some specific examples where we run into trouble: in Mercurial, if there are 5 heads for "default", that means there are 5 phids. And currently, if someone deletes a branch, we lose the PHID for it. Instead, we'd rather retain it so the whole world doesn't break if you accidentally delete a branch and then fix it a little later.

(I'll likely hold this until the rest of the logic is fleshed out a little more in followup changes.)

Test Plan: Ran `bin/storage upgrade`, saw the table get created without warnings.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11823

Differential Revision: https://secure.phabricator.com/D18602
2017-09-15 10:19:17 -07:00
epriestley
bd923d1ce0 Provide an explicit "-R" flag to "hg serve"
Summary:
See <https://discourse.phabricator-community.org/t/unable-to-use-current-mercurial-on-debian-stretch/391>.

The Mercurial commit is helpful in particular: <https://www.mercurial-scm.org/repo/hg/rev/77eaf9539499>

We weren't vulnerable to the security issue (users can not control any part of the command) but pass the working directory explicitly to get past the new safety check.

I left `setCWD()` in place (a few lines below) just because it can't hurt, and in some other contexts it sometimes matter (for example, if commit hooks execute, they might inherit the parent CWD here or in other VCSes).

Test Plan:
  - Cloned from a Mercurial repo locally over HTTP.
  - Verified that SSH cloning already uses `-R` (it does, see `DiffusionMercurialServeSSHWorkflow`).
  - Did not actually upgrade to Mercurial 4.0/4.1.3 to completely verify this, but a user in the Discourse thread asserted that a substantially similar fix worked correctly.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18611
2017-09-15 08:57:11 -07:00
epriestley
5ae3af6691 Fix an outdated HTML anchor link in Diffusion table of contents
Summary:
See <https://discourse.phabricator-community.org/t/navigating-to-changed-files-in-diffusion-does-not-work-anymore/433>.

In D18465, I updated these but this hard-coded the anchor for some reason (???) and I missed it while `grep`-ing.

Test Plan: Viewed a commit (`/rXYZaaaa`) and clicked a file link in the table of contents. Got modern `#change-...` anchor and navigation into the document.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18609
2017-09-15 08:37:36 -07:00
Austin McKinley
c71cb944a4 Add edit methods for Almanac services and devices
Summary: See T12414. This just gets started; we still need edit endpoints for network interfaces and bindings.

Test Plan: Created some devices/services from the conduit UI.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18605
2017-09-14 14:32:58 -07:00
epriestley
939008e64c Correct an issue where Maniphest's awful legacy "reports" UI was extra broken on merges
Summary:
See PHI66. See that issue for context. This UI is bad broken legacy junk, but was especially broken when reporting merges.

These do not currently generate a "status" transaction, so they were never counted as task closures. Pretend they're normal closures.

This is still wrong, but should be much closer to the real numbers. Specifically, if you merge a closed task into another task, it will incorrectly be counted as an extra close. This could result in negative tasks, but the numbers should be much closer to reality than they are today even so.

The "Facts" application (T1562) is the real pathway forward here in the longer term.

Test Plan:
  - Moved my `maniphest_transactions` table aside with `RENAME TABLE ...`.
  - Created a new empty table with `CREATE TABLE ... LIKE ...`.
  - Reloaded reports UI, saw empty chart.
  - Created, closed, and reopened tasks while reloading the chart, saw accurate reporting.
  - Merged an open task into another task, saw bad reporting.
  - Applied patch, saw the right chart again.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18601
2017-09-14 10:09:46 -07:00
epriestley
c310f08b7a Work around workflow blocking error with duplicate "master" refs in "Land Revision"
Summary:
Ref T11823. See PHI68. T11823 has a full description of this issue and a plan to fix it, but the full plan is relatively complicated.

Until that can happen, provide a workaround for the biggest immediate issue, where multiple copies of a ref cursor can cause `executeOne()` to throw, since it expects a single result. In practice, these copies are always identical so we can just pick the first one.

This will get cleaned up once T11823 is fixed properly.

Test Plan:
Forced the table into a duplicate/ambiguous state, reproduced a similar-looking error:

{F5180999}

Applied the patch, got the "Land" to work as expected:

{F5181000}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11823

Differential Revision: https://secure.phabricator.com/D18599
2017-09-13 16:03:10 -07:00
epriestley
6fb3f857fb Stop the bleeding caused by attaching enormous patches to revision mail
Summary:
Ref T12033. This is a very narrow fix for this issue, but it should fix the major error: don't attach patches if they're bigger than the mail body limit (by default, 512KB).

Specifically, the logs from an install in T12033 show a 112MB patch being attached, and that's the biggest practical problem here.

I'll follow up on the tasks with more nuanced future work.

Test Plan: Enabled `differential.attach-patches`, saw a patch attached to email. Set the byte limit very low, saw patches get thrown away.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12033

Differential Revision: https://secure.phabricator.com/D18598
2017-09-13 15:32:55 -07:00
epriestley
29f625ef68 Make "No Notifications" setting less broad, and fix a bug with default display behavior
Summary:
Fixes T12979. In D18457, we added a "No Notifications" setting to let users disable the blue and yellow pop-up notifications that alert you when an object has been updated, since some users found them distracting.

However, the change made "do nothing" the default, so all other `JX.Notification` callsites -- which never pass a preference -- were effectively turned off no matter what your setting was set to. This includes the "Read-Only" mode warning (grey), the "High Security" mode warning (purple), the "timezone" warning, and a few others.

Tweak things a little bit so the setting applies to ONLY blue and yellow ("object you're following was updated" / "this object was updated") notifications, not other types of popup notifications.

Test Plan:
  - With notifications on in settings, got blue notifications and "Read-only".
  - With notifications off in settings, got "Read-only" but no blue notifications.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12979

Differential Revision: https://secure.phabricator.com/D18600
2017-09-13 15:32:46 -07:00
epriestley
da0a08a7e1 Make "mysql" mean "Ferret engine" in Fulltext search
Summary: Ref T12819. Swaps constants so existing configurations that use a "mysql" engine now use the Ferret engine, not an InnoDB/MyISAM FULLTEXT engine.

Test Plan: Swapped my local config back to "mysql" (the default), saw Ferret engine results in the UI.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18590
2017-09-11 18:05:12 -07:00
epriestley
39b74572e6 Return fulltext tokens from the Ferret fulltext engine
Summary:
Ref T12819. These render the little "Searched For: X, Y, U V" hint about how something was parsed.

(This might get a "substring" color or "title only" color or something in the future.)

Test Plan: {F5178807}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18589
2017-09-11 18:04:56 -07:00
epriestley
dd3f05ec25 Remove "Name Contains" query constraint from Diffusion for Repositories
Summary:
Ref T12819. Obsoleted by the Ferret engine "Query" field.

This is a compatibility break, I'll note it in the changelog.

Test Plan: Searched for repositories by name with "Query" instead of "Name Contains".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18588
2017-09-11 18:04:39 -07:00
epriestley
6edf98eb3b Unprototype the Ferret UI fields
Summary:
Ref T12819. Show the new Ferret engine fields (and enable the indexer) unconditionally.

Also pull them to the top since they're fairly general-purpose and appear more broadly now, and also they actually work correctly (WOW).

Some redundant fields (like "Name Contains" in Repositories and Owners) could probably be removed now, I may clean those up in a followup.

Test Plan: Browsed around, saw Ferret fields in UI without "(Prototype)" suffix.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18587
2017-09-11 18:04:25 -07:00
epriestley
495ab7363b Remove "Contains Words" constraint from Maniphest
Summary:
Ref T12819. Obsoleted by the Ferret engine, which is unprototyping shortly.

This breaks compatibility in two ways:

  - `maniphest.query` no longer supports "fullText" (now throws an explicit exception).
  - Existing saved searches with a "Contains Words" constraint will no longer have that constraint.

It seems unlikely (?) that either of these are seeing too much use, and they should be easy to fix. I'll note them in the changelog.

Test Plan: Viewed Maniphest, no more "Contains Words" field. Called `maniphest.query` with "fullText", got explicit exception.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18586
2017-09-11 18:04:12 -07:00
epriestley
d15fb20fe6 Support storage of Differential hunk data in Files
Summary:
Ref T12932. For long-lived installs, one of the largest tables tends to be the hunk data table. Although it doesn't grow tremendously fast, it's also well suited to storage in Files instead of the database (infrequent access, relatively large blobs of data, mostly one-at-a-time access), and earlier work anticipated eventually adding support for Files storage.

Make Files storage work, and provide `bin/differential migrate-hunk` to manually test/migrate hunks. This is currently the only way hunks get moved to file storage, but I expect to add a GC step which moves them to File storage after 30 days shortly.

The immediate motivation for this is to relieve storage pressure on db001/db002 so we have more headroom for deploying the Ferret engine and its larger indexes (see also T12819).

Test Plan:
  - Used `bin/differential migrate-hunk` to move a hunk to and from file storage, verified it survived intact.
  - Downloaded the actual stored file, sanity-checked it. Verified permissions.
  - Destroyed a diff with `bin/remove destroy`, saw the hunk and file storage destroyed.
  - Verified that going from file -> text destroys the old file properly with `migrate-hunk --trace ...`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12932

Differential Revision: https://secure.phabricator.com/D18584
2017-09-11 16:09:02 -07:00
epriestley
d67cc8e5c5 Remove some redundant information from the Ferret engine index
Summary:
Ref T12819. The "full" field has all other fields, and the "core" field has "title" and "body". Due to the way the "full" and "core" fields were being built, the "core" field also got included in the "full" field, so the "full" field has two copies of the title, two copies of the body, and then one copy of everything else.

Put only one copy of each distinct thing in each "full" and "core". Also, simplify the logic a little bit so we build these virtual fields in a more consistent way.

Test Plan: Ran `bin/search index` and looked at the fields in the database, saw less redundant information.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18580
2017-09-08 09:40:12 -07:00
epriestley
7ea6de6e9c Split Ferret engine strings for tokenization on any sequence of whitespace
Summary:
Ref T12819. Currently, strings are split only on spaces, but newlines (and, if they exist, tabs) should also split strings.

Without this, we can fail to get the proper term boundary tokens for words which begin at the start of a line or end at the end of a line.

Test Plan: Reindexed a document with "xyz\nabc", saw `"yz "` and `" ab"` term boundary tokens generate properly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18579
2017-09-08 09:39:57 -07:00
Chad Little
a46a9ff165 Update VCS password UI
Summary: Miss this with earlier pass, updates the VCS password page.

Test Plan: Try to set a vcs password

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18574
2017-09-07 15:50:05 -07:00
Chad Little
f1193afa94 Update passphrase edit UI
Summary: Updates creating credentials

Test Plan: create some notes

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18570
2017-09-07 14:13:40 -07:00
epriestley
b1b638bd14 Support the Ferret engine in Diffusion
Summary: Ref T12819. More ferret engine support.

Test Plan: Indexed and searched commits and repositories.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18572
2017-09-07 13:41:04 -07:00
epriestley
d8132db75b Support Ferret engine in Pholio
Summary: Ref T12819. Support for Pholio.

Test Plan: Indexed and searched mocks.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18569
2017-09-07 13:25:29 -07:00
epriestley
e0f3de9c64 Support Ferret engine in Calendar
Summary: Ref T12819. Adds ferret engine support for Calendar events.

Test Plan: Indexed and queried calendar events.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18568
2017-09-07 13:25:12 -07:00
epriestley
a25bbc1dca Support Ferret engine in Phriction
Summary: Ref T12819. Adds Ferret engine support.

Test Plan: Indexed and searched for documents.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18567
2017-09-07 13:24:40 -07:00
epriestley
184f201ce2 Support Ferret engine in Projects
Summary: Ref T12819. Adds support for projects.

Test Plan: Indexed and searched for projects.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18566
2017-09-07 13:24:23 -07:00
epriestley
b1703c8801 Support Ferret engine in Phame
Summary: Ref T12819. Mostly straightforward, with a couple of minor query modernization things.

Test Plan: Indexed and searched for posts and blogs.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18565
2017-09-07 13:24:07 -07:00
epriestley
c9152b586b Support Ferret engine in Owners
Summary: Ref T12819. Same deal as before, but smaller diffs after D18559.

Test Plan: Indexed and searched for packages.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18564
2017-09-07 13:23:46 -07:00
epriestley
2218caee0f Reduce the amount of boilerplate that implementing FerretInterface requires
Summary:
See brief discussion in D18554. All the index tables are the same for every application (and, at this point, seem unlikely to change) and we never actually pass these objects around (they're only used internally).

In some other cases (like Transactions) not every application has the same tables (for example, Differential has extra field for inline comments), and/or we pass the objects around (lots of stuff uses `$xactions` directly).

However, in this case, and in Edges, we don't interact with any representation of the database state directly in much of the code, and it doesn't change from application to application.

Just automatically define document, field, and ngram tables for anything which implements `FerretInterface`. This makes the query and index logic a tiny bit messier but lets us delete a ton of boilerplate classes.

Test Plan: Indexed objects, searched for objects. Same results as before with much less code. Ran `bin/storage upgrade`, got a clean bill of health.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18559
2017-09-07 13:23:31 -07:00
epriestley
2020c1e7bd Support Ferret engine for Passphrase credentials
Summary: Ref T12819. Adds Ferret support to Passphrase.

Test Plan: Indexed credentials, searched for credentials.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18556
2017-09-07 13:23:13 -07:00
epriestley
f23717b416 Support Ferret engine in Fund initiatives
Summary: Ref T12819. Adds Ferret engine support to initiatives.

Test Plan: Indexed and searched for initiatives.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18555
2017-09-07 13:22:57 -07:00
epriestley
cf0bc32e18 Lightly modernize FundInitiativeSearchEngine
Summary: Ref T12819. Prepares for Ferret engine support.

Test Plan: Queried for various initiatives.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18554
2017-09-07 13:22:43 -07:00
epriestley
60deec36d8 Lightly modernize FundInitiativeQuery
Summary: Ref T12819. Prepares Fund to move to Ferret.

Test Plan: Searched for initiatives in Fund.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18553
2017-09-07 13:22:27 -07:00
epriestley
3ff9d4a4ca Support Ferret engine for searching users
Summary:
Ref T12819. Adds support for indexing user accounts so they appear in global fulltext results.

Also, always rank users ahead of other results.

Test Plan: Indexed users. Searched for a user, got that user.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18552
2017-09-07 13:22:12 -07:00
epriestley
a2a2b3f7f4 Sort global fulltext results by overall relevance
Summary:
Ref T12819. Currently, under the Ferret engine, we query each application's index separately and then aggregate the results.

At the moment, results are aggregated by type first, then by actual rank. For example, all the revisions appear first, then all the tasks.

Instead, surface the internal ranking data from the underlying query and sort by it.

Test Plan: Searched for "A B" with a task named "A B" and a revision named "A". Saw task first. Broadly, saw mixed task and revision order in result sets.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18551
2017-09-07 13:21:58 -07:00
epriestley
8059db894d Use the Ferret engine fulltext document table to drive auxiliary fulltext constraints
Summary:
Ref T12819. I started trying to get individual engines to drive these constraints (e.g., `ManiphestTaskQuery` can do most of the work) but this is a big pain, especially since most engines don't support "any owner" or "no owner", and not everything has an owner, and so on and so on. Going down this path would have meant a huge pile of stub functions everywhere, I think.

Instead, drive these through the main engine using the fulltext document table, which already has everything we need to apply these constraints in a uniform way.

Also tweak some parts of query construction and result ordering.

Test Plan: Searched for documents by author, owner, unowned, any owner, tags, subscribers, fulltext in global search. Got sensible results without any application-specific code.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18550
2017-09-07 13:21:42 -07:00
Chad Little
3720b28c6c Update slowvote for new edit UI
Summary: Updates UI

Test Plan: open new poll

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18560
2017-09-07 12:51:59 -07:00
Chad Little
98185730df Update Phlux edit UI
Summary: Updates

Test Plan: Phlux edit page

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18561
2017-09-07 12:47:36 -07:00
Chad Little
d6bb0d1bfa Update people edit pages UI
Summary: Updates and clarifies UI

Test Plan: New peoples, new bots, new mailing list

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18562
2017-09-07 12:47:24 -07:00
Chad Little
a5232e015a Update herald edit for new UI
Summary: Updates herald update, create

Test Plan: Create some rulez

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18563
2017-09-07 12:46:32 -07:00
Chad Little
d3db3fff59 Update file edit UI
Summary: New white box

Test Plan: /file/upload/

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18557
2017-09-07 11:35:40 -07:00
Chad Little
97f0103a80 Update Spaces for new edit UI
Summary: New edit ui

Test Plan: create a space

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18558
2017-09-07 11:33:59 -07:00
Chad Little
3b2accee7c Add a border on diffusion uri page
Summary: This should have a border

Test Plan: Reload page

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18549
2017-09-06 20:44:37 +00:00
epriestley
4ea677ba97 Skeleton support for running global fulltext queries via the Ferret engine
Summary:
Ref T12819. Provides a Ferret-engine-based fulltext engine to ultimately replace the InnoDB fulltext engine.

This is still pretty basic (hard-coded and buggy) but technically sort of works.

To activate this, you must explicitly configure it, so it isn't visible to users yet.

Test Plan: Searched for objects with global fulltext search, got a mixture of matching revisions and tasks back.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18548
2017-09-06 13:15:36 -07:00
epriestley
551c62b91a Support Ferret engine queries in ApplicationSearch via extension instead of hard-code
Summary: Ref T12819. Uses an extension rather than hard-coding support into Maniphest.

Test Plan: Saw "Query" field appear in Differential, which also implements the interface and has support. Used field in both applications.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18547
2017-09-06 13:15:10 -07:00
epriestley
395a2ed6d1 Add an "only()" edge logic constraint, meaning "only the other constraints, exactly"
Summary:
See PHI57. For example, a query for "ios, only()" finds tags tasked with iOS, exactly, and no other tags.

I called this "only()" instead of "exact()" because we use the term/function "Exact" elsewhere with a different meaning, e.g. in Differential.

Test Plan:
Basic query for a tag:

{F5168857}

Same query with "only", finds tasks tagged with only that tag:

{F5168858}

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18543
2017-09-06 12:16:06 -07:00
Chad Little
2abbb59cb4 Update Phriction Edit page to new UI
Summary: Updates document edit page

Test Plan: review

Reviewers: epriestley, amckinley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18546
2017-09-06 12:11:53 -07:00
epriestley
faca1deea5 Remove the fulltext "reconstructDocument()" method
Summary: Ref T12819. This was originally intended for debugging, but never actually used and not clearly useful. There are no callers and it probably does not work. Just get rid of it.

Test Plan: Grepped for callers; none exist.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18544
2017-09-06 11:48:35 -07:00
Chad Little
571eb39bbc Update drydock console, edit pages to new UI
Summary: Updates Drydock for the new UI

Test Plan: Check console, edit pages

Reviewers: epriestley, amckinley

Reviewed By: epriestley

Spies: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D18545
2017-09-06 11:28:40 -07:00
Chad Little
27ebd8a33b Update Pholio Edit pages to new UI
Summary: Updates the Pholio edit pages

Test Plan: Create mock, Edit mock

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18530
2017-09-06 11:11:20 -07:00
epriestley
e91d72fefb Un-hide the "X added reviewers: ..." transactions in revision creation mail
Summary:
Fixes T12118. See PHI54. This adds a special case for the initial "reviewers" transactions, similar to the existing special case for "projects" transactions.

Although these transactions are redudnant in the web view since you can see the information clearly on the page, they're more reasonably useful in mail.

Test Plan: {F5168838}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12118

Differential Revision: https://secure.phabricator.com/D18542
2017-09-06 10:23:27 -07:00
Chad Little
818b90cf12 Update Create Diff page for new Edit UI
Summary: Create a diff page, new UI

Test Plan: Create a diff from page

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18529
2017-09-06 10:14:58 -07:00
Chad Little
20584a39d9 Update Dashboards for new Edit UI
Summary: Updates Dashboard create/edit pages

Test Plan: Create a Dashboard, edit a dashboard

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18528
2017-09-05 20:17:18 -07:00
Chad Little
a903388d4f Update EditEngine pages to take a page header separate
Summary: This simplifies EditEngine pages in general by removing the dual header, and extending to allow setting of a custom PHUIHeaderView if needed (like settings).

Test Plan:
Review all settings pages, review task, project pages. This should all be fine, but is a big change maybe some layouts I'm not considering. Tested these all mobile, destkop as well.

{F5166181}

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18527
2017-09-05 20:07:11 -07:00
Chad Little
6e25d4c67b Update Settings for WHITE_CONFIG style boxes
Summary: Updates settings panel UI for new white box, cleans up other various UI nitpicks.

Test Plan: Click through each setting that had a local setting page. Edit Engine pages will follow up on another diff.

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18526
2017-09-05 19:42:34 -07:00
Chad Little
fc893658b8 Update menu item names for Applications -> Favorites
Summary: Adds a `MenuName` method to applications that `ProfileMenuItem` uses instead of the application name if set. This improves the home/menu/new user experience at little cost. Also renamed the label from Applications to Favorites, since this menu gets altered to provide more than just applications. This also allows instances to set back to Maniphest if they so choose. Overall I think this direction resolves 95% of my concerns, with maybe a small potential downside which I don't really anticipate. We already name Dashboard panels by their object, and that hasn't really caused confusion. I think these links are similar. I click 'Tasks' and get presented a list of my tasks from Maniphest.

Test Plan: Review each of the name changes as a default new install and a modified install.

Reviewers: epriestley, amckinley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18524
2017-09-05 19:05:03 -07:00
Chad Little
e1fd74ddb5 Update Repository Management pages to new fixed UI
Summary: Simplifies the Repository Management pages to the new fixed column layout. I've also moved "Status" into the Basics page, which feels better, and moved "Documentation" as a nav item to a button in the header. This removed "action list" and "curtain view" from the management panels and uses the new bits from Config/Phacility. Undecided if the icons should stay or go for the nav. Left them in for Diffusion. I want to update the EditEngine pages to display in this UI and not leave the portal, but I haven't dug into that this page. I'm a bit worried it will not easily be possible.

Test Plan:
Generate a svn, git, hg repository, test each of the new pages and each of the new buttons. Activate, deactivate, etc.

{F5164674}

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18523
2017-09-05 19:01:27 -07:00
epriestley
f40f3ca74c Add Ferret engine index support to Differential
Summary: Ref T12819. Adds storage and indexing for the Ferret engine to Differential.

Test Plan: Ran `bin/search index D123 --force`, saw indexes appear in database. No UI/user impact yet.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18540
2017-09-05 16:45:37 -07:00
Chad Little
af7c92f2c6 Config re-design
Summary:
This is a full UI pass at a cleaner "Config" application. The main idea is to simplify the UI, center it, and have a different feel than other UI, a sort of "manage" UI theme for objects with loads of settings. Also adds a new minimalistic "WHITE_CONFIG" box type which may get re-used in Diffusion settings. This is a 90% pass, I'll have a few follow up diffs. Specifically:

 - Build breadcrumbs as a flexible UI to go into headers.
 - One click ObjectItemView option, for hover states.
 - Sidenav doesn't always select (AphrontFilter issue)
 - Mobile touchups, though it's pretty reasonable.

Test Plan:
Click through every page here, edit options, see new navigation UI. Test a few various setup issue layouts including fatals.

{F5163228}

{F5163229}

{F5163230}

{F5163231}

{F5163232}

{F5163233}

{F5163234}

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D18519
2017-09-05 15:24:15 -07:00
epriestley
20aad35e60 Move Ferret engine "title:..." field definitions to the engine itself
Summary: Ref T12819. Move these out of the core engine into the Ferret engine. In the future different applications can define different functions, like "summary:..." or whatever. This may get more formalization when I possibly do "author:" and such some time down the road.

Test Plan: Searched for "title:...". Searched for "dog:...", got a useful error.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18536
2017-09-05 11:57:51 -07:00
epriestley
46abc11114 Reduce the number of magic strings in the Ferret implementation
Summary:
Ref T12819. Push more of the magic `' '` stuff into the engine and simplify calls to ngram construction.

Also fixes a bug where a task with title "apple banana" and description "cherry doughnut" could match query "banana cherry" by separating separate term segments with newlines instead of spaces.

Test Plan:
  - Indexed some objects.
  - Searched (term, substring, quoted terms).
  - Viewed index in database.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18534
2017-09-05 11:57:35 -07:00
epriestley
4a7593f47f Consolidate more Ferret engine code into FerretEngine
Summary: Ref T12819. Earlier I separated some ngram code into an "ngram engine" hoping to share it across the simple Ngrams stuff and the full Ferret stuff, but they actually use slightly different rules. Just pull more of this stuff into FerretEngine to reduce the number of moving pieces and the amount of code duplication.

Test Plan: Searched for terms, rebuilt indexes.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18533
2017-09-05 11:57:18 -07:00
epriestley
d5ba30c777 Fix credential control logic for restricted credentials
Summary: Fixes T12975. This logic didn't deal with PolicyException correctly.

Test Plan: {F5167549}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12975

Differential Revision: https://secure.phabricator.com/D18537
2017-09-05 11:56:46 -07:00
Chad Little
33b4de9acf Update Setup Issue UI
Summary: Slightly cleaner layout

Test Plan: review setup issues resolved and unresolved in local config. fake a fatal.

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18521
2017-09-05 10:40:48 -07:00
epriestley
577d498033 Create a virtual "core" field in the Ferret engine for "title and body together"
Summary: See PHI46. The `core:` function means "find results in either the title or body, but not other auxiliary fields like comments".

Test Plan: Searched for text present in the title (yes), body (yes), and comments (no) with the `core:...` prefix.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18514
2017-09-01 09:40:56 -07:00
epriestley
f4f73e0a7e Separate fulltext engine extensions into "enrich" and "index" phases
Summary:
Ref T12819. Some of the extensions "enrich" the document (adding more fields or relationships), while others "index" it (insert it into some kind of index for later searching).

Currently, these are all muddled under a single "index" phase. However, the Ferret extension cares about fields and relationships which other extensions may add.

Split this into two phases: "enrich" adds fields and relationships so other extensions can read them later if they want. "Index" happens after the document is built and has all the fields and relationships.

The specific problem this solves is that comments may not have been added to the document when the Ferret extension runs. By moving them to the "enrich" phase, the Ferret engine will be able to see and index comments.

Test Plan: Ran `bin/search index ...`, grepped for `indexFulltextDocument`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18513
2017-09-01 09:40:11 -07:00
Chad Little
2ba5968b76 Mobile layouts for Diffusion
Summary: Implements a new mobile view thats more fullscreen, not boxed, so more space. Fixes issues with mobile tables when scrolling overflowed content.

Test Plan: Test home, branch, tags, code, file browse, graph, compare, history, readme, open revisions, owners.

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18505
2017-08-30 12:28:00 -07:00
Chad Little
67c658a7ed Use selected button state on blame button
Summary: Visually selects the button if blame is on.

Test Plan: Turn blame on and off in Diffusion on a file.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18504
2017-08-30 18:33:42 +00:00
epriestley
df9c24e750 Provide some "term vs substring" support for the Ferret engine
Summary:
Ref T12819. Distinguishes between "term" queries and "substring" queries, and tries to match them correctly most of the time. For example:

  - `example` matches "example", obviously.
  - `~amp` matches "example", but `amp` does not.
  - `examples` matches "example" through stemming.
  - `"examples"` does not match "example" (quoted text does not stem).
  - `"an examp"` does not match "an example" (quoted text is still term text).
  - `~"an examp"` matches "an example" (quoted, substring-operator text uses substring search).

Test Plan: Ran searches similar to the above, they seemed to do what they should.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18500
2017-08-30 11:30:04 -07:00
epriestley
e5a495f435 Parse raw Ferret queries into tokens before processing them
Summary:
Ref T12819. Depends on D18492. Instead of passing a raw query into the Query layer, parse it first.

This allows the query layer to figure out which parts should be substring vs term match, and would allow the SearchEngine layer to do `author:...` eventually by picking it out before sending it to the Ferret engine.

Test Plan: Ran some Ferret queries. They work like before, except that nonsense like `-+"quack"` raises an exception now.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18499
2017-08-30 11:29:46 -07:00
epriestley
0e2e525bb4 Add a "terms" corpus to Ferret fields
Summary:
Ref T12819. Ferret currently does substring search, but this is not the default mode users expect: when you search for the "RICO" act, you do not expect to find documents containing "apRICOt" even though "RICO" is a substring.

To support term search, index the corpus as a list of terms with puncutation removed and whitespace normalized so the engine can match against it.

Test Plan:
Ran `storage upgrade`, ran `search index`, saw sensible database results:

```
   rawCorpus: This is the task description.

Hark! Whom'st'dve eaten this "food" shall surely ~perish~?? #blessed
normalCorpus: thi the task descript hark whom dve eaten food shall sure perish bless
  termCorpus:  This is the task description Hark Whom'st'dve eaten this food shall surely perish blessed
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18498
2017-08-30 11:29:14 -07:00
epriestley
77ef38f9a8 Aggregate corpus data in Ferret field rows
Summary:
Ref T12819. This addresses two issues:

  - One practical issue is that right now, if you search for "dog cat", and they appear in different fields (for example, "dog" appears ONLY in the title, while "cat" appears ONLY in a comment) we won't find the document. This is somewhat rare -- usually, if "dog" appears in the title, it's also repeated in the description -- but I think clearly a bug. To attack this, start automatically creating a virtual "ALL" field with the full document text which we'll use as the primary thing we match against.
  - For fields which may occur more than once -- today, only comments -- aggregate them all into one big "all of the text" row instead of writing one row per comment. This partly addresses the first point ("dog" in one comment and "cat" in a different comment won't be found) and partly makes some of the query gymnastics easier.

Test Plan:
Ran `bin/storage upgrade`, ran `bin/search index <Txxx>`, saw sensible corpus values in the database:

```
mysql> select * from maniphest_task_ffield\G
*************************** 1. row ***************************
          id: 3
  documentID: 1981
    fieldKey: full
   rawCorpus: This is the task title
This is the task description.
normalCorpus: thi the task titl
thi the task descript
*************************** 2. row ***************************
          id: 4
  documentID: 1981
    fieldKey: titl
   rawCorpus: This is the task title
normalCorpus: thi the task titl
*************************** 3. row ***************************
          id: 5
  documentID: 1981
    fieldKey: body
   rawCorpus: This is the task description.
normalCorpus: thi the task descript
3 rows in set (0.00 sec)
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18497
2017-08-30 11:28:30 -07:00
epriestley
72cb3d3c84 Limit the damage that degenerate project name typeahead queries can cause
Summary:
See PHI47. When users copy/paste a wall of text into a project tokenizer, we can end up performing a very large number of JOINs.

These JOINs seem okay locally and on `secure`, but the install in PHI47 reports hitting issues.

Since these queries are almost certainly illegitimate (I think no one uses 5+ words to find a project), just limit the search to the 5 longest tokens.

Note that typing 6 tokens will still almost always work, since the UI does additional filtering. However, if you have 100+ projects named "a b c d e ..." and search for "a b c d e z", you may not hit it. This is so degenerate that it's hard to imagine any users encountering it.

This is a stopgap fix, I'll file something longer-term as a followup.

Test Plan: Used `/typeahead/class/PhabricatorProjectDatasource/` to run queries. Saw the same results with shorter query plans for all reasonable queries.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18506
2017-08-30 11:23:38 -07:00
Chad Little
11046d495d Add a selected button ui state
Summary: Only for grey buttons, but can expand. Sets a selected class.

Test Plan: Review new changes in UIExamples.

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18501
2017-08-30 10:14:29 -07:00
epriestley
b4cbea9018 Make legacy revision statuses from "differential.query" have type "string" again
Summary:
Ref T2543. The type on these got changed by accident, it should be "string" (crazy nonsense, compatible) not "int" (sensible, not compatible).

(New API uses sensible strings like "accepted" only.)

Test Plan: Called `differential.query` from web UI, saw `"2"` and similar statuses.

Reviewers: chad, jmeador, lvital

Reviewed By: jmeador, lvital

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18493
2017-08-29 13:05:02 -07:00
epriestley
f49d103af5 Fix an issue where "Close Revision" did not appear in the UI
Summary:
Ref T2543. When called from the UI to build the dropdown, there's no Editor, since we aren't actually in an edit flow.

This logic worked for actually performing the edits, just not for getting the option into the dropdown.

Test Plan: Used the dropdown to close an "Accepted" revision which I authored.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18490
2017-08-29 09:58:48 -07:00
Chad Little
f3f671aa90 Align first nav item in settings
Summary: This removes the redundant "Account" label and item, and just keeps the page better aligned.

Test Plan: Review personal settings

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18489
2017-08-29 09:40:49 -07:00
epriestley
4005a465f7 Make Ferret indexing more robust (UTF8, exception handling)
Summary:
Ref T12819. Two minor improvements from live data:

  - Tokenize in a UTF8-aware way.
  - When one document fails to index, kill the transaction explicitly (rather than leaving it hanging) so we don't cause other failures later.

Test Plan: Created some UTF8 documents locally, indexed them, got clean results.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819

Differential Revision: https://secure.phabricator.com/D18487
2017-08-28 15:49:57 -07:00
Chad Little
0609133f45 Limit notifications to latest 10, instead of 15
Summary: This panel just gets super tall at 15 now that date is on it's own line.

Test Plan: Reload panel, count to 10.

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18486
2017-08-28 15:15:06 -07:00
epriestley
f97157e7ed Build a prototype fulltext engine ("Ferret") using only basic MySQL primitives
Summary:
Ref T12819. I gave this stuff a sweet code name because all the terms related to "fulltext" and "search" already mean 5 different things. It, uh, ferrets out documents for you?

I'm building this to work a lot like the existing ngram index, which seems to work pretty well. If this sticks, it will auto-resolve the join issue (in T12443) by letting us do the entire thing locally in a JOIN and thus dodge a lot of mess.

This index gets built alongside other indexes, but only shows up in the UI if you have prototypes enabled. If you do, it appears under the existing fulltext field in Maniphest. No existing functionality is affected or disrupted.

NOTE: The query engine half of this is still EXTREMELY primitive, and this probably performs worse than the existing field for now. If this doesn't show obvious signs of being awful on `secure` I'll improve that in followup changes.

Test Plan:
Indexed my tasks, ran some simple queries, got the results I wanted, even for queries "ko", "k", "v0.1".

{F5147746}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12819, T12443

Differential Revision: https://secure.phabricator.com/D18484
2017-08-28 14:52:59 -07:00
epriestley
643877b467 Don't prompt to mark notifications as read if we don't need to
Summary: Fixes whatever task is tracking this junk, if one exists. Don't prompt unless there's a security issue.

Test Plan:
  - Generated notifications from a test account.
  - Clicked "Mark All" from dropdown menu, no prompt.
  - Clicked "Mark All" from notifications screen, no prompt.
  - Command-Clicked "Mark All" from dropdown menu to open in new window, got normal prompt.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18483
2017-08-28 13:05:08 -07:00
Chad Little
b8b701faf7 Clarify language when Autoclose is disabled for a repository
Summary: Fixes T12051, adds additional language.

Test Plan:
Disable Autoclose in Actions, see updated language under Branches.

{F5147291}

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Maniphest Tasks: T12051

Differential Revision: https://secure.phabricator.com/D18482
2017-08-28 12:00:46 -07:00
Chad Little
37843127e9 Widen blame age line in blame view
Summary: 50% more line, no additional cost! Order Now! Operators are standing by.

Test Plan: Blame a file

Reviewers: epriestley, avivey

Reviewed By: avivey

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18481
2017-08-28 11:32:06 -07:00
Chad Little
79c6b50049 Fix fatal on logged out Phame Post
Summary: Just deletes the view code until I have time to better plan this out, or just not ship.

Test Plan: Visit Phame post on public logged out page, view count doesnt cause transaction fatal.

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18475
2017-08-25 08:47:59 -07:00
epriestley
213e4ec9b5 Add a missing (int) cast to diff IDs for new "transaction.search" method
Summary: These come out of the database as strings (see T12678), force them to integers for the API.

Test Plan: Called `transaction.search`, got integers in JSON instead of strings.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18476
2017-08-25 07:31:22 -07:00
Chad Little
94cad30ac3 Fix bad tables in diffusion blame
Summary: My fake data was 100%, and not all tables have full revision history. This leads to a broken table. Instead check if we have //any// revisions at all, then always show the column, with or without a link inside.

Test Plan: going on a limb this is the correct fix and test on secure... again ...

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18474
2017-08-24 20:02:35 -07:00
Chad Little
12ae08b6b1 Move differential revision to its own table column in blame view
Summary: There is still some layout issues with revisions, so I've tested it better and moved it to it's own column

Test Plan: Fake in some revision data, test various sizes and shapes.

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18473
2017-08-24 19:36:42 -07:00
Chad Little
336fe5cdc5 Dont send an email when someone views a Phame post
Summary: lulz. :(

Test Plan: Load PhamePost, get email. Fix. Reload PhamePost, no email.

Reviewers: epriestley, avivey

Reviewed By: avivey

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18471
2017-08-24 19:00:20 -07:00
epriestley
fa5bcf5d94 Provide some more detailed information about inline comments in "transaction.search"
Summary:
Ref T5873. This provides paths and line numbers for inline comments.

This is a touch hacky but I was able to keep it mostly under control.

Test Plan:
  - Made inline comments.
  - Called API, got path/line information.

{F5120157}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T5873

Differential Revision: https://secure.phabricator.com/D18469
2017-08-24 15:26:50 -07:00
epriestley
9639ec0dfa Slightly simplify logic for determining if an inline comment has an effect
Summary: Minor cleanup, this logic can be simpler. Instead of special-casing inlines as having an effect if the have a comment, just consider any transaction with a comment to have an effect. I'm fairly certain this is always true.

Test Plan: Made inlines, tried to submit empty comments. Behavior unchanged.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18468
2017-08-24 15:26:32 -07:00
epriestley
6c9026c33a Allow ModularTransactions to opt in to providing data to Conduit
Summary:
Ref T5873. See PHI14. I don't want to just expose internal transaction data to Conduit by default, since it's often: unstable, unusable, sensitive, or some combination of the three.

Instead, let ModularTransactions opt in to providing additional data to Conduit, similar to other infrastructure. If a transaction doesn't, the API returns an empty skeleton for it. This is generally fine since most transactions have no real use cases, and I think we can fill them in as we go.

This also probably builds toward T5726, which would likely use the same format, and perhaps simply not publish stuff which did not opt in.

This doesn't actually cover "comment" or "inline comment", which are presumably what PHI14 is after, since neither is modular. I'll probably just put a hack in place for this until they can modularize since I suspect modularizing them here is difficult.

Test Plan: Ran `transaction.search` on a revision, saw some transactions (title and status transactions) populate with values.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T5873

Differential Revision: https://secure.phabricator.com/D18467
2017-08-24 15:25:55 -07:00
epriestley
2722c167d8 Add the skeleton for a "transaction.search" Conduit API method
Summary:
Ref T5873. See PHI14. This does the basics that are shared across everything (IDs, PHIDs, dates, comments).

It doesn't do types (I think I don't necessarily want to expose internal types over the API?) or transaction-specific data.

In the next change, I'm going to add ways to let ModularTransactions "opt-in" to providing more data to Conduit. I'll use this to flesh out the actual desired transaction types (comments, presumably inline comments) and likely leave the rest as skeletons for now until use cases arise so we don't create a backward compatibility issue (or a security issue!) by exposing tons of internal stuff as public-facing API.

Test Plan:
Ran queries, used paging. Retrieved an edited, deleted, and normal comment.

{F5120060}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T5873

Differential Revision: https://secure.phabricator.com/D18466
2017-08-24 15:25:34 -07:00
epriestley
ba1925b155 Prevent Differential changeset HTML anchors from colliding with comment anchors
Summary:
Fixes T12970. This is easier than I expected, and appears to occur in only one place.

This prevents a change from ever generating with an anchor like `#12345678`, which is ambiguous because it may be a comment anchor.

Test Plan: Viewed a revision, saw new `change-xxxyyyzzz` anchors, clicked one, got jumped to the right place.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12970

Differential Revision: https://secure.phabricator.com/D18465
2017-08-24 15:25:17 -07:00
epriestley
47da632a22 Separate saved queries in applications into "personal" and "global" queries
Summary:
Ref T12956. UI changes:

  - Administrators get a new `[X] Save as global query` option when saving a query.
  - "Edit Queries..." is split into "Personal" and "Global" sections. For administrators, each section can be edited. For non-admins, only the top section can be edited, but any query can be pinned.

A couple notes:

  - This doesn't support "pin for everyone by default". New users just get the first query from the bottom set. That seems reasonable for now.
  - Reordering is currently a little buggy (it works if you've reordered before, but not if you're reordering for the first time), but I need to migrate before I can fix / test that properly. So that'll get cleaned up in the next change or two.

Test Plan:
  - As an admin and non-admin, viewed, edited, disabled, saved-as-personal and saved-as-global various queries.

{F5098581}

{F5098582}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12956

Differential Revision: https://secure.phabricator.com/D18426
2017-08-24 15:24:34 -07:00
epriestley
58b889c5b0 Make the default ApplicationSearch query explicit, not just the first item in the list
Summary:
Ref T12956. Currently, when you visit `/maniphest/` (or any other ApplicationSearch application) we execute the first query in the list by default.

In T12956, I plan to make changes so that personal queries are always first, then global/builtin queries. Without changing the "default query" rule, this will make it harder to have, for example, some custom queries in Differential but still run a global query like "Active" by default. To make this work, you'd have to save a personal copy of the "Active" query, then put it at the top.

This feels a bit cumbersome and this rule is kind of implicit and a little weird anyway. To make this work a little better as we make changes here, add an explicit pinning action, like the one we have in Project ProfileMenus.

You can now explicitly choose a query to make default.

Test Plan:
  - Browsed without pinning anything, saw normal behavior.
  - Pinned queries, viewed `/maniphest/`, saw a non-initial query selected by default.
  - Pinned a query, deleted it, nothing exploded.

{F5098484}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12956

Differential Revision: https://secure.phabricator.com/D18422
2017-08-24 15:21:00 -07:00
epriestley
d6e47eef19 Don't set a default "group by priority" in the task search engine
Summary:
See PHI42. Currently, `maniphest.search` incorrectly applies this default (group by priority) to all queries via Conduit.

The correct behavior is to apply no grouping constraint.

I think this is also a reasonable general behavior, and the current code seems to date from D6960 in 2013 and didn't seem particularly carefully considered.

This is a minor compatibility break -- saved queries which are more than 4 years old might change their group behavior. I'll note this in the change logs but expect essentially no one to be affected.

Test Plan: Ran a `maniphest.search` Conduit call and observed the underlying query. Before this change, it executed `ORDER BY priority, id`. After this change, it correctly executed `ORDER BY id` only.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18459
2017-08-24 12:37:44 -07:00
Chad Little
66613240fa Have text-less dropdown buttons look better
Summary: Using icons and dropdown buttons without text looks a little wonky, this resets the CSS a bit.

Test Plan: Review button with icon and text, just icon, just test, and dropdowns.

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18461
2017-08-24 12:36:56 -07:00
epriestley
68df3cebc8 Allow task parents and subtasks to be edited via Conduit API
Summary:
See PHI39. This adds support for editing parents and subtasks of a task via Conduit.

It might be nice to tie this into the `PhabricatorObjectRelationship` stuff eventually, but I think we'd effectively end up in the same place anyway in terms of what the API looks like.

Test Plan: {F5116163}

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18456
2017-08-23 14:52:31 -07:00
Chad Little
63bd1784b0 Allow more granularity on real-time notifications
Summary: Fixes T12792. Expands the Notifications to "web, desktop, both, or none" for real-time notifications in settings.

Test Plan: Test with "test notifications" button, and while logged into two accounts with each of the 4 settings.

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Maniphest Tasks: T12792

Differential Revision: https://secure.phabricator.com/D18457
2017-08-23 14:45:13 -07:00
Chad Little
8c4f5aba33 Fix Back to HEAD link
Summary: I missed an anchor tag here, adds it back

Test Plan: View blame, click a previous version of the file, click Back to HEAD link.

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Differential Revision: https://secure.phabricator.com/D18451
2017-08-23 09:47:52 -07:00
Chad Little
ac91ab1ef9 Update blame view in Diffusion
Summary: Ref T12824, adds more information to the blame view, exposes date, commit summary, lighter colors.

Test Plan:
Review many diffs with and without blame on.

{F5111758}

{F5111759}

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Maniphest Tasks: T12824

Differential Revision: https://secure.phabricator.com/D18452
2017-08-23 09:47:35 -07:00
Chad Little
748725a47d Don't select disabled menu items as default
Summary: Fixes T12969. If you disable "Home" but leave it at the top, we still load it.

Test Plan: Disabled "Home". Move Dashboard into first position, see correct home layout.

Reviewers: epriestley

Reviewed By: epriestley

Spies: Korvin

Maniphest Tasks: T12969

Differential Revision: https://secure.phabricator.com/D18455
2017-08-23 09:40:30 -07:00
epriestley
df21391b8e When "apcu_clear_cache()" exists, prefer it as a cache clear callback over "apc_clear_cache()"
Summary:
See PHI36. APCu originally had `apc_` methods, but at some point dropped these and only provides `apcu_` methods.

When the `apcu_` method is present, use it. It may not be present for older versions of APCu, so keep the fallback.

Test Plan:
  - With modern APCu, clicked "Purge Caches" in Config > Caches.
  - Before: fatal on bad `apc_clear_caches` call.
  - After: Valid cache clear.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18449
2017-08-21 15:32:44 -07:00
Chad Little
e40c002a6d Add a basic view count to Phame
Summary: This adds a very very basic view count to Phame, so bloggers can get some idea which posts are more popular than others. Anything more than this I think should be Facts or Google Analytics.

Test Plan: Write a new post, see post count. Reload page, post count goes up. Archive post, post count stays the same.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18446
2017-08-21 14:03:21 -07:00
Chad Little
39d19c33ec Redirect users back to where they added an SSH Key
Summary: Ref T12964. This feels like a cheat, but works well. Just redirect the user back to the form they came from instead of to the key page.

Test Plan: Add a key to a user profile, add a key to an Alamanac device. Grep for PhabricatorAuthSSHKeyTableView and check all locations.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12964

Differential Revision: https://secure.phabricator.com/D18445
2017-08-21 14:02:27 -07:00
Chad Little
a145d00be6 Update Diffusion File UI for single column
Summary: Moves browseFile to single column, implements Owners as a list under the file (and now directory as well), improved information listed in Owners, and moves actions into the Diffusion action bar instead of the header.

Test Plan:
Test browsing directories, files, text, images, binaries, enabling blame. Mobile and desktop.

{F5111045}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18448
2017-08-21 13:35:25 -07:00
Chad Little
d2a3f2da73 Add indication of hg branch open/closed in branch list
Summary: Adds some basic UI for open / closed state when viewing a list of branches in Mercurial. Fixes T12838

Test Plan: Close and open branches, view list.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12838

Differential Revision: https://secure.phabricator.com/D18447
2017-08-21 09:09:16 -07:00
Chad Little
295c806219 Hide branch status if repository is not hg
Summary: Better table layouts here for branches view

Test Plan: Test git, hg repositories. See column go away.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18444
2017-08-17 14:44:55 -07:00
Chad Little
864dd9a196 List branch on main repository view
Summary: This is in the crumbs, but a little hidden. Puts branch name at the top of the browse table header.

Test Plan: Review a few branchs, change branch, see new name.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18441
2017-08-17 12:15:53 -07:00
Chad Little
281fc19f3f Add more information to Branch status page in Manage Repository
Summary: Adds an icon for default branch, status for branch status

Test Plan: Review `hg` and `git` repositories, change default branch, etc.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18443
2017-08-17 12:09:20 -07:00
epriestley
68008dce60 Fix a possible database ref fatal during MySQL setup checks if a host is unreachable
Summary:
Ref T12966. See that task for a description and reproduction steps.

If you put Phabricator in a master/replica configuration and then restart it, we may fatal here if the master is unreachable. Instead, we should survive setup checks.

Test Plan: Put Phabricator in a master/replica configuration, explicitly disabled the master by misconfiguring the port, restarted Phabricator. Before: fatal; after: login screen in read-only mode.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12966

Differential Revision: https://secure.phabricator.com/D18442
2017-08-17 11:43:13 -07:00
epriestley
c9986fd5de Don't fatal in ElasticSearch setup check if no "master" database is configured
Summary:
Ref T12965. See that task for discussion, and PHI36 for context.

This sweeps the fatal under the rug by skipping it, letting things move forward for now.

Test Plan: Followed instructions in T12965, got a read-only recovery after restart instead of a fatal.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12965

Differential Revision: https://secure.phabricator.com/D18440
2017-08-17 10:39:00 -07:00
Chad Little
053cab4d59 Update VCS Password settings page
Summary: Use proper background.

Test Plan: Visit page, see correct background.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18435
2017-08-17 08:51:38 -07:00
Chad Little
dc10bb1f49 Update Settings to use TwoColumn fixed layout
Summary: Simplifies the page, adds base support for PHUITwoColumn fixed from Instances (which I'll delete css there).

Test Plan:
click on every settings page, UI seems in tact, check mobile, desktop, mobile menus.

{F5102572}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18436
2017-08-17 08:51:17 -07:00
Chad Little
5019960b61 Set border on crumbs on Lint page
Summary: Minor, sets the border, corrects a page header.

Test Plan: View lint pages

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18433
2017-08-16 12:19:35 -07:00
Chad Little
19dae88728 Add branch, tag info to Diffusion Headers
Summary: Improves overall UX of browsing Diffusion. Clarifies branch and tag when possible, changes 'home' to 'code', uses tabs in more locations. Fixes T12837

Test Plan: Review branchs, tags, git, hg, search, browse, history.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12837

Differential Revision: https://secure.phabricator.com/D18434
2017-08-16 12:16:15 -07:00
Chad Little
7bbd26427f Add pattern search to diffusion home
Summary: Moves the method up to DiffusionController, so it can be more universally used. Also now center aligns tabs on mobile. Still todo, get search nicely toggled on mobile

Test Plan: Test mobile, desktop. Test search from home, from browse, and browsing a specific path.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18432
2017-08-15 14:16:33 -07:00
Chad Little
f4fdb92e13 Move Diffusion Actions into action bar on home
Summary: Moving this down the the "bar" to allow pattern search on home. Rebuilds the mobile layout a little.

Test Plan:
Test actions on mobile, desktop, tablet.

{F5100460}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18431
2017-08-15 12:19:49 -07:00
Chad Little
3a50ea4f47 Simplify Create Repository page
Summary: Also adds images, nice images.

Test Plan: Create a repository, test mobile, tablet, desktop layouts.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18430
2017-08-15 11:05:50 -07:00
Chad Little
4d335b7bef Build a basic DiffusionPatternSearchView
Summary: Roughs this in a little, kinda basic. Allows for grouping results by page. A bit better on mobile. Would like more content return from conduit though.

Test Plan:
Test `CMS`, `cms`, and `OMGLOLWTFBBQ`, desktop and mobile

{F5099081}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18429
2017-08-15 06:38:47 -07:00
Chad Little
0a9ad6d5e7 Move pattern search into Diffusion header
Summary: This is only on browse pages, but I think could be global (home) also. Moves it from a button, field, to just a field.

Test Plan:
Review search on desktop, mobile.

{F5098886}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18428
2017-08-14 19:03:56 -07:00
Chad Little
37489865d4 Remove "File Name" search tool
Summary: Removing this cleanly in event we want to put it back later. 99% of these cases are likely workable either by command line or the typeahead. Will gauge feedback if users notice.

Test Plan: Reload page, perform file grep search.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18425
2017-08-14 18:58:25 +00:00
Chad Little
07c0032491 Add a link directly to Browse in Diffusion
Summary: Getting to the straight browse view went away, this adds a link back. I'll look at more long term solution for getting to grep search.

Test Plan: Click on header, get take to browse view.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18421
2017-08-14 11:14:14 -07:00
epriestley
8c3243ef68 Lightly modernize NamedQueryQuery
Summary: Ref T12956. No real behavioral changes here, just slightly more modern code.

Test Plan: Reviewed named queries in Maniphest and "Edit Queries...".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12956

Differential Revision: https://secure.phabricator.com/D18420
2017-08-14 09:07:11 -07:00
epriestley
48a74de0b6 Move all revision status transactions to modern values and mechanics
Summary:
Ref T2543. This updates and migrates the status change transactions:

  - All storage now records the modern modular transaction ("differential.revision.status"), not the obsolete non-modular transaction ("differential:status").
  - All storage now records the modern constants ("accepted"), not the obsolete numeric values ("2").

Test Plan:
  - Selected all the relevant rows before/after migration, data looked sane.
  - Browsed around, reviewed timelines, no changes after migration.
  - Changed revision states, saw appropriate new transactions in the database and timeline rendering.
  - Grepped for `differential:status`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18419
2017-08-12 04:05:57 -07:00
epriestley
7b695aa43b Migrate revision storage to modern status constants ("accepted") instead of legacy numeric values ("2")
Summary:
Ref T2543. Rewrites all the storage to use constants.

Note that transactions still use legacy values, I'll migrate and update them separately.

Test Plan:
  - Ran migration.
  - Browsed around, changed revision states, viewed dashboard, etc.
  - Selected `DISTINCT()` and `GROUP_CONCAT()` of the `status` field in the database, saw sane/expected before and after values.
  - Verified that old Conduit methods still return numeric constants for compatibility.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18418
2017-08-12 04:02:10 -07:00
epriestley
5348f34c9e Make all revision status readers explicitly read modern or legacy status
Summary: Ref T2543. All writers now write modern statuses. Make all readers explicit about whether they are reading modern or legacy statuses, so I can swap the storage format.

Test Plan:
  - Grepped for `getStatus()`, scanned the list. Other applications have methods with this name so it's possible I missed something.
  - Browed around, changed revision statuses.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18417
2017-08-11 17:22:22 -07:00
epriestley
0b1d6a3f6e Convert straggling Herald rules to modern revision status constants
Summary:
Ref T2543. These are the last `ArcanistDifferentialRevisionStatus` callsites.

This removes the very old legacy `precommitRevisionStatus` field, which has no other readers. This was obsoleted by the `CLOSED_FROM_ACCEPTED` stuff, but retained for compatibility.

Test Plan:
  - Poked these with the test console, although they're a little tricky to be sure about.
  - Grepped for `ArcanistDifferentialRevisionStatus`, no more hits.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18416
2017-08-11 17:22:05 -07:00
epriestley
cd15c2d545 Swap transactions and initialization over to modern status constants
Summary: Ref T2543. Update these for the modern stuff.

Test Plan: Created a new revision, got a revision in the right state ("Needs Review"). Accepted, planned, requested, abandoned revision; state transitions looked good.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18415
2017-08-11 17:21:51 -07:00
epriestley
895f0cde1f Use modern revision statuses when bucketing revisions on the Differential dashboard
Summary: Ref T2543. Swaps these over to modern constants.

Test Plan: Viewed dashboard, no chagnes to bucketing.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18414
2017-08-11 17:21:27 -07:00
epriestley
7f743c14d5 Remove remaining ArcanistDifferentialRevisionStatus references in revision state logic
Summary: Ref T2543. This cleans up all the "when no one is rejecting/blocking and someone accepted, mark the revision overall as accepted" logic to use more modern status stuff instead of `ArcanistDifferentialRevisionStatus`.

Test Plan:
  - Updated revisions, saw them go to "Needs Review".
  - Accepted, requested changes to revisions.
  - Updated one with changes requested, saw it go to "needs review" again.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18413
2017-08-11 17:21:09 -07:00
epriestley
2b9838b482 Modularize remaining TYPE_ACTION transactions in Differential, reducing calls to ArcanistDifferentialRevisionStatus
Summary:
Ref T2543. This cleans up a couple of remaining rough edges:

  - We could do an older TYPE_ACTION "close" via the daemons.
  - We could do an older TYPE_ACTION "close" via `arc close-revision`, explicitly or implicitly in `arc land`, via API (`differential.close`).
  - We could do an older TYPE_ACTION "rethink" ("Plan Changes") via the API, via `arc diff --plan-changes` (`differential.createcomment`).

Move these to modern modular transactions, then get rid of all the validation and application logic for them. This nukes a bunch of `ArcanistDifferentialRevision::...` junk.

Test Plan:
  - Used `bin/repository reparse --message rXYZ...` to reparse a commit, closing a corresponding revision.
  - Used `differential.close` to close a revision.
  - Used `differential.createcomment` to plan changes to a revision.
  - Reviewed transaction log for full "closed by commit" message (linking to commit and mentioning author).
  - Grepped for `::TYPE_ACTION` to look for remaining callsites, didn't find any.
  - Grepped for `differential.close` and `differential.createcomment` in `arcanist/` to look for anything suspicious, seemed clean.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18412
2017-08-11 17:20:55 -07:00
epriestley
19bc91fd20 Modularize the Differential "status" transaction and move away from ArcanistDifferentialRevisionStatus
Summary:
Ref T2543. Converts the TYPE_STATUS transaction (used to render "This revision now requires changes to proceed.", "This revision is accepted and ready to land.", etc) to ModularTransactions.

Also, continue consolidating all the status-related information (here, more colors and icons) into a single place. By the end of this, we may learn that NEEDS_REVIEW uses //every// color.

Test Plan:
Reviewed old status transactions (unchanged) and created new ones (looked the same as the old ones).

(I plan to migrate all of these a few diffs from now, around when I change the storage format.)

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18410
2017-08-11 17:20:40 -07:00
epriestley
77bf245637 Continue reducing callsites to ArcanistDifferentialRevisionStatus in transactions
Summary: Ref T2543. Cleans up some more references to ArcanistDifferentialRevisionStatus, moving toward getting rid of it completely.

Test Plan: Planned changes, requested review, inspected the "close" one since it isn't trivial to trigger.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18408
2017-08-11 13:43:21 -07:00
epriestley
36197bf783 Provide revision status information via API all "differential.revision.search"
Summary: Ref T2543. Now that the integer status constants are banished to the internals, we can expose status information from "differential.revision.search".

Test Plan:
Searched for revisions.

{F5093873}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18400
2017-08-11 13:42:45 -07:00
epriestley
ef8d4e2126 Fix an inverted condition for the "Reopen Revision" action
Summary: Ref T2543. I converted this condition the wrong way, missing a `!`. I'll cherry-pick this to `stable`.

Test Plan: No more "Reopen Revision" action available on open revisions.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18399
2017-08-11 13:41:54 -07:00
epriestley
153e4d8a38 Remove old reviewer double writes to legacy edge table in Differential
Summary:
Ref T2543. Ref T10967. This isn't precisely related to "draft" status, but while I'm churning this stuff anyway, get rid of the old double writes to clean the code up a bit.

These were added in T10967 to make sure the migration was reversible/recoverable, but we haven't seen any issues with it in several months so I believe they can now be removed safely. Nothing has read this table since ~April.

Test Plan: Took various review actions on revisions (accept, reject, resign, comment, etc). If this change is correct, there should be no visible effect.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967, T2543

Differential Revision: https://secure.phabricator.com/D18398
2017-08-11 13:38:52 -07:00
epriestley
42020e1357 Completely remove "differential.find" Conduit API method
Summary:
Ref T2543. I believe there have been no upstream callsites of this method since D1646, in February 2012.

The method works, and we can revert this if needbe, but this seems like a good time to remove support.

Test Plan: Grepped for `differential.find`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18397
2017-08-11 13:38:28 -07:00
epriestley
13ddb15bbc Remove legacy withStatus() method from RevisionQuery
Summary: Ref T2543. All callsites are now in terms of `withStatuses()`.

Test Plan:
  - Called `differential.query` and `differential.find` from Conduit API.
  - Grepped through all `withStatus()` callsites.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18396
2017-08-11 13:37:47 -07:00
epriestley
50dfdb8d03 Replace legacy Differential queries for "open" revisions with a modern mechanism
Summary: Ref T2543. Several queries want only open revisions. Provide a tailored, non-legacy way to issue that query.

Test Plan: Viewed some of these callsites (e.g., "Similar open revisions affecting these files"), saw only open revisions.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18395
2017-08-11 13:37:11 -07:00
epriestley
53516093ae Replace Differential hard-coded status "<select />" with tokenizer
Summary:
Ref T2543. This updates the UI control in the web UI. Also:

  - This implicitly makes this queryable with the API (`differential.revision.search`); it previously was not.
  - This does NOT migrate existing saved queries. I'll do those in the next change, and hold this until it happens.
  - This will break some existing `/differential/?status=XYZ` links. For example, `status=open` now needs to be `status=open()`. I couldn't find any of these in the upstream, and I suspect these are rare in the wild (users would normally link directly to saved queries, not use URI query construction).

Test Plan: {F5093611}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18393
2017-08-11 13:36:00 -07:00
epriestley
8160baec2a Add a Differential revision status tokenizer datasource
Summary:
Ref T2543. This adds a tokenizer, similar to the Maniphest tokenizer, so the hard-coded `<select />` control in Differential ApplicationSearch can be replaced with a more flexible control that handles the addition of new statuses with more grace.

This only adds the new datasource.

Test Plan: Used `/typeahead/class/` to preview the behavior of the new datasource.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18392
2017-08-11 13:35:15 -07:00
Alex Vandiver
45b0fd8f9b Remove a debugging "echo" that crept in in dccd799b
Summary: This echo was accidentally added in dccd799b

Test Plan: Inspection.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D18391
2017-08-11 05:50:43 -07:00
epriestley
2c150076b0 Stop populating or updating working copies in observed Mercurial repositories
Summary:
Ref T12961. Fixes T4416. Currently, for observed Mercurial repositories, we build a working copy with `pull -u` (for "update").

This should be unnecessary, and we don't do it for hosted Mercurial repositories. We also stopped doing it years ago for Git repositories. We also don't clone Mercurial repositories with a working copy.

It's possible something has slipped through the cracks here so I'll hold this until after the release cut, but I believe there are no actual technical blockers here.

Test Plan:
  - Observed a public Mercurial repository on Bitbucket.
  - Let it import.
  - Browsed commits, branches, file content, etc., without any apparent issues.

Reviewers: chad

Reviewed By: chad

Subscribers: cspeckmim

Maniphest Tasks: T12961, T4416

Differential Revision: https://secure.phabricator.com/D18390
2017-08-10 19:14:56 -07:00
epriestley
794e185bf9 Pass SSH wrappers to VCS commands unconditonally, not just if there's an SSH remote
Summary:
Ref T12961. In Mercurial, it's possible to have "subrepos" which may use a different protocol than the main repository.

By putting an SSH repository inside an HTTP repository, an attacker can theoretically get us to execute `hg` without overriding `ui.ssh`, then execute code via the SSH hostname attack.

As an immediate mitigation to this attack, specify `ui.ssh` unconditionally. Normally, this will have no effect (it will just be ignored). In the specific case of an SSH repo inside an HTTP repo, it will defuse the `ssh` protocol.

For good measure and consistency, do the same for Subversion and Git. However, we don't normally maintain working copies for either Subversion or Git so it's unlikely that similar attacks exist there.

Test Plan:
  - Put an SSH subrepo with an attack URI inside an HTTP outer repo in Mercurial.
  - Ran `hg up` with and without `ui.ssh` specified.
  - Got dangerous badness without `ui.ssh` and safe `ssh` subprocesses with `ui.ssh`.

I'm not yet able to confirm that `hg pull -u -- <uri>` can actually trigger this, but this can't hurt and our SSH wrapper is safer than the native behavior for all Subversion, Git and Mercurial versions released prior to today.

Reviewers: chad

Reviewed By: chad

Subscribers: cspeckmim

Maniphest Tasks: T12961

Differential Revision: https://secure.phabricator.com/D18389
2017-08-10 17:49:55 -07:00
Chad Little
a7124f8f7a Add status table to Diffusion Branch manage page
Summary: Fixes T12832. Adds a basic table (not paginated?) to view tracking and autoclose status.

Test Plan:
Review a large repository (Krita) with setting various states of tracking and autoclose.

{F5092117}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12832

Differential Revision: https://secure.phabricator.com/D18386
2017-08-10 13:26:30 -07:00
epriestley
7f90ef2d82 Make the "Requested Changes to Prior Diff" reviewer icon red, not bluegrey
Summary: See PHI31. The "Accepted Older Revision" icon is (more reasonably) bluegrey, but that rule spilled over here where it doesn't make much sense. "Requested Changes to Prior Diff" remains in effect across updates, but the coloration implies otherwise.

Test Plan:
"Requested Changes to This Diff" (unchanged):

{F5092019}

"Requested Changes to Prior Diff" (now red, previously bluegrey):

{F5092020}

Note that the icons are different so this is technically colorblind-safe, and it's normally not important to distinguish between these two reds anyway.

Reviewers: chad, lvital

Reviewed By: lvital

Subscribers: lvital

Differential Revision: https://secure.phabricator.com/D18385
2017-08-10 11:04:21 -07:00
epriestley
8443366f32 Remove bin/files purge workflow
Summary:
Fixes T12948. See that task for substantial discussion and context. Briefly:

  - This workflow is very old, and won't work for large (>2GB) files.
  - This workflow has become more dangerous than it once was, and can fail in several ways that delete data and/or make recovery much more difficult (see T12948 for more discussion).
  - This was originally added in D6068, which is a bit muddled, but looks like "one install ran into a weird issue so I wrote a script for them"; this would be a Consulting/Support issue and not come upstream today. I can't identify any arguments for retaining this workflow there, at least.

Test Plan:
  - Grepped for `files purge`, got nothing.
  - Grepped for `purge`, looked for anything that looked like instructions or documentation, got nothing.

I don't recall recommending anyone run this script in many years, and didn't even remember that it existed or what it did when T12948 was reported, so I believe it is not in widespread use.

Reviewers: joshuaspence, chad

Reviewed By: joshuaspence

Maniphest Tasks: T12948

Differential Revision: https://secure.phabricator.com/D18384
2017-08-10 08:49:06 -07:00
Chad Little
92c49c3772 Minor UX tweaks to Phortune autopay
Summary: Fixes T12958. Adds a success message when card is added, also switches to use radio buttons for clarity. Updated redirect uri for deleting methods as well.

Test Plan:
Add cards, remove cards.

{F5091084}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12958

Differential Revision: https://secure.phabricator.com/D18381
2017-08-10 07:39:18 -07:00
Chad Little
3ba196152f Clean up some dialog spacing
Summary: Makes dialogs a little wider, form dialogs a lot wider (space controls). Also cleans up Passphrase dialogs. Fixes T12833. I think forms probably need to move to tables for better layout flexibility like veritical alignment.

Test Plan: Passphrase create, edit, etc. Other dialogs.

Reviewers: epriestley

Subscribers: Korvin

Maniphest Tasks: T12833

Differential Revision: https://secure.phabricator.com/D18382
2017-08-09 20:04:39 -07:00
epriestley
46d1596bf7 Pull legacy revision query status filters out of the main Query class
Summary:
Ref T2543. Currently, Differential uses a set of hard-coded query filters (like "open" and "closed") to query revisions by status (for example, "open" means any of "review, revision, changes planned, accepted [usually]").

In other applications, like Maniphest, we've replaced this with a low level list of the actual statuses, plus higher level convenience UI through tokenizer functions. This basically has all of the benefits of the hard-coded filters with none of the drawbacks, and is generally more flexible.

I'd like to do that in Differential, too, although we'll need to keep the legacy maps around for a while because they're used by `differential.find` and `differential.getrevision`. To prepare for this, pull all the legacy stuff out into a separate class. Then I'll modernize where I can, and we can get rid of this junk some day.

Test Plan: Grepped for `RevisionQuery::STATUS`. Ran queries via Differential UI.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18343
2017-08-09 11:06:15 -07:00
epriestley
03ab7224bb Reduce STATUS_CLOSED (now internally "Published") revision status callsites
Summary:
Ref T2543. Add `isPublished()` to mean: exactly the status 'closed', which is now interally called 'published', but still shown as 'closed' to users.

We have some callsites which are about "exactly that status", vs "any 'closed' status", e.g. including "abandoned".

This also introduces `isChangePlanned()`, which felt less awkward than `isChangesPlanned()` but more consistent than `hasChangesPlanned()` or `isStatusChangesPlanned()` or similar.

Test Plan: `grep`, loaded revisions, requested review.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18341
2017-08-09 11:05:42 -07:00
epriestley
70088f7eec Continue reducing callsites to ArcanistDifferentialRevisionStatus
Summary:
Ref T2543. Further consolidates status management into DifferentialRevisionStatus.

One change I'm making here is internally renaming "CLOSED" to "PUBLISHED". The UI will continue to say "Closed", at least for now, but this should make the code more clear because we care about "is closed, exactly" vs "is any closed status (closed, abandoned, sometimes accepted)". This distinction is more obvious as `isClosed()` vs `isPublished()` than, e.g., `isClosedWithExactlyTheClosedStatus()` or something. I think "Published" is generally more clear, too, and more consistent with modern language (e.g., "pre-publish review" replacing "pre-commit review" to make it more clear what we mean in Git/Mercurial).

I've removed the IN_PREPARATION status since this was just earlier groundwork for "Draft" and not actually used, and under the newer plan I'm trying to just abandon `ArcanistDifferentialRevisionStatus` entirely (or, at least, substantially).

Test Plan:
- Viewed revisions.
- Viewed revision list.
- Viewed revisions linked to a task in Maniphest.
- Viewed revision graph of dependencies in Differential.
- Grepped for `COLOR_STATUS_...` constants.
- Grepped for removed method `getRevisionStatusIcon()` (no callsites).
- Grepped for removed method `renderFullDescription()` (one callsite, replaced with just building a `TagView` inline).
- Grepped for removed method `isClosedStatus()` (no callsites after other changes).

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18340
2017-08-09 11:05:22 -07:00
epriestley
2e36653965 Reduce callsites to "ArcanistDifferentialRevisionStatus" in Phabricator
Summary:
Ref T2543. These are currently numeric values, like "0" and "3". I want to replace them with strings, like "accepted", and move definitions from Arcanist to Phabricator.

To set the stage for this, reduce the number of callsites where Phabricator invokes `ArcanistDifferentialRevisionStatus`.

This is just the easy ones. I'll hold this until the release cut.

Test Plan:
- Called `differential.find`.
- Called `differential.getrevision`.
- Called `differential.query`.
- Removed all reviewers from a revision, saw warning.
- Abandoned the no-reviewers revision, no more warning.
- Attached a revision to a task to get it to show the state icon with the status on a tooltip.
- Viewed revision bucketing on dashboard.
- Used `bin/search index` to reindex a revision.
- Hit the "Land Revision" endpoint.

I didn't explicitly test these cases:

  - Doorkeeper Asana integration, since setup takes a thousand years.
  - Disambiguation logic when multiple hashes match, since setup is also very involved.
  - Releeph because it's Releeph.

Reviewers: chad

Reviewed By: chad

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D18339
2017-08-09 11:04:52 -07:00
Chad Little
b1e3cf627d Add more repo images
Summary: Just a few more.

Test Plan: Edit Picture, see new image, choose image.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18370
2017-08-08 17:51:15 -07:00
Chad Little
7119c98744 Add a UIExamples page for PHUIBigInfoView
Summary: Fixes the icon bug and builds a basic examples page for future testing.

Test Plan: Visit uiexampls and various types of info views.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18356
2017-08-07 15:58:49 -07:00
Chad Little
ca8ae2d4ca Add a red button to PHUIButtonView
Summary: Danger Danger

Test Plan:
UIExamples

{F5084035}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18347
2017-08-06 08:09:40 -07:00
Chad Little
8ca29a607a Remove incorrect policy language on Diff reviewers
Summary: Fixes T12952. This never work AFAIK, so resolves this mis-information. See T4411 for follow up.

Test Plan: Click on policy for a diff, no longer see text.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12952

Differential Revision: https://secure.phabricator.com/D18349
2017-08-06 08:08:06 -07:00
Chad Little
83f66ce55e Update Settings to use full side-navigation
Summary: Moves Settings to use a normal side navigation vs. a two column side navigation. It also updates Edit Engine to do the same, but I don't think there are other callsites. Added a consistent header for better clarification if you were editng your settings, global settings, or a bot's settings.

Test Plan: Test each page on a personal account, create global settings, test each page there, create a bot account, and test each page on the bot account. Anything else?

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18342
2017-08-04 10:23:01 -07:00
epriestley
cfb86dddd2 Warn users that compound terms separated by apostrophes don't work in the MySQL FULLTEXT index either
Summary: Ref T12928. Like `v0.1`, terms in the form `yo's` (sequences of two or fewer characters separated by apostrophes) do not get indexed.

Test Plan: {F5078192}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12928

Differential Revision: https://secure.phabricator.com/D18324
2017-08-02 16:06:08 -07:00
Chad Little
ba4b936dff Use Log In vs. Login when it's a verb
Summary: Cursory research indicates that "login" is a noun, referring to a form, and "log in" is a verb, referring to the action of logging in. I went though every instances of 'login' I could find and tried to clarify all this language. Also, we have "Phabricator" on the registration for like 4-5 times, which is a bit verbose, so I tried to simplify that language as well.

Test Plan: Tested logging in and logging out. Pages feel simpler.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18322
2017-08-02 12:26:47 -07:00
epriestley
ab018e1b49 When destorying a repository, print a notification about removing the working copy
Summary:
Fixes T12946. `bin/remove destroy` does not remove working copies: it's more dangerous than usual, and we can't do it in the general (clustered) case.

Print a notification message after destroying a repository.

Test Plan:
  - Destroyed a repository, got a hint about the working copy.
  - Destroyed a task, things worked normally.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12946

Differential Revision: https://secure.phabricator.com/D18313
2017-08-01 08:57:39 -07:00
epriestley
f48f2dae9f Move Phabricator to use PhutilBinaryAnalyzer and show binary versions
Summary:
Fixes T12942.

  - Adds binary version and path information to {nav Config > Version Information}.
  - Replaces old code all over the place with new consolidated code.

Test Plan:
{F5073531}

Also faked some cases of missing binaries, bad versions, etc.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12942

Differential Revision: https://secure.phabricator.com/D18306
2017-08-01 07:14:48 -07:00
epriestley
a546b029b0 Censor credentials possibly present in Git remote URIs when pulls fail because of a URI mismatch
Summary: Fixes T12945.

Test Plan:
Mostly faked this, got a censored error:

```
$ ./bin/repository update R38
[2017-07-31 19:40:13] EXCEPTION: (Exception) Working copy at "/Users/epriestley/dev/core/repo/local/38/" has a mismatched origin URI, "https://********@example.com/". The expected origin URI is "https://github.com/phacility/libphutil.git". Fix your configuration, or set the remote URI correctly. To avoid breaking anything, Phabricator will not automatically fix this. at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryEngine.php:186]
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12945

Differential Revision: https://secure.phabricator.com/D18304
2017-07-31 09:43:01 -07:00
Chad Little
ddd7cbb698 Add setImage to PHUIActionPanelView
Summary: Additonal option to use newly made images in these views.

Test Plan:
Built an example in UIExamples.

{F5071682}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18299
2017-07-30 13:20:26 -07:00
Chad Little
1ad369b306 Remove PHUIInfoPanelView
Summary: We've never used this, and no current plans to.

Test Plan: grep for use cases.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18298
2017-07-30 12:29:18 -07:00
Chad Little
2538f67178 Add some more builtin project images
Summary: Moves over some of the icons we build for SAAS that can be useful for projects to. Also make builtin list dynamic.

Test Plan: Edit a project image, select a cool sword.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18297
2017-07-30 12:29:02 -07:00
Chad Little
ea0db5aa9d Clean up dropdown carets
Summary: Adds dropdown carets to buttons more universally that are actually dropdowns.

Test Plan: Differential, Application Search, Diffusion. Mobile and Desktop.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18292
2017-07-28 15:11:25 -07:00
epriestley
ee884db1f9 Don't fatal when viewing tags pointing at commits we haven't imported/parsed yet
Summary:
In Diffusion, the "Tags" view may read commits which haven't imported or parsed yet, and thus don't have loadable objects.

Most of this logic tests for `if ($commit)`, but the author part did not. Instead, don't render author information if `$commit` is not present.

Test Plan:
  - Loaded tags view with commits present.
  - Faked `$commit = null;`, loaded tag view, got this instead of a fatal:

{F5068432}

Reviewers: chad, amckinley

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18290
2017-07-28 10:43:24 -07:00
Chad Little
9cf5bc30cd Fix some copy and bugs in Ponder
Summary: Fixes T12939. Fixes some feed stories, mailtags, and headers.

Test Plan: Review changes locally by asking how babby is formed.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12939

Differential Revision: https://secure.phabricator.com/D18285
2017-07-28 06:57:26 -07:00
epriestley
e47f85cd98 In Differential, filter repository operations to just "Land" operations again
Summary:
Reverts D18276. See PHI18 for discussion. The additional rules here (roughly, "only show the first successful operation") didn't actually work out for the other types of operations.

This is all just figuring out a stopgap, T12935 and other changes should eventually provide real pathways here.

Test Plan: Straight revert.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18281
2017-07-26 11:51:18 -07:00
Chad Little
e7f94d7528 Properly version Legalpad documents
Summary: Fixes T12933. This now creates a new DocumentBody when creating or editing a legalpad document.

Test Plan:
Create a new document, edit document. Check database that version is saved as new row, and timestamps are correct.

```mysql> select * from legalpad_documentbody;
+----+--------------------------------+--------------------------------+--------------------------------+---------+---------------+--------+-------------+--------------+
| id | phid                           | creatorPHID                    | documentPHID                   | version | title         | text   | dateCreated | dateModified |
+----+--------------------------------+--------------------------------+--------------------------------+---------+---------------+--------+-------------+--------------+
|  1 | PHID-LEGB-nsgzqklzfmjahlcgobm7 | PHID-USER-72xwu7eurrpsu2kxgrvw | PHID-LEGD-v7mc3xyithjvbiqeksbj |       2 | Legal Title 1 | Body 2 |  1501037011 |   1501037081 |
|  2 | PHID-LEGB-2kaytwmjusljib6pjycc | PHID-USER-72xwu7eurrpsu2kxgrvw | PHID-LEGD-v7mc3xyithjvbiqeksbj |       3 | Legal Title 1 | Body 3 |  1501037521 |   1501037521 |
|  3 | PHID-LEGB-h6q6bi42w4rgxrhk3qdb | PHID-USER-72xwu7eurrpsu2kxgrvw | PHID-LEGD-7gxuhafvkoy2izkv4gdd |       1 | New 2         | asdf   |  1501037553 |   1501037553 |
+----+--------------------------------+--------------------------------+--------------------------------+---------+---------------+--------+-------------+--------------+
3 rows in set (0.00 sec)```

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: tmakarios, Korvin

Maniphest Tasks: T12933

Differential Revision: https://secure.phabricator.com/D18280
2017-07-26 09:29:56 -07:00
Chad Little
ca17e2283d Have Maniphest use create transactions when using email
Summary: Fixes T12929. Sets a create transaction if new.

Test Plan: test a new task over email via command line

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12929

Differential Revision: https://secure.phabricator.com/D18279
2017-07-25 13:36:50 -07:00
epriestley
1588d3e224 In Differential, render status for any active Drydock repository operation, not just "Land" operations
Summary:
See PHI18. Third parties can currently define other types of Drydock operations (like "Merge Check" or "Cherry-Pick") but we won't show them in the UI.

This is a simple change which improves third-party support for now. These kinds of operations generally make sense in the upstream, but the pathways to support are longer.

Test Plan:
  - Verified that there are no other types of repository operation which we'd want to exclude in the upstream today by reviewing the "Repository Operation" subclasses.
  - Will click some buttons in production to make sure this works.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18276
2017-07-25 09:55:40 -07:00
epriestley
c217d7619c Make "A" hide or show all inline comments
Summary: Ref T12733. See PHI17.

Test Plan: Pressed "A", then pressed "A".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12733

Differential Revision: https://secure.phabricator.com/D18274
2017-07-25 05:12:39 -07:00
epriestley
8034b9d819 Don't require a device be registered in Almanac to do cluster init/resync steps
Summary:
Fixes T12893. See also PHI15. This is complicated but:

  - In the documentation, we say "register your web devices with Almanac". We do this ourselves on `secure` and in the production Phacility cluster.
  - We don't actually require you to do this, don't detect that you didn't, and there's no actual reason you need to.
  - If you don't register your "web" devices, the only bad thing that really happens is that creating repositories skips version initialization, creating the bug in T12893. This process does not actually require the devices be registered, but the code currently just kind of fails silently if they aren't.

Instead, just move forward on these init/resync phases even if the device isn't registered. These steps are safe to run from unregistered hosts since they just wipe the whole table and don't affect specific devices.

If this sticks, I'll probably update the docs to not tell you to register `web` devices, or at least add "Optionally, ...". I don't think there's any future reason we'd need them to be registered.

Test Plan:
This is a bit tough to test without multiple hosts, but I added this piece of code to `AlmanacKeys` so we'd pretend to be a nameless "web" device when creating a repository:

```
if ($_REQUEST['__path__'] == '/diffusion/edit/form/default/') {
  return null;
}
```

Then I created some Git repositories. Before the patch, they came up with `-` versions (no version information). After the patch, they came up with `0` versions (correctly initialized).

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12893

Differential Revision: https://secure.phabricator.com/D18273
2017-07-25 05:12:10 -07:00
Chad Little
69a7d57c3f Add a branch selector to Diffusion
Summary: Fixes T12931. Adds a branch selector that's always visible if the repo has commits.

Test Plan:
Test a plain hg, svn, git repository. Test setting a bad default branch. Test a good default branch. Test on desktop, mobile layouts.

{F5058061}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12931

Differential Revision: https://secure.phabricator.com/D18267
2017-07-24 13:41:23 -07:00
Aviv Eyal
27a243cd88 Link to docs from metamta.mail-adapter config
Test Plan: Look at new config page, click link.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D18259
2017-07-21 00:51:30 +00:00
epriestley
018d1b77bf Identify compound short search tokens in the form "xx.yy" as unqueryable in the search UI
Summary:
Ref T12928. The index doesn't work for these, so show the user that there's a problem and drop the terms.

This doesn't fix the problem, but makes the behavior more clear.

Test Plan:
{F5053703}

{F5053704}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12928

Differential Revision: https://secure.phabricator.com/D18254
2017-07-20 14:24:09 -07:00
Chad Little
2f26dd76de Lots of little fixes for Dark Mode (Experimental)
Summary: Cleans up a bunch of Differential odd/special colors. Adds some basic "highlight" colors instead of pure yellow.

Test Plan: Test each color change in normal and dark modes.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18239
2017-07-19 14:41:23 -07:00
Chad Little
db546e5558 Fix Pholio new mock feed story
Summary: Gives some strength to name (needed to over-ride new images) and new create copy.

Test Plan: Create a new mock, see proper story in feed.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18246
2017-07-19 10:52:59 -07:00
Chad Little
652d87ac6b Fix project creation feed story
Summary: Adds feed story copy to project create transactions.

Test Plan: Create a new project, visit manage page, feed, see correct text.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18245
2017-07-19 10:41:07 -07:00
epriestley
4d1ed12a9e Skip Conduit call log writes in read-only mode, allowing "conduit.ping" to run
Summary: Ref T10769. See PHI8. We have an unconditional logging write which we can skip in read-only mode.

Test Plan:
  - Put Phabricator in read-only mode with `cluster.read-only`.
  - Called `conduit.ping` via web UI.
    - Before: write-on-read-only exception.
    - After: good result.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T10769

Differential Revision: https://secure.phabricator.com/D18233
2017-07-18 13:32:52 -07:00
Aviv Eyal
d1f144b214 Fix Search Application Config
Summary: Fix T12924. Looks like this melted in D17384, and nobody noticed yet.

Test Plan: Visit page, see fancy table.

Reviewers: epriestley, 20after4, #blessed_reviewers

Reviewed By: epriestley, 20after4, #blessed_reviewers

Subscribers: Korvin

Maniphest Tasks: T12924

Differential Revision: https://secure.phabricator.com/D18230
2017-07-18 17:44:56 +00:00
Chad Little
7aeefc0cca Add an Experimental Dark Mode to Phabricator
Summary: Mostly this is an exercise to clean up our CSS and Celerity processor by making sure all important color decisions are generatable. It's somewhat resonable to use if you don't review code. Posting it up here mostly so I don't lose the work.

Test Plan: Visit lots and lots of pages with dark mode on and off.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18227
2017-07-18 06:44:32 -07:00
epriestley
b8cd5b0eb8 Use a less-esoteric spelling of "capabilities" in several places
Summary: This spelling can definitely feel a little overplayed at times, but I still think it's a gold standard in spellings of "capabilities".

Test Plan: Felt old and uncool.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18215
2017-07-12 15:27:57 -07:00
Chad Little
0c4cff28df Clean up NUX a bit on Diffusion
Summary: Just some cleanup. Make sure action-bar has consistent space if locate is there or not, hide tabs if repository has no content. Use clone or checkout language depending on SCM. Fixes T12915.

Test Plan:
Test git, hg, svn blank states.

{F5042707}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12915

Differential Revision: https://secure.phabricator.com/D18208
2017-07-12 07:05:33 -07:00
Chad Little
db57da0f74 Fix SVN form_box error
Summary: Fixes T12915.

Test Plan: Test a SVN repository locally, ensure page loads.

Reviewers: epriestley

Subscribers: Korvin

Maniphest Tasks: T12915

Differential Revision: https://secure.phabricator.com/D18207
2017-07-11 19:51:34 -07:00
Chad Little
a6b550ba03 Move Clone Repository to Dialog
Summary: This moves the clone details on the Repository Home to a button / dialog. Functionally this is to pull content on the page way up, while giving full space to all the clone options. I think we can build this into some FancyJS if needed, but this seems to clean ui the UI dramatically with little overhead. I don't want to attempt the JS dropdown unless we're sure that's the best path (it exposes the most common URI by default, saving a click).

Test Plan: Tested hg, svn, git repositories and the raw URL page. Test close button.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18203
2017-07-11 13:16:47 -07:00
Chad Little
b987b4dd64 Rudamentary PHUILeftRightView
Summary: First pass at providing a skeleton framework for laying out basic items in a left/right view. Will likely add some mobile-responsive options.

Test Plan: UIExamples

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18200
2017-07-10 18:19:57 -07:00
Chad Little
af71c990ee Test 0 and "" cases in Project Icon Config
Summary: Better validation for setting a default image in project.icon

Test Plan: Test adding `"0"` and `""` as image options in project.icon, see error.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18197
2017-07-10 12:01:22 -07:00
Chad Little
5f1a359a92 Add a UIExamples page for new project image builtins
Summary: Adds a page to view all and their path in UIExamples.

Test Plan: Review page in UIExamples, hover over image for path.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18196
2017-07-10 11:22:23 -07:00
Chad Little
646ad36b15 Move actions into Diffusion header
Summary: This moves actions into the Diffusion main header, removes the locate file box, and widens description and cloning details. Projects are not currently in this layout, but will follow up in another diff. Trying to keep these changes small and iterative.

Test Plan:
Locate some files, test actions dropdown, repository with and without description. Also tablet, mobile layouts.

{F5040026}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18193
2017-07-10 06:51:40 -07:00
Chad Little
250aaabd64 Choose default project image by icon
Summary: Builds out a map for icon->image in Projects, selects the icon's image as the default project image if there is no custom image chosen by the user.

Test Plan: Select various icons, see image change. Test choose picture, pick a new image.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18174
2017-07-09 11:41:02 -07:00
Chad Little
39e5da7ea7 Simplify Diffusion Browse Table
Summary: Cleans up colors, removes commit hash and links the text instead. Also unsure how valuable "lint" column is here, but left it. I'd maybe like to understand that workflow since it just seems like clutter overall. Also Fixes T12905

Test Plan:
Review Phabricator, hg, and a few other test repositories locally. Holler if anything here seems bad, but this feels easier to read and use to me.

{F5038425}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12905

Differential Revision: https://secure.phabricator.com/D18189
2017-07-09 09:43:57 -07:00
Chad Little
0bd1dfd524 Add 8 more project images to builtins
Summary: More pretty images.

Test Plan: Set a robot as image for security project. So pretty.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18191
2017-07-09 09:39:10 -07:00
Chad Little
0b3117bb68 Fix comparison check for SVN in browsing Diffusion
Summary: Fixes T12905. Missed setting this variable.

Test Plan: Browse an SVN repository.

Reviewers: epriestley

Subscribers: Korvin

Maniphest Tasks: T12905

Differential Revision: https://secure.phabricator.com/D18190
2017-07-09 06:43:43 -07:00
epriestley
301750f6e6 Add an after-purchase hook to subscriptions in Phortune
Summary:
Ref T12681. We need this to update the "paid until" window on support pacts.

(Instance billing doesn't use this because everything just checks if you have unpaid invoices, nothing actually happens when you pay them.)

Test Plan: See D18187.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12681

Differential Revision: https://secure.phabricator.com/D18188
2017-07-07 16:39:47 -07:00
epriestley
8d11e127ff Fix several pieces of UI language describing "draft/archive" rules in Phame
Summary: Ref T12900. We implement one rule, but tell users a different (older) rule. See T12900 for discussion and history.

Test Plan:
  - Verified draft/archived posts can't be seen by users who don't have permission to edit the blog.
  - Drafted, archived, and published posts and read the related text.
  - Looked through the changes I dug up in T12900#228748 for other strings I might have missed.

{F5033860}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12900

Differential Revision: https://secure.phabricator.com/D18182
2017-07-06 08:13:53 -07:00
Chad Little
e516358d54 Add tabs to Diffusion for consistent navigation
Summary:
Adds a responsive tab bar navigation to Diffusion. Working through the new design here in pieces, so keep in mind M1477 is the target. Notably:

- Removes "branches" and "tags" from RevisionView, now on tabs
- Keeps "browse", "history", "readme" on RevisionView
- Adds tabs for all main views, including Graph... unless how that feels, so let me know.

Test Plan: Browse all pages, desktop and mobile. Test hg, svn, git repositories.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18161
2017-07-05 22:09:36 +01:00
epriestley
a6f0182104 Fix an issue where repositories with hyphens could sort improperly in typeaheads
Summary: Fixes T12894. See that task for discussion.

Test Plan:
  - Created repositories `abcdef`, then `abcdef-a` through `abcdef-f`.
  - Before patch, awkward sort order.
  - After patch, query for `abcdef` hits `abcdef` first.
  - See T12894 for details and screenshots.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12894

Differential Revision: https://secure.phabricator.com/D18179
2017-07-03 09:28:02 -07:00
Chad Little
3536fe2877 Update Diffusion conduit text
Summary: Fixes T12888

Test Plan: Verify text is there.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12888

Differential Revision: https://secure.phabricator.com/D18178
2017-07-02 14:25:10 +00:00
Chad Little
b25b379ca0 Move Diffusion Browse to a single column layout
Summary: The main change here is moving (compare, search, history) into buttons in the header bar on all browse views. This allows Directory Browsing to be full width, since there is no other curtain information. File, Image, LFS, Binary all stay in TwoColumn layouts with the same buttons in the header.

Test Plan: Test viewing a directory, file, image, binary file, readme, and fake a gitlfs.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17766
2017-07-01 20:45:56 +02:00
epriestley
eab8d8a22c Use the correct "completed" time in Harbormaster display UI
Summary: Fixes T12883. The task seems correct to me and I think this is a copy/paste mistake that probably blames to me.

Test Plan: Fiddled these numbers, viewed a build in Harbormaster, saw the adjusted time.

Reviewers: chad, amckinley

Reviewed By: chad

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12883

Differential Revision: https://secure.phabricator.com/D18177
2017-06-30 07:11:41 -07:00
epriestley
4e047f7b31 Correct a datasource issue when viewing repository URIs in "Manage Repository"
Summary:
Fixes T12884. In cases other than this UI, applications access URIs through the Repository they're part of. This means that applications interact with URIs which have gone through the correction/adjustment logic in `PhabricatorRepository->attachURIs()`, which fixes up "builtin" URIs to have the right values based on configuration.

In this case (and, as far as I can tell, only this case) we load the URI directly //and// act on its properties which depend on configuration and repository state.

This can mean we're using a different view of the URI than we should be.

To fix this: after loading the URI, reload it through the repository so the relevant adjustments are applied.

I think this is the most reasonable fix. We could try to make `RepositoryURIQuery` somehow enforce this, but the cost of this error is small (mild confusion about display state), the other things which do direct loads don't depend on this state (editing), and everything else loads via a repository and is likely to continue doing that forever.

Test Plan: {F5026633}

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12884

Differential Revision: https://secure.phabricator.com/D18176
2017-06-30 07:09:53 -07:00
epriestley
596b83a712 Move misplaced validation for ambiguous fields in "Test Plan" to the right place
Summary:
When users use the web UI to enter text like "Reviewers: x" into the "Summary" or "Test Plan", we can end up with an ambiguous commit message.

Some time ago we added a warning about this to the "Summary" field, and //attempted// to add it to the "Test Plan" field, but it actually gets called from the wrong place.

Remove the code from the wrong place (no callers, not reachable) and put it in the right place.

This fixes an issue where users could edit a test plan from the web UI to add the text "Tests: ..." and cause ambiguities on a later "arc diff --edit".

Test Plan: {F5026603}

Reviewers: chad, amckinley

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18175
2017-06-30 06:36:05 -07:00
epriestley
b46e2bb4cc Convert cluster/projects config options to newer modular structure
Summary: Ref T12845. Converts the cluster and project config options to the new stuff; this is mostly just shifting boilerplate around.

Test Plan: Edited, deleted, and mangled these options from the web UI and CLI.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18166
2017-06-27 12:35:54 -07:00
epriestley
6984d239b0 Convert Maniphest custom config to new config types
Summary:
Fixes T12870. Ref T12845.

Technically, this addresses the core issue in T12845 too, but I'm going to convert the rest of the `custom:...` types before closing that.

In particular, for T12870:

  - Validates that keywords are unique across priorities.
  - Fixes missing newline in documentation.
  - Updates documentation to note that keywords are now mandatory and must be unique across priorities.

Test Plan: Edited, deleted and mangled all the Maniphest custom options (priorities, statuses, points, subtypes).

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12870, T12845

Differential Revision: https://secure.phabricator.com/D18165
2017-06-27 12:35:29 -07:00
epriestley
a14b82d4f4 Move "wild" config types to new code
Summary:
Ref T12845. This is the last of the hard-coded types.

These are mostly used for values which users don't directly edit, so it's largely OK that they aren't carefully validated. In some cases, it would be good to introduce a separate validator eventually.

Test Plan: Edited, deleted and mangled these values via the web UI and CLI.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18164
2017-06-27 12:34:56 -07:00
epriestley
ec2af08625 Move 'set' config option type to new structure
Summary: Ref T12845. This move 'set' options (a set of values).

Test Plan: Set, deleted and mangled 'set' options from CLI and web UI.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18160
2017-06-27 12:34:37 -07:00
epriestley
0afdabff00 Convert 'class' config options to new validation
Summary: Ref T12845. These options prompt the user to select from among concrete subclasses of some base class.

Test Plan: Set, deleted and mangled these values from the web UI and CLI.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18159
2017-06-27 12:14:19 -07:00
epriestley
72119e786c Convert "bool" config values to new modular system
Summary: Ref T12845. Moves the "bool" values over.

Test Plan: Set, deleted, and mangled bool values from CLI and web UI.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18158
2017-06-27 12:13:55 -07:00
epriestley
467be5e53f Convert the "list<string>" and "list<regex>" Config option types
Summary: Ref T12845. This updates the "list<string>" and "list<regex>" options.

Test Plan: Set, deleted, and mangled options of these types from the web UI and CLI.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18157
2017-06-27 12:13:33 -07:00
epriestley
9d30d49cfc Convert "enum" and "string" config options to new modular option types
Summary: Ref T12845. This moves the "enum" and "string" types to the new code.

Test Plan: Set, deleted, and tried to set invalid values for various enum and string config values (header color, mail prefixes, etc) from the CLI and web.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18156
2017-06-27 12:13:15 -07:00
epriestley
03b6bdde19 Begin modularizing config options in a more modern way
Summary:
Ref T12845. Config options are "modular", but the modularity is very old, half-implemented, and doesn't use modern patterns.

Half the types are hard-coded, while half the types are semi-modular but in a weird hacky way where you prefix the type with `custom:...`.

The actual API is also weird and requires types to return a lot of `array($stuff, $thing, $other_thing, $more_stuff)` sorts of tuples.

Instead:

  - Add a new replacement layer which uses modern modularity patterns and overrides the older stuff if available, so we can migrate things one at a time.
  - New layer uses a more modern API -- no `return array($thing, $other_thing, ...)`, and more modern building blocks (like AphrontHTTPParameterType).
  - New layer allows custom types to be deleted, which will ultimately let us deal with T12845.

Then, convert the `'int'` type to use the new layer.

Test Plan:
  - Set, edited, tried-to-change-in-an-invalid-way, and deleted an `'int'` option from the web UI.
  - Same from the CLI.
  - Edited `config.json` to have an invalid value, verified that the error was detected and config was repaired.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12845

Differential Revision: https://secure.phabricator.com/D18155
2017-06-27 12:12:37 -07:00
Chad Little
d6450bf7d2 New icons for Projects
Summary: Updates the builtin images, leaves the old choose... icons for now. I'd like to automate this based on icon when creating a project.

Test Plan: Visit edit picture page, pick a few. Purge cache, see new default image.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18162
2017-06-27 15:02:20 +02:00
epriestley
d4632f4b78 Restore "Land Revision" action to UI
Summary: This was accidentally caught in the crossfire in D18150. This is stable enough to formalize instead of adding with an event hook.

Test Plan: Looked at a candidate revision, saw "Land Revision" appear in UI again.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18154
2017-06-26 06:50:05 -07:00
epriestley
a198590533 Degrade more gracefully when ProfileMenu dashboards fail to render
Summary:
Ref T12871. This replaces a dead end UI (user totally locked out) with one where the menu is still available, if the default menu item is one which generates a policy exception (e.g., because users can't see the dashboard).

Really, we should do better than this and not select this item as the default item if the viewer can't see it, but there is currently no reliable way to test for "can the viewer see this item?" so this is a more involved change. I'm thinking we get this minor improvement into the release, then pursue a more detailed fix afterward.

Test Plan:
  - Added a dashboard as the top item in the global menu.
  - Changed the dashboard to be visible to only user B.
  - Viewed Home as user A.
  - Before patch: entire page is a policy exception dialog.
  - After patch, things are better:

{F5014179}

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12871

Differential Revision: https://secure.phabricator.com/D18152
2017-06-23 12:31:36 -07:00
epriestley
f704f905d2 Let PhabricatorSearchCheckboxesField survive saved query data with mismatched types
Summary:
Fixes T12851.

This should fix the error I'm seeing, which is:

* `Argument 1 passed to array_fuse() must be of the type array, boolean given`

There may be a better way to patch this up than overriding the getValue() method,
however.

Test Plan:
- Changed the default "Tags" filter to specify `true` instead of `array('self')`, then viewed that filter in the UI.
- Before patch: fatal.
- After patch: page loads. Note that `true` is not interpreted as `array('self')`, but the page isn't broken, which is a big improvement.

Reviewers: #blessed_reviewers, 20after4, chad, amckinley

Reviewed By: #blessed_reviewers, amckinley

Subscribers: Korvin

Maniphest Tasks: T12851

Differential Revision: https://secure.phabricator.com/D18132
2017-06-23 12:29:47 -07:00
epriestley
219ae8b6c9 Remove old "Landing Strategy" code
Summary:
Fixes T12869. This is a very old, pre-Drydock chunk of code from D7486 and some followups.

It does three things:

  - "Land to Hosted Git": Obsoleted by Drydock, has been commented out in HEAD for a very long time with no complaints. Disabled by D8719 in 2014.
  - "Land to Hosted Mercurial": Could be obsoleted by Drydock with a fairly small amount of work, but currently has no replacement. Unclear if this sees any real use. Not actually disabled at HEAD.
  - "Land to GitHub": Use GitHub OAuth credentials to land to GitHub. This is sort of theoretically useful and has no analog today. Disabled by D13022 in 2015.

This stuff was largely disabled a long time ago and we haven't seen users hitting issues with it. This could all be moved to an extension today if anyone still relies on it.

Test Plan: Grepped for removed classes, browsed Differential.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12869

Differential Revision: https://secure.phabricator.com/D18150
2017-06-23 08:13:53 -07:00
epriestley
224c4692ee Add a cache purger for builtin files
Summary: Fixes T12859.

Test Plan:
  - Loaded Diffusion builtin icons before recent updates, saw cached builtins.
  - Ran `bin/cache purge --caches builtin-file`.
  - Reloaded, saw new files.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12859

Differential Revision: https://secure.phabricator.com/D18147
2017-06-22 11:13:23 -07:00
epriestley
bd3f441098 Modularize "bin/cache" purgers
Summary: Ref T12859. This is an older command with a lot of hard-coded flags. Modularize cache purging in a modern way so it can be extended.

Test Plan: Ran `bin/cache purge --trace` with various valid and invalid flags.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12859

Differential Revision: https://secure.phabricator.com/D18146
2017-06-22 11:13:05 -07:00
epriestley
519bec3e6f Make searching by tags work in Phriction
Summary: Fixes T12860. Some joins were being dropped because we didn't call `parent::...`

Test Plan:
  - Tagged a document.
  - Searched for documents with that tag.
  - Before change: got all documents.
  - After change: got only tagged documents.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12860

Differential Revision: https://secure.phabricator.com/D18145
2017-06-22 11:01:31 -07:00
epriestley
149e6aaa21 Let "<select />" EditEngine fields canonicalize saved defaults
Summary:
Ref T12124. After D18134 we accept either "25" or "low" via HTTP parameters and when the field renders as a control, but if the form has a default value for the field but locks or hides it we don't actually run through that logic.

Canonicalize both when rendering the control and when using a raw saved default value.

Test Plan:
  - Created a form with "Priority: Low".
  - Hid the "Priority" field.
  - Before patch: Tried to create a task, was rebuffed with a (now verbose and helpful, after D18135) error.
  - Applied patch: things worked.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12124

Differential Revision: https://secure.phabricator.com/D18142
2017-06-20 17:42:33 -07:00
epriestley
17fc447503 Don't compute MIME type of noninitial chunks from diffusion.filecontentquery
Summary:
Ref T12857. This is generally fairly fuzzy for now, but here's something concrete: when we build a large file with `diffusion.filecontentquery`, we compute the MIME type of all chunks, not just the initial chunk.

Instead, pass a dummy MIME type to non-initial chunks so we don't try to compute them. This mirrors logic elsewhere, in `file.uploadchunk`. This should perhaps be centralized at some point, but it's a bit tricky since the file doesn't know that it's a chunk until later.

Also, clean up the `TempFile` immediately -- this shouldn't actually affect anything, but we don't need it to live any longer than this.

Test Plan:
  - Made `hashFileContent()` return `null` to skip the chunk cache.
  - Added `phlog()` to the MIME type computation.
  - Loaded a 12MB file in Diffusion.
  - Before patch: Saw 3x MIME type computations, one for each 4MB chunk.
  - After patch: Saw 1x MIME type computation, for initial chunk only.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12857

Differential Revision: https://secure.phabricator.com/D18138
2017-06-20 05:45:20 -07:00
epriestley
474d528c3b Allow numeric constants to act as aliases for task priorities in the web UI <select />
Summary:
Ref T12124. This is a fairly narrow fix for existing saved EditEngine forms with a default priority value.

These saved forms have a numeric (or probably "string-numeric") default value, like "50". They lost their meaning after D18111, when "50" no longer appears in the dropdown. Instead, these forms all select the highest available priority.

At time of writing, this form was broken on this install, for example:

> https://secure.phabricator.com/transactions/editengine/maniphest.task/view/13/

Additionally, `/task/edit/form/123/?priority=...` (for templating forms) stopped working with `priority=50`. This isn't nearly as important, but a larger and more sudden compatiblity break than we need to make.

Add support for an "alias map" on `<select />` controls, so if the value comes in with something we don't recognize we'll treat it like some other value. Then alias all the numeric constants -- and other keywords -- to the right constants.

This ended up only affecting the `<select />` control in the web UI.

Test Plan:
  - On `stable`, created a form with "Priority: Low".
  - Before patch: form has "Priority: Unbreak Now!" on `master`.
  - After patch: form has "Priority: Low" again.
  - Used `?priority=25`, `?priority=wish`, `?priority=wishlist` to template forms: all forms worked.

Reviewers: amckinley, chad

Reviewed By: amckinley

Maniphest Tasks: T12124

Differential Revision: https://secure.phabricator.com/D18134
2017-06-19 14:06:34 -07:00
epriestley
cd19ddf111 Improve validation errors for changing task priorities
Summary:
Ref T12124. Currently, Conduit provides a fairly rough error message if you provide an invalid priority.

Instead, provide a more tailored message. Also, block `!!unknown!!` except from web edits.

Test Plan:
Before:

{F5007964}

After:

{F5007965}

Also, changed a priority to `999` in the database, edited it with the normal web UI form, it let me make the edit without being forced to adjust the priority.

Reviewers: amckinley, chad

Reviewed By: amckinley

Maniphest Tasks: T12124

Differential Revision: https://secure.phabricator.com/D18135
2017-06-19 14:05:47 -07:00
Chad Little
d0898116d8 Add a graph view page to Diffusion
Summary: Fixes T12840. This adds a parallel "graph" button next to history on home and on the history list page. I'll think more about better placement of how to get to this page with the upcoming redesign that's still sitting in Pholio.

Test Plan: View History, View Graph, Try pager, go to a file, click view history, see no graph button.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12840

Differential Revision: https://secure.phabricator.com/D18131
2017-06-19 17:57:20 +02:00
epriestley
9b93697d52 Move "List Inline Comments" to the inline header dropdown menu
Summary: See D18128. Ref T12733. Ref T8250.

Test Plan: {F5003153}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12733, T8250

Differential Revision: https://secure.phabricator.com/D18129
2017-06-15 07:11:30 -07:00
epriestley
b3b30dde6a Add options for hidding inlines to the Differential header banner
Summary:
Fixes T8909. Ref T12733.

UI attempts to follow the mock, but is a bit rough since PHUIXButtonView without text in this menu gets weird spacing, we don't have circular buttons yet, and PHUIXActionView without an icon also gets odd spacing.

Test Plan: {F5003125}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12733, T8909

Differential Revision: https://secure.phabricator.com/D18128
2017-06-15 05:23:14 -07:00
epriestley
887bd2d66e In the UI, rename "Hide Inline" to "Collapse Inline"
Summary:
Ref T12733. This paves the way for a separate "hide" operation which completely hides things.

(I didn't extend this to the server side because that would require schema changes and the new "hide" state is client-only.)

Test Plan: Collapsed and expanded inlines, viewed tooltips.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12733

Differential Revision: https://secure.phabricator.com/D18126
2017-06-15 05:22:44 -07:00
Austin McKinley
8008ade9af Use keywords instead of ints to update task priority in ManiphestEditEngine
Summary: Fixes T12124. Changes `ManiphestEditEngine` to populate the select using priority keywords instead of the integer value. Marks `maniphest.querystatuses` as frozen. Adds a new Conduit method for fetching potential task statuses.

Test Plan: Created tasks and changed their priorities, observed that transactions in the DB still have the same type (integers as strings). Invoked `maniphest.update` with `priority => '90'` and observed that it still works. Invoked `maniphest.edit` with `priority => 'unbreak'` and observed that it now works.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T12124

Differential Revision: https://secure.phabricator.com/D18111
2017-06-14 14:43:03 -07:00
epriestley
3d70db9eb5 Queue a worker task to send mail only after committing the mail transaction
Summary:
Fixes T12844. This code is misleading: the daemon insert is happening on a different connection, and is not inside the transaction on the Mail connection.

What actually happens is this:

  - (Connection A) `BEGIN`
  - (Connection A) `INSERT INTO mail ...`
  - (Connection B) `INSERT INTO worker ...` <-- This is a different connection, and it is NOT in a transaction!
  - There's a race window here: the worker row is globally visible but the mail row is still isolated inside the transaction.
  - (Connection A) `COMMIT`
  - Now we're clear: the mail row is globally visible.

Change this code to reflect what's actually happening.

This means that if the worker row insert fails for some reason, we'll now throw with a mail row written to the database. But this is fine: it doesn't send on its own (so it can't cause mail loops or anything) and it can be re-queued with `bin/mail resend` if necessary without too much trouble.

Test Plan: See T12844 for particulars. Made some comments on tasks, saw the daemons send mail.

Reviewers: chad, amckinley, jmeador

Reviewed By: jmeador

Maniphest Tasks: T12844

Differential Revision: https://secure.phabricator.com/D18124
2017-06-14 12:27:00 -07:00
Mukunda Modell
e1850b3c4e Allow dashboard panels to be found by monogram
Summary:
Just add the monogram to the datasource's `name` field
so that it will match when typing Wnn in the typeahead field.

Test Plan:
Tested locally on my dev phab. Try searching for a panel
by monogram in the 'Add existing panel' dialog on the
'arrange workboard' interface.

Previously: typing W123 showed no results.

After this change: typing W123 finds the panel W123

Reviewers: chad, epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18121
2017-06-13 16:48:36 -05:00
Chad Little
6f7b31fbf8 Add a DiffusionTagListView
Summary: Moves DiffusionTagsListView to uhhh, list. Separates out table view which is still in use now, implements mobile friendly UI for tags.

Test Plan:
Review KDE's Krita repository locally with lots of tags, desktop and mobile.

{F4997708}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12824

Differential Revision: https://secure.phabricator.com/D18115
2017-06-13 11:32:18 -07:00
Chad Little
df6ad07566 Add DiffusionBranchListView for browsing branches
Summary: Adds a new DiffusionBranchListView which replaces the BranchTable when browsing all branches in Diffusion. Has all the same capabilities, but is easier to read, adds a Compare button, and plays nicely on mobile. It does take up more space, but I think that's generally OK here since we expect our branches to not be heaping piles of intern revert branches.

Test Plan:
Follow a few repositories with branches, like Phabricator and KDE's Krita. View layouts on mobile, tablet, desktop. Try out new compare button.

{F4996207}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: avivey, Korvin

Maniphest Tasks: T12824

Differential Revision: https://secure.phabricator.com/D18113
2017-06-13 11:07:03 -07:00
Chad Little
283a95d2aa Build a page for viewing all inline comments
Summary: Adds a very basic list of all inline comments, threaded, and their status. Kept this a little simpler than the mock, mostly because sorting here feels a little strange given threads would be all over the place. Not sure sorted is needed in practice anyways. I'd probably lean towards just adding a JS checkbox to hide certain rows if needed in the future.

Test Plan:
Test various commenting structures:

 - Leave Comment
 - Update Diff
 - Leave new comment
 - Reply to comment
 - Reply to comment as revision author
 - Mark items as done
 - Update diff again

{F4996915}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D18112
2017-06-12 11:31:20 -07:00
epriestley
4b15871ced Fix specification of ttl.relative via LFS
Summary: Fixes T12828. This API was tightened up D17616, but I missed this callsite.

Test Plan: I've just got a good feeling in my bones.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12828

Differential Revision: https://secure.phabricator.com/D18119
2017-06-12 11:24:20 -07:00
epriestley
0610b86f57 Allow Herald rules to apply "only the first time" to Calendar events
Summary: Fixes T12821. (Some events only happen once, so the default behavior doesn't let you pick this rule, and objects must opt into it by saying "yeah, I support multiple edits".)

Test Plan:
Note "only the first time" selected:

{F4999093}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12821

Differential Revision: https://secure.phabricator.com/D18117
2017-06-12 11:23:05 -07:00
Chad Little
2a3022913f Add a tooltip method to PHUIIconView
Summary: We seem to use these a lot. Makes the code cleaner.

Test Plan: UIExamples.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18114
2017-06-12 18:00:04 +00:00
Chad Little
83a89166ee Add profile images to Repositories
Summary: Builds out some images to use to identify repositories. Fixes T12825.

Test Plan:
Try setting custom, built in, and null images.

{F4998175}

{F4998192}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12825

Differential Revision: https://secure.phabricator.com/D18116
2017-06-12 07:51:39 -07:00
epriestley
3400f24c8b Send permanent dameon failures to the log, even when not running in verbose mode
Summary:
Fixes T12803. An install is having difficulty diagnosing mail failures, and one component is that permanent task failures aren't reaching the log.

It's reasonable to send these to the log even when "phd.verbose" is off. See T12803 for a rough review of when we generate these failrues today.

Test Plan:
  - Faked some exceptions.
  - Got a result in the log (P2058) with `phd.verbose` turned off.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12803

Differential Revision: https://secure.phabricator.com/D18106
2017-06-08 15:26:19 -07:00
epriestley
d5163d0143 Write patterns to "git grep" on stdin instead of passing them with "-e"
Summary:
Fixes T12807. Some shells may apparently mangle/strip UTF8 characters? Just dodge this whole problem by sending the pattern over stdin rather than actually figuring out the particulars.

Related tasks, like T7339 and T5554, discuss finding broader fixes for this class of issue, and this definitely isn't exactly a fully legitimate fix, but in many cases (as here) we can reasonably just avoid the problem rather than actually fixing it, at least for a long time.

Test Plan: Searched for emoji and non-emoji locally, but this worked fine (on OSX) for me before the patch too.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12807

Differential Revision: https://secure.phabricator.com/D18105
2017-06-08 15:25:55 -07:00
epriestley
e26cd3ffe0 Give "x committed <commit>" feed stories an explicitly higher action strength than other transactions
Summary:
Fixes T12811. The issue here //appears// to be that both the "alice committed rXYZabc" and "Herald added projects..." actions have the same (default) strength and end up applying in arbitrary order, and probably got shuffled around as this transitioned to Modular transactions.

Give "alice committed rXYZabc" an explicitly higher action strength.

Test Plan:
  - Wrote an "Always, add project X" Herald rule for commits.
  - Ran `bin/repository reparse --herald ...`.
  - Saw an "alice committed rXYZabc" story instead of a "Herald added projects: X" story.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12811

Differential Revision: https://secure.phabricator.com/D18104
2017-06-08 15:25:39 -07:00
Austin McKinley
fb9d036e57 Show task duplicates as related objects in Maniphest and migrate old duplicates
Summary: Does the UI work that's part of T12234 and adds migrations for both of the old-style duplicate transactions.

Test Plan:
 - Started with a clean DB.
 - Checked out really old code that marks tasks as dupes using comments.
 - Made a bunch of tasks and closed some as dupes. Made a bunch of additional comments.
 - Checked out D10427 and did a `storage upgrade`.
 - Made a bunch more new tasks and dupes.
 - Snapshotted DB.
 - Ran migration repeatedly until all expected edges showed up in the `phabricator_maniphest.edge`table.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T12234

Differential Revision: https://secure.phabricator.com/D18037
2017-06-07 13:30:20 -07:00
Chad Little
38904ea4ad More button grey conversions
Summary: Ran across a few straglers. Convert to the correct color.

Test Plan: grep for profile-image-button, check profile image selection page

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18096
2017-06-07 09:34:50 -07:00
Chad Little
7c955d795e Remove badge support from PHUIHeaderView
Summary: These are now unused.

Test Plan: grep, remove uiexamples

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18090
2017-06-07 13:27:26 +00:00
Chad Little
b7f147ea0f Clean up user profile commit list view
Summary: Porting over a fix that we could miss the tail end of commits. Also use the new tag borderless option.

Test Plan: Review various commit pages in profile.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18086
2017-06-06 11:07:04 -07:00
Chad Little
ece651255c Optimize mobile layout of DiffusionHistoryView
Summary: Little nits and spacing changes to viewing diffusion commit history on phones.

Test Plan:
Review in Chrome, iOS Simulator.

{F4990749}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18085
2017-06-06 10:29:11 -07:00
Chad Little
d3c464a610 Separate button CSS classes
Summary: Try to dis-ambiguate various button types and colors. Moves `simple` to `phui-button-simple` and moves colors to `button-color`.

Test Plan: Grep for buttons still inline, UIExamples, PHUIX, Herald, and Email Preferences.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18077
2017-06-05 20:14:34 +00:00
epriestley
48c6ca40c4 Add an "Unsaved" button to the Differential persistent header
Summary: Ref T12733.

Test Plan: {F4987956}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12733

Differential Revision: https://secure.phabricator.com/D18070
2017-06-05 09:48:08 -07:00
Chad Little
65c9d789d2 Add a borderless tag style
Summary: Formally support borderless tags in PHUITagView.

Test Plan: Used in Diffusion History List

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18074
2017-06-04 11:52:35 -07:00
epriestley
17f092307c Fix an issue where Phriction moves to new locations would fail with a "content required" error
Summary:
Ref T12793. I'd like to understand exactly when we broke this, but this seems to be a minimal fix that shouldn't do anything surprising.

When you move document `/a/` to `/a/b/` and that path doesn't exist yet, the Content transaction currently fails because there's "no content". The content gets added later by the "move" transaction but this is implicit.

To make this work, just ignore the "missing field" error. This is a little roundabout but unlikely to break anything in weird ways.

Test Plan:
  - Moved document `/a/b/` to `/a/b/c/`.
  - Before patch: error about missing content.
  - After patch: move worked properly.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12793

Differential Revision: https://secure.phabricator.com/D18069
2017-06-02 16:27:24 -07:00
Chad Little
0a3b391136 Add more simple button colors
Summary: Saturate the color a little more, add yellow

Test Plan: uiexamples

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18068
2017-06-02 17:32:39 +00:00
epriestley
5335f29ff2 Correct an issue where the commit list could group commits by server time
Summary:
Commits in the list are grouped by the date they occurred in server time. This may not be the date they occurred in client time.

Use client time, not server time, to group commits.

Test Plan:
- Set server timezone to "Asia/Famagusta".
- Set client timezone to "America/Los_Angeles".
- Viewed Phabricator repository history.

Here's what it looks like before the change:

{F4987094}

Note that the headers of the first two groups both say "Yesterday".

This is because the first commits in each group occurred on June 1 and June 2, respectively, in Famagusta, but both occurred on June 1 in Los Angeles.

Here's what it looks like after the change:

{F4987095}

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18067
2017-06-02 08:39:13 -07:00
epriestley
335c3a7d12 In commit history list view, show all commits
Summary:
Currently, the last group of commits is not shown in the list view because the final `$list` is never added to `$view`.

For example, if the first page would contain commits from "April 7", "April 6", and "April 5", commits from "April 5" are not shown.

(If a repository has 100 commits in a single day, nothing is shown.)

On this server, here's the bottom of page 1:

{F4987087}

Here's the top of page 2:

{F4987088}

However, here's `git log` between those commits:

```
$ git log --oneline 7e46^..5f49f
5f49f9c793 Add sound to logged out Conpherence
1644b45050 Disperse task subpriorities in blocks
c6a7bcfe89 Make Pholio description behave as a remarkup field (e.g., subscribe mentioned users)
bbc5f79227 Make membership lock/unlock feed stories read more naturally
789d57522b Make editing project images redirect to "Manage" more consistently
10b3879232 Make Project slug/hashtag transactions render a little more nicely
abd791889c Update Maniphest title transaction again
5a34b299e4 Update Maniphest title language
601622013d Clarify milestone/subproject creation language
c9889e3d55 Fix an issue in Phriction where moving a document just copied it instead
fdf00f6df4 Clean up some minor UI behaviors in Differential
6c46f27d98 Add quest objectives to the minimap
d783299a19 Fix Phriction status not set property on new document
93e28da76e Add more "disabled" UI to PHUIObjectItemView
7e46d7ab6a Migrate Project color to modular transactions
```

This group of commits does not currently appear anywhere in the list.

Test Plan: Viewed a page of commits, saw 100 commits.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18066
2017-06-02 08:38:29 -07:00
epriestley
d42d69aef6 Clean up some PHUI/PHUIX button behaviors
Summary:
Ref T12733. Some minor issues:

  - The `strlen(...)` test against `$this->text` fails if a caller does something like `setText(array(...))`. This is rare, but used in `DiffusionBrowseController`, from D15487.
  - Add PHUIX examples for icon-only buttons.
  - Remove unused `SIMPLE` constant now that no callsites remain.

Test Plan:
  - Viewed a directory in Diffusion's "Browse" view in a Git repository, no longer saw a warning / error log.
  - Viewed PHUIX Components UI examples.
  - Grepped for `::SIMPLE`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12733

Differential Revision: https://secure.phabricator.com/D18065
2017-06-02 08:37:52 -07:00
epriestley
9f8907ccf7 Make Owners behavior when multiple packages own the same path with different dominion rules more consistent
Summary:
Fixes T12789. See that task for discussion. Currently, when multiple packages own the same path but have different dominion rules we get some weird/aribtrary/inconsistent results.

Instead, implement these rules:

  - If zero or more weak and one or more strong packages claim a path, the strong packages (exactly) all own it.
  - If one or more weak packages and zero strong packages claim a path, the weak packages all own it.

The major change here is that instead of keeping the //first// weak package we run into, we keep all the weak packages with the longest claim that we run into.

This needs to be implemented twice because Owners has two different near-copies of this logic, only one of which has test coverage. Some day maybe this will get fixed.

Test Plan:
  - Added failing unit tests, made them pass.
  - Viewed all A/B strong/weak combinations in Diffusion, saw sensible ownership results.

Reviewers: chad, lvital

Reviewed By: lvital

Subscribers: lvital

Maniphest Tasks: T12789

Differential Revision: https://secure.phabricator.com/D18064
2017-06-01 16:42:09 -07:00
epriestley
b9a4988df3 Mark "Settings" and "Nuance" as launchable applications
Summary:
Fixes T12790. I don't think this was actually a regression, Settings just wasn't launchable before global settings (since it had no real landing page, and the profile menu always had a link) and didn't get marked launchable once we added them.

I also double-checked other un-launchable apps; Nuance is probably close enough to make launchable now while I'm in here.

Test Plan: Typed "settings" into global typeahead, got settings.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12790

Differential Revision: https://secure.phabricator.com/D18062
2017-06-01 12:40:25 -07:00
Chad Little
0d8aba8550 Revert some changes to Diffusion History List
Summary: Ref rPf2fcafb40dde94ddf4ee22716fea74fca0334a64#38208, I think this is a more usable layout. Gets rid of clippy, audit. Adds back Differential link as tag, Build Status as button.

Test Plan: Faked data on this for Differential, Builds, should all work though. Test on real and fake repositories.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18061
2017-06-01 12:39:25 -07:00
epriestley
b69174a4c8 Remove non-operational shouldHideFromFeed() from ManiphestTaskPointsTransaction
Summary:
See D18018. Ref T12787. This doesn't actually work; we started publishing these stories as a side effect of converting to ModularTransactions, then I fixed the rendering.

This mechanism has very few callsites and I suspect we may want to get rid of it (see T12787) so just keep publishing these stories for now.

Test Plan: Changed the point value of a task, saw a feed story both before and after the patch.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12787

Differential Revision: https://secure.phabricator.com/D18059
2017-06-01 09:14:20 -07:00
epriestley
995c1503e7 Hide "X created Y, a subtask of P." feed stories again
Summary: Fixes T12787. Modular Transactions don't actually support `shouldHideForFeed()`. I'll add some discussion to the task.

Test Plan: Created a subtask, saw no more "X reopened Y, a subtask of P" feed story.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12787

Differential Revision: https://secure.phabricator.com/D18058
2017-06-01 08:08:24 -07:00
Chad Little
fbb7673439 Diffusion History List cleanup
Summary: Removes the odd circle buttons, adds copy-pasta button.

Test Plan: Review new layout locally.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18057
2017-06-01 06:47:32 -07:00
Chad Little
c001781264 Allow buttons to just be icons
Summary: Let's buttons just be an icon, no pressure to also have text.

Test Plan: UIExamples, Search, Home, Policy Controls... Probably 99% of them.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18056
2017-06-01 06:37:42 -07:00
epriestley
f2fcafb40d Help PROFESSIONAL SOFTWARE ENGINEERS copy text to their clipboard
Summary: Ref T12780. I'd like 18,000 GitHub stars now please thank you

Test Plan: this feature is awful

Reviewers: chad

Reviewed By: chad

Subscribers: cspeckmim

Maniphest Tasks: T12780

Differential Revision: https://secure.phabricator.com/D18053
2017-05-31 10:13:49 -07:00
Chad Little
6295e37857 Have Browse button in History actually work
Summary: Ref T12780. Makes the button do something useful, like link to the history at the right spot in the graph.

Test Plan: Click on various browse buttons, get correct url.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12780

Differential Revision: https://secure.phabricator.com/D18054
2017-05-30 20:18:21 -07:00
epriestley
683647f1fb Add PHUIXButtonView and a UIExample
Summary:
Ref T12733. Ref M1476. This adds `PHUIXButtonView`, for client-side button rendering.

It also adds a PHUIX example which renders the server and client versions of each component side-by-side so it's easier to see if they're messed up.

Test Plan: {F4984128}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12733

Differential Revision: https://secure.phabricator.com/D18051
2017-05-30 18:01:16 -07:00
epriestley
2ad521b96d Categorize UIExamples a little bit
Summary:
Ref M1476. I'm planning to add some PHUIX examples, but sort out the existing examples a little first. I added some categories:

  - Catalogs: these are what I look at most often (emoji, icons, colors).
  - Single use: elements with only one use (badges, feed stories, hovercards, setup issues).
  - Technical: examples which are really just "test this thing in the browser" (avatars, gestures, notifications, remarkup).
  - Other: evrything else, mostly general-purpose multi-use components.

Test Plan:
(See left nav.)

{F4984042}

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18050
2017-05-30 18:00:46 -07:00
epriestley
7725d7cc45 Remove some old UIExamples
Summary: Ref M1476. I'm going to see if I can set up side-by-side "PHUI" vs "PHUIX" to make maintaining them a touch easier. Before doing that, nuke some really old UI examples that don't seem very useful.

Test Plan: Viewed UIExamples, saw fewer bad ones.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18049
2017-05-30 18:00:28 -07:00
epriestley
2c0dab055f Make "simple" a "button type", not a "color"
Summary:
Ref M1476. Currently, `setColor('simple')` is meaningful. Instead, `setButtonType('simple')`.

Depends on D18047.

Test Plan: Looked at UI examples, Phame, Auth. Notifications mooted by D18047.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18048
2017-05-30 17:59:37 -07:00
epriestley
d20221dc7d Remove Differential "objectives" UI
Summary: Ref T12733. Completely removes the objectives UI.

Test Plan:
  - Grepped for `objective`, etc.
  - Browsed revisions, no JS errors / broken stuff.
  - (If I missed anything, it's likely to turn up in followup changes.)

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12733

Differential Revision: https://secure.phabricator.com/D18043
2017-05-30 17:58:49 -07:00
Chad Little
c5bb69fd7d Use a list view for DiffusionHistory
Summary: This moves Diffusion History to use an easier to parse list view for commits and their (diff, audit, build) status. I left TableView around, which is used on a repositories home, and we can maybe add a "graph view" history back as another controller. Not sure what the real use is for that kind of feature though. I don't have Harbormaster set up locally so I could use another install to give this a run. I also expect to maybe not live with this UI as final, I like the UX, but the icons for indicating status don't really feel great to me, just OK.

Test Plan:
pull various repositories, check various history displays.

{F4980356}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18039
2017-05-30 17:31:48 -07:00
Chad Little
87c59c0867 Fix design atrocity
Summary: So bad.

Test Plan: Reload notification search page.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18047
2017-05-30 14:29:30 -07:00
Chad Little
88c5c02e72 Group Maniphest Tasks by Priority on Profiles
Summary: Ref T12423. Set the grouping by priority. Note this doesn't render headers but I don't want to spend a lot of time on this.

Test Plan: Review tasks in my sandbox, see them ordered by priority.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12423

Differential Revision: https://secure.phabricator.com/D18046
2017-05-30 14:15:44 -07:00
epriestley
7b290b94a7 Correct file PHID extraction on image update in Pholio
Summary: Ref T12776. This extraction of file PHIDs extracted "PholioImage" object PHIDs (`PHID-PIMG-...`), not "File" PHIDs (`PHID-FILE-...`). Instead, dig into the Pholio images and actually extract the file PHIDs. This method is now similar to the `PholioImageFileTransaction` method, which works already.

Test Plan:
  - Create a mock.
  - Update one of the images.
  - In Files, view the "Attached" tab of the updated image.
  - Before patch: not attached to mock.
  - After patch: properly attached to mock.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12776

Differential Revision: https://secure.phabricator.com/D18042
2017-05-30 12:00:05 -07:00
epriestley
c4e45c6c8c Detect and prevent invalid configuation of "ui.footer-items"
Summary: Fixes T12775. Currently, we do not validate this option and it's possible to configure it in an invalid way.

Test Plan: Tried to misconfigure things, was helpfully pointed toward errors.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12775

Differential Revision: https://secure.phabricator.com/D18041
2017-05-30 10:15:24 -07:00
Chad Little
3fb4ca2429 Add needActiveDiffs to differential.createcomment method
Summary: Ref T12766. Adds missing attachment for stacking actions in differential

Test Plan: Asked end user to verify patch.

Reviewers: epriestley, amckinley

Reviewed By: epriestley, amckinley

Subscribers: reed, Korvin

Maniphest Tasks: T12766

Differential Revision: https://secure.phabricator.com/D18038
2017-05-26 20:12:34 -07:00
Austin McKinley
9d37ad3022 Add maniphest.priority.search method
Summary: Start on plan outlined in T12124. Adds a new Conduit method for querying information about task priorities.

Test Plan: Ran locally; observed expected output: {F4979109}

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D18035
2017-05-26 16:02:39 -07:00
Austin McKinley
04fd93e51e Drop DifferentialDraft storage
Summary: Fixes T12104.

Test Plan: Ran `bin/storage upgrade` and observed table dun got dropped.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Maniphest Tasks: T12104

Differential Revision: https://secure.phabricator.com/D18034
2017-05-26 13:59:26 -07:00
epriestley
742c3a834f Provide UI hints about task subtypes
Summary:
Ref T12314. Open to counterdiffs / iterating / suggestions / skipping most or all of this, mostly just throwing this out there as a maybe-reasonable first pass.

When a task has a subtype (like "Plant" or "Animal"), provide some hints on the task list, workboards, and task detail.

To make these hints more useful, allow subtypes to have icons and colors.

Also use these icons and colors in the typeahead tokens.

The current rule is that we show the subtype if it's not the default subtype. Another rule we could use is "show the subtype if there's more than one subtype defined", but my guess is that most installs will mostly have something like "normal task" as the default subtype.

Test Plan:
The interfaces this affects are: task detail view, task list view, workboard cards, subtype typeahead.

{F3539128}

{F3539144}

{F3539167}

{F3539185}

Reviewers: chad

Reviewed By: chad

Subscribers: johnny-bit, bbrdaric, benwick, fooishbar

Maniphest Tasks: T12314

Differential Revision: https://secure.phabricator.com/D17451
2017-05-26 13:58:41 -07:00
Chad Little
81809713e0 Try layering state icons on PHUICircleIconView
Summary: I think this is reasonable for my current use case, but stacking icons overally is pretty clunky.

Test Plan:
UIExamples

{F4978899}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18032
2017-05-26 13:55:42 -07:00
Chad Little
aefc006ba5 Move DiffusionHistoryListView to DiffusionCommitListView
Summary: I think this name is more accurate, also add proper links to author image.

Test Plan: Review commits in sandbox, see new URL on image.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18026
2017-05-26 13:11:59 -07:00
Austin McKinley
b27c2ed6d1 Index Project milestones to accurately reflect milestone membership
Summary: Fixes T12505. `PhabricatorProjectsMembershipIndexEngineExtension->materializeProject()` was incorrectly bailing early for milestone objects, which prevented milestone members from being calculated correctly. This was causing problems where (for example) an Owners package owned by a milestone wasn't being satisfied when a member of the milestone approved a revision.

Test Plan: Invoked migration, observed that a user's milestones correctly showed up when searched for. Also observed that accepting a revision on behalf of a milestone now satisfies Owners rules.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12505

Differential Revision: https://secure.phabricator.com/D18033
2017-05-26 13:10:41 -07:00
epriestley
fc8465252f Add "View All" header buttons to user and project feed boxes
Summary:
Fixes T12762. Currently, there's no way to get from these boxes into generaly history in Feed, and it isn't clear that the operation is possible.

For now, add some simple links. See T12762 for future work.

Test Plan:
  - Viewed user profles, saw "View All".
  - Viewed project profiles, saw "View All".

{F4978858}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12762

Differential Revision: https://secure.phabricator.com/D18030
2017-05-26 12:24:20 -07:00
epriestley
46a33c07dc Allow users to query feed by a date range
Summary: Ref T12762.

Test Plan:
  - Ran queries with start date, end date, both, neither.
  - Used EXPLAIN to try to make sure doing the bitshift isn't going to be a performance issue.

{F4978842}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12762

Differential Revision: https://secure.phabricator.com/D18029
2017-05-26 12:23:56 -07:00
epriestley
6fed08a98e Modernize FeedSearchEngine a little bit
Summary:
Ref T12762. This updates FeedSeachEngine to user modern construction.

I've tried to retain behavior exactly, although the "Include stories about projects I'm a member of" checkbox is now nonstandard/obsolete. We could likely fold that into "Include Projects" in a future change which does a backward compatibility break.

Test Plan:
  - Queried feed without constraints.
  - Queried feed by user, project, "stuff I'm a member of", prebuilt "Tags" query.
  - Viewed user profile / project profile feeds.
  - Used function tokens (`viewer()`, etc).

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12762

Differential Revision: https://secure.phabricator.com/D18028
2017-05-26 12:22:35 -07:00
epriestley
2d79229083 Modernize FeedQuery a little bit
Summary: Ref T12762. Updates some conventions and methods. This has no (meaningful) behavioral changes.

Test Plan:
  - Grepped for `setFilterPHIDs()`.
  - Viewed main feed, user feed, project feed.
  - Called `feed.query`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12762

Differential Revision: https://secure.phabricator.com/D18027
2017-05-26 11:53:19 -07:00
epriestley
69538274c1 Garbage collect old daemon records based on modification date, not creation date
Summary:
Fixes T12720. Currently, old daemon records are collected based on creation date. By default, the GC collects them after 7 days.

After T12298, this can incorrectly collect hibernating daemons which are in state "wait".

In all cases, this could fail to collect daemons which are stuck in "running" for a long time for some reason. This doesn't seem to be causing any problems right now, but it makes me hesitant to do "dateCreated + not running or waiting" since that might make this become a problem, or make an existing problem with this that we just haven't bumped into worse.

Daemons always heartbeat periodically and update their rows, so `dateModified` is always fresh, so collect rows based only on modification date.

Test Plan:
  - Ran daemons (`bin/phd start`).
  - Waited a few minutes.
  - Verified that hibernating daemons in the "wait" state had fresh timestamps.
  - Verified that very old daemons still got GC'd properly.

```
mysql> select id, daemon, status, FROM_UNIXTIME(dateCreated), FROM_UNIXTIME(dateModified) from daemon_log;
+-------+--------------------------------------+--------+----------------------------+-----------------------------+
| id    | daemon                               | status | FROM_UNIXTIME(dateCreated) | FROM_UNIXTIME(dateModified) |
+-------+--------------------------------------+--------+----------------------------+-----------------------------+
| 73377 | PhabricatorTaskmasterDaemon          | exit   | 2017-05-19 10:53:03        | 2017-05-19 12:38:54         |
...
| 73388 | PhabricatorRepositoryPullLocalDaemon | run    | 2017-05-26 08:43:29        | 2017-05-26 08:45:30         |
| 73389 | PhabricatorTriggerDaemon             | run    | 2017-05-26 08:43:29        | 2017-05-26 08:46:35         |
| 73390 | PhabricatorTaskmasterDaemon          | wait   | 2017-05-26 08:43:29        | 2017-05-26 08:46:35         |
| 73391 | PhabricatorTaskmasterDaemon          | wait   | 2017-05-26 08:43:33        | 2017-05-26 08:46:33         |
| 73392 | PhabricatorTaskmasterDaemon          | wait   | 2017-05-26 08:43:37        | 2017-05-26 08:46:31         |
| 73393 | PhabricatorTaskmasterDaemon          | wait   | 2017-05-26 08:43:40        | 2017-05-26 08:46:33         |
+-------+--------------------------------------+--------+----------------------------+-----------------------------+
17 rows in set (0.00 sec)
```

Note that:

  - The oldest daemon is <7 days old -- I had some other older rows but they got GC'd properly.
  - The hibernating taskmasters (at the bottom, in state "wait") have recent/more-current `dateModified` dates than their `dateCreated` dates.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12720

Differential Revision: https://secure.phabricator.com/D18024
2017-05-26 09:18:26 -07:00
epriestley
5b43d5c89c Allow Nuance commands to try to apply immediately
Summary:
Ref T12738. By default, we process Nuance commands in the background. The intent is to let the user continue working at full speed if Twitter or GitHub (or whatever) is being a little slow.

Some commands don't do anything heavy and can be processed in the foreground. Let commands choose to try foreground execution.

Test Plan: Threw complaints in the trash, saw them immediately go into the trash.

Reviewers: chad

Reviewed By: chad

Subscribers: avivey

Maniphest Tasks: T12738

Differential Revision: https://secure.phabricator.com/D18015
2017-05-26 08:36:21 -07:00
Chad Little
9bbea869b3 Move setLaunchButton to setSideColumn for ObjectItem
Summary: Makes this a bit more flexible and allow UI to take over `col-2` completely. Also cleaned up application search a little with tags

Test Plan: Review various pages, grep for callsites.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18021
2017-05-25 15:31:19 -07:00
Chad Little
6b3d04683d Clean up SIMPLE button styles
Summary: Some of these are unused, defaults to a lighter color naturally.

Test Plan: uiexamples, grep, phriction

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18020
2017-05-25 14:53:59 -07:00
epriestley
19572f53fd Don't consider accepting on behalf of valid-but-accepted reviewers to be a validation error
Summary:
Fixes T12757. Here's a simple repro for this:

  - Add a package you own as a reviewer to a revision you're reviewing.
  - Open two windows, select "Accept", don't submit the form.
  - Submit the form in window A.
  - Submit the fomr in window B.

Previously, window B would show an error, because we considered accepting on behalf of the package invalid, as the package had already accepted.

Instead, let repeat-accepts through without complaint.

Some product stuff:

  - We could roadblock users with a more narrow validation error message here instead, like "Package X has already been accepted.", but I think this would be more annoying than helpful.
  - If your accept has no effect (i.e., everything you're accepting for has already accepted) we currently just let it through. I think this is fine -- and a bit tricky to tailor -- but the ideal/consistent beavior is to do a "no effect" warning like "All the reviewers you're accepting for have already accepted.". This is sufficiently finnicky/rare (and probably not terribly useful/desiable in this specific case)that I'm just punting.

Test Plan: Did the flow above, got an "Accept" instead of a validation error.

Reviewers: chad, lvital

Reviewed By: chad, lvital

Subscribers: lvital

Maniphest Tasks: T12757

Differential Revision: https://secure.phabricator.com/D18019
2017-05-25 14:30:19 -07:00
epriestley
84742a94db Restore missing feed rendering for Maniphest points transactions
Summary: See downstream <https://phabricator.wikimedia.org/T166321>. These got dropped in refactoring, or maybe never existed.

Test Plan: {F4977212}

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18018
2017-05-25 14:26:14 -07:00
Chad Little
7cfa8a8315 Add ImageHref attribute for PHUIObjectItemListView
Summary: In some cases we may want a different URI for the image on an item than the header/title of the item (like user / title). This prioritizes ImageHref over Href.

Test Plan: uiexamples

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18016
2017-05-25 10:58:25 -07:00
Austin McKinley
88466addee Migrate Project workboard background color to modular transactions
Summary: Removes now-unused method as well. Fixes T12673.

Test Plan: UI fiddling.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Maniphest Tasks: T12673

Differential Revision: https://secure.phabricator.com/D18014
2017-05-24 12:56:25 -07:00
Austin McKinley
eb296796aa Migrate Project sort and filter defaults to modular transactions
Test Plan: Unit tests pass, manually changed the default sort and filter on a workboard and observed expected transactions in the DB.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D18013
2017-05-24 12:36:06 -07:00
epriestley
8374ec46fd Make throwing things into the trash actually work in Nuance
Summary: Ref T12738. Implements some modular behavior for Nuance commands.

Test Plan: {F4975322}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12738

Differential Revision: https://secure.phabricator.com/D18011
2017-05-24 11:05:30 -07:00
epriestley
7e91f42b02 Issue commands to Nuance items, at least roughly
Summary:
Ref T12738. This makes clicking "Throw In Trash" technically do something, sort of.

In Nuance, the default mode of operation for actions is asynchronous -- so you don't have to wait for a response from Twitter or GitHub after you mash the "send default reply tweet" / "close this pull request with a nice response" button and can move directly to the next item instead.

In the future, some operations will attempt to apply synchronously (e.g., local actions like "ignore this item forever"). This fakes our way through that for now.

There's also no connection to the action actually doing anything yet, but I'll probably rig that up next.

Test Plan: {F4975227}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12738

Differential Revision: https://secure.phabricator.com/D18010
2017-05-24 11:04:57 -07:00
epriestley
e1b8532e24 Allow Nuance items to put commands actions into the work UI
Summary:
Ref T12738. This doesn't actually do anything yet, but allows items to define commands that show up in the UI.

Adds a "Throw in Trash" item for complaints.

This construction will allow future changes to add an `EngineExtension` which can provide generic/default commands across item types.

Test Plan: {F4975086}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12738

Differential Revision: https://secure.phabricator.com/D18009
2017-05-24 11:04:15 -07:00
epriestley
cbf9008d15 Rough in a Nuance "work" controller
Summary:
Ref T12738. This is mostly just laying in groundwork and prerequisites, like the ability to query items by queue.

Eventually, this will become the main UI which staff use to process a queue of items. For now, it does nothing and renders nonsense.

This and probably the next big chunk of changes are all going to be made-up, nonfinal things that just make basic operations work until we have fundamental flows -- like "assign", "comment", "close" -- working at a basic level and can think more about UI/workflow.

Test Plan:
Visited the page, it loaded a mostly-reasonable item and then rendered nonsense:

{F4975050}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12738

Differential Revision: https://secure.phabricator.com/D18008
2017-05-24 11:03:56 -07:00
epriestley
272a5d668f Fix a handful of Nuance fatals
Summary: Ref T12738. Some of the Nuance "form" workflows currently fatal after work on the GitHub stuff. Try to make everything stop fataling, at least.

Test Plan: Using "Complaints Form" no longer fatals, and now lodges a complaint instead.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12738

Differential Revision: https://secure.phabricator.com/D18007
2017-05-24 11:02:55 -07:00
Chad Little
c4392ba067 Add slugs to project manage page
Summary: Minor, just shows the slugs on the manage project page, also normalized language to "details"

Test Plan: review a project with slugs, description.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D17985
2017-05-24 10:23:37 -07:00
Chad Little
684ce701fb Add a description/toggle to PHUIObjectItemView
Summary: Gives the ability to hide a big long block of text in an ObjectListItem without cluttering the UI.

Test Plan:
Added a test case to UIExamples. Click on icon, see content. Click again, content go away.

{F4974153}

{F4974311}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18006
2017-05-24 09:18:13 -07:00
Chad Little
1b36252ef3 Add a dedicated HistoryListView for Diffusion
Summary: Going to play a bit with this layout (diffusion sans audit) and see how it feels on profile. Uses a user image, moves the commit hash (easily selectible) and separates commits by date.

Test Plan:
Review profiles with and without commits.

{F4973987}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18005
2017-05-23 14:03:49 -07:00
Austin McKinley
774fba3ce9 Migrate Project workboard to modular transactions
Summary: This was interesting, because there were a mix of callsites using transactions and others that just set the property on the `Project` object. I made everything consistent in using transactions to change this property. I also found an implementation of `getTitle()` that I don't think is ever being invoked since `shouldHide()` is returning `true`, but I migrated it anyway.

Test Plan: Unit tests pass + enabling/disabling workboards (and importing).

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D18004
2017-05-23 12:38:27 -07:00
Chad Little
7e822fc94f Update PhrictionContent for modular transactions
Summary: Fixes T12625. Moves TYPE_CONTENT in Phriction over to modular transactions.

Test Plan: Edit some documents.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12625

Differential Revision: https://secure.phabricator.com/D18000
2017-05-23 12:19:56 -07:00
Chad Little
1a75ae2405 Modernize Diffusion Create with sidenav, curtain
Summary: This moves the navigation to a standard sidebar, and moves all actions to the curtain. Also pulled out info view when available for cleaner UI.

Test Plan:
Create a git, svn, hg test repository and verify each page in the sidebar renders as expected.

{F4973792}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D18003
2017-05-23 11:29:47 -07:00
Austin McKinley
cd136a6af8 Migrate Project parent and milestone to modular transactions
Test Plan: Unit tests pass. Went through the UI for creating new subprojects and milestones, but didn't setup some API calls to check that all the validation errors were still caught.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D17999
2017-05-23 11:20:01 -07:00
Chad Little
93d8b33cca Check for strlen instead of null for Maniphest title
Summary: Fixes T12744. Unclear why `null` doesn't work here but does for the title, but `!strlen` seems to work fine in both cases.

Test Plan: Create a new task, check mail folder, see [Created]

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12744

Differential Revision: https://secure.phabricator.com/D18002
2017-05-23 16:59:15 +00:00
Chad Little
b164d2d04b Update Phriction Move Away transaction to modular transactions
Summary: Ref T12625

Test Plan: Move a document to a new location, verify the old and new document. Edit both. Grep for MOVE_AWAY

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12625

Differential Revision: https://secure.phabricator.com/D17988
2017-05-22 15:43:47 -07:00
Chad Little
2325c61ad7 Make the Install Dashboard button green
Summary: GREEN

Test Plan: View a dashboard page, see green button

Reviewers: amckinley, epriestley

Reviewed By: epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17997
2017-05-22 22:26:58 +00:00
Chad Little
a1b1082464 Update Phriction Delete to modular transactions
Summary: Ref T12625. Updates to modular transactions

Test Plan: Delete a document, restore a document, see feed story.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12625

Differential Revision: https://secure.phabricator.com/D17996
2017-05-22 15:25:44 -07:00
Chad Little
00400ae6f9 Search and Replace calls to setShade
Summary: grep for setShade and update to setColor. Add deprecated warning.

Test Plan: Diffusion, Workboards, Maniphest, Project tags, tokenizer, uiexamples

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, O14 ATC Monitoring

Differential Revision: https://secure.phabricator.com/D17995
2017-05-22 18:59:53 +00:00
epriestley
3a0a086a09 Convert Nuance sources to modular transactions
Summary: Ref T12738. Update sources to modular transactions.

Test Plan: Created and edited a source.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12738

Differential Revision: https://secure.phabricator.com/D17994
2017-05-22 11:18:23 -07:00
epriestley
5b035af6ff Convert Nuance queues to modular transactions
Summary:
Ref T12738. Swaps queues over.

Also fixes a typo from D17990.

Test Plan: Renamed a queue.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12738

Differential Revision: https://secure.phabricator.com/D17992
2017-05-22 11:18:06 -07:00
epriestley
1069c2bff9 Convert Nuance Items to Modular Transactions
Summary:
Ref T12738. Moves existing non-modular transactions to modular transactions.

Some of these are pretty flimsy, but a lot of them don't actually work or do anything in Nuance yet anyway.

Test Plan: Gently poked Nuance, nothing fell over.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12738

Differential Revision: https://secure.phabricator.com/D17990
2017-05-22 11:17:19 -07:00
Chad Little
03d4d674f8 Clean up some colors missing from PHUITagView type shade
Summary: Grep for phui-tag-shade and verify we're no longer calling shade-color directly.

Test Plan: Search, workboard, story points, etc.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17993
2017-05-22 10:52:10 -07:00
Chad Little
5d966897f1 Add Outline tag type to PHUITagView
Summary: Adds a new tag type, starts to try to clean up the mess that are PHUITags

Test Plan:
Review UIExamples.

{F4972323}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17991
2017-05-22 10:16:27 -07:00
Austin McKinley
179d80dd57 Migrate Project lock to modular transactions
Summary: See T12673

Test Plan: Unit tests pass. Locked and unlocked a project and saw timeline changes.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17986
2017-05-21 11:36:53 -07:00
Chad Little
11578bfc90 Add Revisions to User Profiles
Summary: Ref T12423. Adds back revisions as a user profile page. I don't want to think about custom profiles for a while.

Test Plan: Make some diffs, visit my profile, see diffs.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12423

Differential Revision: https://secure.phabricator.com/D17987
2017-05-21 09:24:37 -07:00
Chad Little
bc4ef0d22f Reorder workboard menu items
Summary: Adds a divider and better grouping

Test Plan: Click on dropdown menu on a workboard

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17984
2017-05-20 09:27:33 -07:00
epriestley
af07600aaa Make Differential objective markers show a brighter "editing" state
Summary:
Ref T12733.

  - While editing a comment, show a pink star ({icon star, color=pink}) with a tooltip.
  - Slight UI tweaks, including draft comments getting an indigo pencil ({icon pencil, color=indigo}).

Test Plan: {F4968470}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12733

Differential Revision: https://secure.phabricator.com/D17977
2017-05-20 07:57:38 -07:00
epriestley
bdecff7d67 Show "objectives" UI only if prototypes are enabled
Summary: See D17955.

Test Plan: Loaded a revision, no longer saw annotations with prototypes off. Still saw annotations with prototypes on.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17983
2017-05-20 07:55:48 -07:00
Chad Little
5f49f9c793 Add sound to logged out Conpherence
Summary: Fixes T12735. Adds a sound if the user is logged out, skips checking a setting.

Test Plan: set participants to null and verify sound plays, no exceptions1111

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12735

Differential Revision: https://secure.phabricator.com/D17973
2017-05-20 06:51:33 -07:00
epriestley
1644b45050 Disperse task subpriorities in blocks
Summary:
Ref T7664. The current algorithm for moving task subpriorities can end up stuck in a real sticky swamp in some unusual situations.

Instead, use an algorithm which works like this:

  - When we notice two tasks are too close together, look at the area around those tasks (just a few paces).
  - If things look pretty empty, we can just spread the tasks out a little bit.
  - But, if things are still real crowded, take another look further.
  - Keep doing that until we're looking at a real nice big spot which doesn't have too many tasks in it in total, even if they're all in one place right now.
  - Then, move 'em out!

Also:

  - Just swallow our pride and do the gross `INSERT INTO ... "", "", "", "", "", "", ... ON DUPLICATE KEY UPDATE` to bulk update.
  - Fix an issue where a single move could cause two different subpriority recalculations.

Test Plan:
  - Changed `ManiphesTaskTestCase->testTaskAdjacentBlocks()` to insert 1,000 tasks with identical subpriorities, saw them spread out in 11 queries instead of >1,000.
  - Dragged tons of tasks around on workboards.
  - Ran unit tests.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7664

Differential Revision: https://secure.phabricator.com/D17959
2017-05-19 15:45:14 -07:00
epriestley
c6a7bcfe89 Make Pholio description behave as a remarkup field (e.g., subscribe mentioned users)
Summary: Ref T12732. This is pre-existing but fix it since I caught it while banging around.

Test Plan: {F4967442}

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12732

Differential Revision: https://secure.phabricator.com/D17970
2017-05-19 14:27:09 -07:00
epriestley
bbc5f79227 Make membership lock/unlock feed stories read more naturally
Summary: Ref T12732. This is pre-existing.

Test Plan: {F4967438}

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12732

Differential Revision: https://secure.phabricator.com/D17969
2017-05-19 14:25:39 -07:00
epriestley
789d57522b Make editing project images redirect to "Manage" more consistently
Summary:
Ref T12732. Currently, different ways of setting a profile image can leave you in different places.

Instead, always send the user back to the "Manage" page.

Test Plan: Used "Current Picture", "use picture", "Build picture" and "upload picture", always ended up in the same spot.

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12732

Differential Revision: https://secure.phabricator.com/D17967
2017-05-19 14:06:48 -07:00
epriestley
10b3879232 Make Project slug/hashtag transactions render a little more nicely
Summary: Ref T12732. Use `renderValue()` to build `renderValueList()` so we get nice fancy text for these.

Test Plan: {F4967410}

Reviewers: chad, amckinley

Reviewed By: amckinley

Maniphest Tasks: T12732

Differential Revision: https://secure.phabricator.com/D17966
2017-05-19 13:59:27 -07:00
Chad Little
abd791889c Update Maniphest title transaction again
Summary: Ref T12732, third time is the charm?

Test Plan: Read

Reviewers: epriestley

Subscribers: Korvin

Maniphest Tasks: T12732

Differential Revision: https://secure.phabricator.com/D17965
2017-05-19 13:46:13 -07:00
Chad Little
5a34b299e4 Update Maniphest title language
Summary: Ref T12732

Test Plan: Read

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12732

Differential Revision: https://secure.phabricator.com/D17964
2017-05-19 13:41:29 -07:00
Chad Little
601622013d Clarify milestone/subproject creation language
Summary: Ref T12732

Test Plan: Read new language

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12732

Differential Revision: https://secure.phabricator.com/D17961
2017-05-19 13:30:02 -07:00
epriestley
c9889e3d55 Fix an issue in Phriction where moving a document just copied it instead
Summary: Ref T12732. See D17918. With modular transactions, `getCustomTransactionNewValue()` isn't actually called.

Test Plan: Moved document `/x/` to `/y/`, saw document gone at `/x/` instead of copied.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12732

Differential Revision: https://secure.phabricator.com/D17963
2017-05-19 13:04:24 -07:00
epriestley
fdf00f6df4 Clean up some minor UI behaviors in Differential
Summary:
Minor UI tweaks:

  - Use the dynamic icon for each file (e.g., image, text), not a hard-coded icon.
  - Render the path (less important) in grey and the filename (more important) in black.

Test Plan: {F4966176}

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17957
2017-05-19 12:01:58 -07:00
epriestley
6c46f27d98 Add quest objectives to the minimap
Summary:
Add important objectives (like waygates and quest markers) to the minimap.

This also probably fixes @cspeckmim's bug with the {key @} keyboard shortcut.

Test Plan:
(This is probably easier to undestand if you `arc patch` + click around.)

{F4966037}

Reviewers: chad, amckinley

Reviewed By: chad

Subscribers: cspeckmim

Differential Revision: https://secure.phabricator.com/D17955
2017-05-19 12:01:01 -07:00
Chad Little
d783299a19 Fix Phriction status not set property on new document
Summary: I deleted too many lines of code here and TYPE_MOVE was always being applied when CONTENT was set. This should fix on next document save, but should I write some migration tool anyways?

Test Plan: Create a new document, see document with correct status.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17960
2017-05-19 17:57:38 +00:00
Chad Little
93e28da76e Add more "disabled" UI to PHUIObjectItemView
Summary: Brings more UI tweaks to disabled objects, like projects/people. Also fixes a missing icon in projects.

Test Plan: Application search with people and projects that have disabled results.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12732

Differential Revision: https://secure.phabricator.com/D17962
2017-05-19 17:54:53 +00:00
Austin McKinley
7e46d7ab6a Migrate Project color to modular transactions
Test Plan: Unit tests + changing project colors.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17958
2017-05-18 16:46:06 -07:00
Austin McKinley
1bff5309e6 Migrate Project icons to modular transactions
Test Plan: Unit tests pass. Changed some icons, observed expected timeline entries.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17956
2017-05-18 16:25:59 -07:00
Austin McKinley
eb84bf98a4 Migrate Project image to modular transactions
Summary: I'm not sure you can actually remove a project's image (maybe via the API?), but I kept the code for rendering the relevant title/feed anyway.

Test Plan: Unit tests + adding/changing project pictures.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D17954
2017-05-18 16:08:37 -07:00
Austin McKinley
f92059d84c Migrate Project status to modular transactions
Test Plan: Unit tests pass. Archived/activated some projects a couple times; observed expected transactions on timeline.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17953
2017-05-18 11:36:13 -07:00
Austin McKinley
91eb22cb3a Migrate Project slugs to modular transactions
Test Plan: Unit tests all pass. Added/removed/altered some project hashtags and observed expected transactions in timeline.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17952
2017-05-18 11:15:16 -07:00
Chad Little
1599c56217 Add Pinboard Items to Timeline
Summary: This allows adding of pinboard items to a timeline. I'm hoping we can get this in for Maniphest (Pholio, Cover Image) and Macro (because, Macro), but unsure how to scalably do this. Anyways, here's the front end.

Test Plan:
Make some fake timeline items in UIExamples, test mobile, tablet, and desktop breakpoints.

{F4965798}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17950
2017-05-18 10:34:57 -07:00
epriestley
fb9f3cc0b4 Restore the "buoyant" header in Differential
Summary:
Fixes T1591. This was removed long ago because it was a mess to implement and caused a bunch of weird issues, and also my tolerance for dealing with weird JS issues was much, much lower.

I have now survived the fires of JX.Scrollbar and would love to address 200 small nitpicks about obscure browser behaviors on Linux, so open the floodgates again.

A secondary goal here is to create room to add a global view state menu on the right, with 300 options like "hide all inlines", "hide done inlines", "hide collapsed inlines", "hide ghosts", "show ghosts", "enable filetree", "disable filetree", etc, etc. Not sure how much of this I'll actually do. I have one more experiment I want to try first.

Test Plan: {F4963294}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T1591

Differential Revision: https://secure.phabricator.com/D17945
2017-05-18 10:24:26 -07:00
epriestley
f78ce156f1 Restore "h" to hide or show files, and modernize file visibility toggling
Summary:
Ref T12616. This puts "h" back to collapse or expand the current file.

This removes some very complicated/messy code around following links in the table of contents and getting files auto-expanded. I suspect no one will miss this, but we can restore it if ayone notices.

Test Plan: Pressed "h" to collapse/expand a file. Also used the menu items.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12616

Differential Revision: https://secure.phabricator.com/D17940
2017-05-18 10:21:37 -07:00
Austin McKinley
1e3c8df1c8 Migrate Project names to modular transactions
Summary: Also changes access modifiers on `PhabricatorProjectTransactionEditor` and sets up `storage` for `applyExternalEffects`.

Test Plan: Created new projects, attempted to create without name, with too long of a name, and with a name that conflicts with other projects and observed expected errors.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Maniphest Tasks: T12673

Differential Revision: https://secure.phabricator.com/D17947
2017-05-17 17:12:22 -07:00
Austin McKinley
d12be0fc21 Change Pholio editor access modifier
Summary: Used by `PholioImageFileTransaction::mergeTransactions()`. I forgot to test adding multiple images to a Mock at the same time after migrating `mergeTransactions` over to the modular framework.

Test Plan: Added multiple images in a single transaction and didn't get an exception about accessing a protected function.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17946
2017-05-17 16:24:40 -07:00
Chad Little
0e8f72a0d9 Clarify Subscription page language
Summary: This says "Edit Subscription" but the page only lets you update the payment method for autopay. I think this is confusing users. I tried to find the best language here, but suggest something else if you prefer.

Test Plan: Review language on sample subscription page.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12451

Differential Revision: https://secure.phabricator.com/D17944
2017-05-17 14:48:22 -07:00
epriestley
9eb285f4aa Fix "left"/"right" changeset ID selection for synthetic deletions
Summary:
Fixes T8323. See that task for a description.

We were using `nonempty()`, but that rule doesn't cover synthetic deletions (file present in an earlier diff, but no longer present in the later diff).

Test Plan: Followed the steps in T8323, got a clean comment.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8323

Differential Revision: https://secure.phabricator.com/D17929
2017-05-17 08:54:13 -07:00
epriestley
51df02821b Move the "select a line range" inline code to DiffInline
Summary: Ref T12616. This makes line range selection use the new code, and removes the remainder of the old "hover a line number" / "select a line range" code.

Test Plan: Hovered line numbers; selected line ranges.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12616

Differential Revision: https://secure.phabricator.com/D17927
2017-05-17 08:41:26 -07:00
epriestley
422eb9db83 Correct the generation of "<th />" IDs on left-hand-side of image changesets
Summary:
Fixes T7682. The left-hand-side "<th />" row did not generate with the correct ID.

(I couldn't reproduce the exact issue described in T7682, but hovering comments on either side now works properly for me.)

Test Plan: {F4962479}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7682

Differential Revision: https://secure.phabricator.com/D17926
2017-05-17 06:26:45 -07:00
epriestley
e4e91ebf6f In Differential, allow "r" to create comments and "R" to quote
Summary:
Ref T11401. Fixes T5232. Ref T12616.

Partly, this moves more code over to the new stuff.

This also allows "r" to work if you have code selected (not just comments). If you "reply" to code, you start a new comment.

You can "R" a comment to quote it. This just starts a new comment normally if you "R" a block of code. This is sort of a power-user version of "quote" since it seems like it probably doesn't really make sense to put it in the UI ever (maybe).

With the new click-to-select, you can click + "R" to reply-with-quote.

Test Plan: Used "r" and "R" to reply to comments and code.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12616, T11401, T5232

Differential Revision: https://secure.phabricator.com/D17920
2017-05-16 17:37:54 -07:00
Chad Little
abc9bc77b2 Move Phriction MOVE_TO transaction to Modular Transactions
Summary: Moves this transaction over to modular transactions.

Test Plan: Move a document, re-title a document, try to move over an existing document.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17918
2017-05-16 15:52:35 -07:00
Chad Little
2aa146ffac Set an icon for Maniphest story points
Summary: It's an icon. For story points.

Test Plan: Set some points, see icon.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17915
2017-05-16 12:44:28 -07:00
Austin McKinley
6888472b56 Migrate Pholio inline comments to modular transactions
Summary: Fixes T12626.

Test Plan: Made lots of comments, confirmed no UI changes

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Maniphest Tasks: T12626

Differential Revision: https://secure.phabricator.com/D17914
2017-05-16 12:22:46 -07:00
Chad Little
a53d387ea6 Move Phriction Title transaction to Modular Transactions
Summary: Ref T12625. Moves TYPE_TITLE to modular transaction.

Test Plan: New Document, Edit Document, test validation, verify feed stories.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12625

Differential Revision: https://secure.phabricator.com/D17912
2017-05-16 11:08:38 -07:00
Chad Little
fdf001739c Normalize headers and actions in Project sub pages
Summary: Run through all the pages in projects and make sure they all feel similar. Adds back curtain on board manage page, even though it is sad for only having a single action.

Test Plan: Test all pages on a project for consistency in UI.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17909
2017-05-16 09:58:33 -07:00
epriestley
8052ab84bf Remove "^" (Prev) and "V" (Next) actions on Differential inline comments
Summary:
Ref T12616. Fixes T12715. I suspect these are very rarely used. (I think you tried to get rid of them before but I pushed back since we couldn't really offer great alternatives at the time?)

Now that the code is in a better place:

  - Click an inline's header (just the colored part) to select it with the keyboard selection cursor.
  - Click again to deselect it.
  - You can use "n" and "p" to jump to comments, so "click + n" is the same as the old "V" action.
  - This also makes it easier to swap between keyboard and mouse workflows, since you can jump into things with the keyboard at any inline.

Also, make "Reply" render more consistently.

Test Plan:
  - Did all that stuff, things seemed to work OK.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12715, T12616

Differential Revision: https://secure.phabricator.com/D17908
2017-05-16 09:44:00 -07:00
Chad Little
29cfcc82ef Update Milestone/Subproject page
Summary: Cleans up the UI, moves details over to curtain, adds some fallback no data strings.

Test Plan: Review with and without subprojects, milestones.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17907
2017-05-16 09:31:25 -07:00
Chad Little
bf753c8b5a Make passing an object to newCurtain optional
Summary: We seem to already support this, just takes it fully there. We don't need to see things like "Flag", etc, on certain subpages of projects/people/etc.

Test Plan: Review Members, Subproject pages, no longer see "Flag for Later" which only is for the Project itself. Check manage, still there.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17897
2017-05-16 08:51:05 -07:00
epriestley
1b5a276a02 Add Differential keyboard shortcuts for "mark done" and "hide/show"
Summary:
Fixes T8130. Allows selected comments to be shown/hidden (with "q") or marked done/not-done (with "w").

(These key selections are because "qwer" are right next to each other on QWERTY keyboards, and now mean "hide, done, edit, reply".)

Also, allow "N" and "P" to do next/previous inline, including hidden inlines. This makes "q" to hide/show a little more powerful and a little easier to undo.

Test Plan: Used "q", "w", "N" and "P" to navigate and interact with comments.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8130

Differential Revision: https://secure.phabricator.com/D17906
2017-05-16 08:23:22 -07:00
Chad Little
ef839192aa Only show member/watcher notes on Members page in Projects
Summary: Restricts the view of the membership privileges to just the Members page itself, and not other pages like Home/Details.

Test Plan: Test Home, Test Members, see correct layouts.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17896
2017-05-16 07:17:18 -07:00
Chad Little
1e47ba2481 Use setDrag UI for reordering workboard columns
Summary: This UI can use the setDrag call to reduce clutter on the reodering dialog.

Test Plan:
Reorder some columns, save.

{F4959906}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17898
2017-05-16 07:13:25 -07:00
epriestley
2fb1edfeb1 Restore the Differential "edit" and "reply" keyboard shortcuts
Summary: Ref T12616. This makes "edit" and "reply" work again.

Test Plan:
Used "e" and "r" to edit and reply.

Also used them in bogus ways and got useful UI feedback.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12616

Differential Revision: https://secure.phabricator.com/D17895
2017-05-16 06:25:35 -07:00
epriestley
588a66c04d Move most Differetial keyboard shortcuts into DiffChangesetList
Summary: Ref T12616. This moves most keyboard shortcuts into DiffChangesetList. It breaks some shortcuts that I plan to restore later, noted in T12616 (toggle file, edit inline, reply to inline), since I think ripping them out now and rebuilding them in a little bit will make things much simpler.

Test Plan:
  - Used j, k, n, p, J, K shortcuts to navigate a revision.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12616

Differential Revision: https://secure.phabricator.com/D17859
2017-05-16 06:24:42 -07:00
epriestley
4fd4ec3d27 Hide inlines one-by-one, instead of in a big group
Summary:
Ref T12616. Fixes T12153. Currently, when you hide inlines, they hide completely and turn into a little bubble on the previous line.

Instead, collapse them to a single line one-by-one. Narrowly, this fixes T12153.

In the future, I plan to make these changes so this feature makes more sense:

  - Introduce global "hide everything" states (T8909) so you can completely hide stuff if you want, and this represents more of a halfway state between "nuke it" and "view it".
  - Make the actual rendering better, so it says "epriestley: blah blah..." instead of just "..." -- and looks less dumb.

The real goal here is to introduce `DiffInline` and continue moving stuff from the tangled jungle of a million top-level behaviors to sensible smooth statefulness.

Test Plan:
  - Hid and revealed inlines in unified and two-up modes.
  - These look pretty junk for now:

{F4948659}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12616, T12153

Differential Revision: https://secure.phabricator.com/D17861
2017-05-16 06:19:56 -07:00
epriestley
fe44e987fb Translate "Loading..." text in inline comments
Summary: Ref T12616. This cements the relationship between ChangesetList (parent container) and Changeset (child) and passes translations down so Changeset can use them to translate the text "Loading..."

Test Plan: Viewed loading changes.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12616

Differential Revision: https://secure.phabricator.com/D17846
2017-05-16 06:19:01 -07:00
epriestley
64a54aac9d Merge "differential-dropdown-menus" behavior into DiffChangesetList
Summary: Ref T12616. This ends up being a little messy ("one giant function") and maybe I'll clean it up a bit later, but continue consolidating the wild jungle of behaviors into a smaller set of responsible objects.

Test Plan: Clicked all the menu options, saw them work properly. Grepped for removed methods.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12616

Differential Revision: https://secure.phabricator.com/D17845
2017-05-16 06:18:26 -07:00
Austin McKinley
545d6347dd Convert remaining Pholio image transactions to modular transaction framework
Summary: Also cleans up now-dead code relating to old transactions

Test Plan: Created lots of mocks, replaced their images, added/removed images, changed the sequence, verified expected DB xaction rows, Mock updates, and correct rendering of timeline.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17892
2017-05-15 16:25:34 -07:00
Chad Little
f600bc0811 Clean up watchers and members project page
Summary: Various little fixes, mostly moves information from the "Details" section either into the curtain or into the specific watchers or members list based on user viewership. I think this page is both cleaner and more informative.

Test Plan:
Lock, Unlock, Watch, Join, various projects with multiple users.

{F4959101}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17891
2017-05-15 15:56:14 -07:00
Austin McKinley
78544334cd Fix breakage of Pholio
Summary: Removal of `PholioMockEditor::applyCustomInternalTransaction()` in D17868 broke creation of new mocks in Pholio. Puts the empty method back until we finish migrating Pholio to modular transactions.

Test Plan: Created some mocks, observed lack of unhandled exception.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17889
2017-05-15 13:30:13 -07:00
Chad Little
904480dc3c Generate newValue for ManiphestTaskPointTransaction
Summary: I think this is the correct fix, sets a consistent value for transactions, old and new, for Maniphest point values.

Test Plan:
Edit title, see no point feed story, set points, see point story, set points to same value, see no story, remove points, see remove point story.

{F4958233}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17885
2017-05-15 11:18:54 -07:00
Chad Little
d6a620be45 Update Maniphest for modular transactions
Summary: Ref T12671. This modernized Maniphest transactions to modular transactions.

Test Plan:
- Create Task
- Edit Task
- Raise Priority
- Change Status
- Merge as a duplicate
- Create Subtask
- Claim Task
- Assign Project
- Move on Workboard
- Set a cover image
- Assign story points
- Change story points
- Generate lots via lipsum
- Bulk edit tasks
- Leave comments
- Award Token

I'm sure I'm missing something.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: hazelyang, Korvin

Maniphest Tasks: T12671

Differential Revision: https://secure.phabricator.com/D17844
2017-05-15 10:29:06 -07:00
Chad Little
ea874a2825 Restrict watching and member project display better
Summary: Fixes T12713. We don't need to show watching and member info on other views other than ApplicationSearch (for now) so add a few methods to restrict the calls.

Test Plan: Visit project search, profile, project home, project home with subprojects

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12713

Differential Revision: https://secure.phabricator.com/D17883
2017-05-15 09:04:54 -07:00
Chad Little
6fa91caf32 Clean up Project Board Manage / Column Manage pages
Summary: Slightly nicer, more consistent UI. Also removed "Column History" from dropdowns as this is available on the general board manage page.

Test Plan: Review Board and Column management pages.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17881
2017-05-15 07:26:34 -07:00
epriestley
7f54f79fd1 Add required needMembers/needWatchers calls to Project Profile/Subprojects tabs
Summary:
Fixes T12710. See that task for discussion. This is pretty ugly/redundant but not broken.

(Feel free to reject this and pursue something else.)

Test Plan:
  - For a project with active subprojects/milestones, viewed the project profile and subprojects tabs.
  - After patch: they're ugly, but no longer fatal.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12710

Differential Revision: https://secure.phabricator.com/D17882
2017-05-15 06:44:18 -07:00
Chad Little
89e567ffd9 Move Board Manage actions up a level
Summary: Moves "reorder columns" and "change background" up a level, redesigns "manage" page to be a little cleaner.

Test Plan: Change colors, reorder columns, manage page, disable board, re-enable board.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17879
2017-05-14 17:48:14 -07:00
Chad Little
a5ad11d2d2 Add Member/Watcher info to search results
Summary: Fixes T12707 Adds additional information to search results for if user is a member or a watcher of a project. Also removed the icon colors, which I'll find a better way to denote in future.

Test Plan: Join a project, watch a project, view results list in /projects/

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12707

Differential Revision: https://secure.phabricator.com/D17880
2017-05-14 17:38:37 -07:00
epriestley
c7a6422559 Make a handful of minor Slowvote behaviors more consistent with other applications
Summary:
Ref T12685.

  - Better icon/color/label consistency.
  - Make "0" a valid response.
  - Fix a bug where creating a poll with Quicksand enabled led to bad times (form submitted back to the search screen).

Test Plan: {F4956412}

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17878
2017-05-14 14:01:33 -07:00
epriestley
d2e1dd0ef0 In Applications, always redirect back to the detail page after edits
Summary:
Ref T12685. When you edit an application's policies but don't make any changes, you currently get stuck on the same page. This isn't how other edit screens work, and I think it's a holdover from eras long ago.

Make it consistent with other applications.

Also, fix a missing `pht()`.

Test Plan:
  - Edited an application configuation.
  - Clicked "Save" without making changes.
  - After patch, was redirected back to detail page like in other applications.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17877
2017-05-14 14:01:01 -07:00
epriestley
95925ad58f Make getApplicationTransactionEditor() in PhabricatorApplication return an editor
Summary:
Ref T12685. I provided this incorrect (`return new` rather than `throw`) implementation earlier; it can now be replaced with a proper implementation.

This caused application policy edits to spew this into the daemon log:

```
[2017-05-14 15:35:27] EXCEPTION: (Error) Call to undefined method PhutilMethodNotImplementedException::setActor() at [<phabricator>/src/applications/transactions/worker/PhabricatorApplicationTransactionPublishWorker.php:69]
```

Test Plan:
  - Used `bin/worker execute --id <id>` to execute a previously-failing task.
  - Saw a feed story publish.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17876
2017-05-14 14:00:25 -07:00
epriestley
aa81979922 Fix a runaway indentation level in LegalpadDocumentEditEngine
Summary: Ref T12685.

Test Plan: No behavioral changes.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17875
2017-05-14 13:59:57 -07:00
Chad Little
a25c79bbc0 Add Column Edit / History actions to workboard columns
Summary: This brings up "Edit Column" as an action item under the main column dropdown as well as a "Column History" for completeness. Unsure column history is actually useful, but leaving it in anyways. It might be nice to have some sort of dialog version of a history page.

Test Plan: Make a workboard, add a column, edit column name, stay on workboard.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17874
2017-05-14 16:41:20 -04:00
epriestley
11c5638832 Fix some error log issues with uninitialized commit/revision lists
Summary:
Fixes T12679. Reproduction steps appear to be:

  - As a logged-out user, view revision list or commit list.
  - Enable bucketing by action required.
  - Before patch: `foreach (null as ...)` causes error spew.
  - After patch: `foreach (array() as ...)` works great.

Test Plan:
  - Reproduced issue by following steps above in Differential (revisions) and Diffusion (audits/commits).
  - After patches, no more errors in the log.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12679

Differential Revision: https://secure.phabricator.com/D17872
2017-05-14 13:28:02 -07:00
Chad Little
db631b423f Add basic Watching filter to /projects/
Summary: Ref T12707. Adds a simple filter for the viewer if logged in.

Test Plan: Watch a project, click on watching list, see project I'm watching.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley

Maniphest Tasks: T12707

Differential Revision: https://secure.phabricator.com/D17873
2017-05-14 15:06:04 -04:00
Austin McKinley
74d86f0902 Migrate Pholio image description and mock status to modular transactions
Summary: Also removes more now-dead code from `PholioTransaction`.

Test Plan: opened and closed a bunch of mocks, edited a bunch of image descriptions

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D17868
2017-05-12 16:07:18 -07:00
Chad Little
b3692cc12c Fix PonderAnswer transaction callsite
Summary: Caught this in the logs, calling an old transaction, update it.

Test Plan: Answer a question, tail log, see no error.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17870
2017-05-12 19:04:48 +00:00
Chad Little
7aeb498565 Fix missing transaction constants in Phame
Summary: Trailing logs for errors here, picked up some unset constants in Phame.

Test Plan: New Blog, New Post, tail daemon log for errors. See feed stories.

Reviewers: epriestley, amckinley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17871
2017-05-12 19:04:16 +00:00
Chad Little
28a23e3822 Update Legalpad to use comment transactions
Summary: Updates Legalpad to use comment transactions

Test Plan: Leave some comments.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17867
2017-05-11 10:14:13 -07:00
Chad Little
e1a97616cb Add a large profile picture to Projects
Summary: The ports over a similar "profile image" menu item to Projects. It gives us some room to use the project icon in the sidenav along with a larger photo. It also will open up some room in the sub-page headers for us to focus on that page, and not the identity of the project at hand. Expect a few more project related touch up diffs.

Test Plan:
Review new projects menu on a few projects, update the image, see new image. Great for team photos.

{F4951264}

Reviewers: epriestley, amckinley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17869
2017-05-11 10:13:38 -07:00
Austin McKinley
fcf64953f0 Migate more of Pholio to Modular Transaction
Summary: Another Franken-transaction. Adds transactions for ImageName and MockName. Adds transaction-level validation for presence of mock name; removed same from edit controller. Removes `PholioTransaction::getRemarkupBodyForFeed()`, which appears to be dead code since that method isn't defined on any other types.

Test Plan: made a bunch of changes to pholio mocks and images and observed expected results

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, chad, epriestley

Differential Revision: https://secure.phabricator.com/D17865
2017-05-10 15:13:48 -07:00
Chad Little
b32a00dbac Update Legalpad for EditEngine
Summary: Updates Legalpad to use EditEngine, paving the way for //transaction comments//. Spooky.

Test Plan:
- New Document
  - Require signing, Corp - see fail
  - Require signing, Noone - see fail
  - Require signing, Ind - get asked to sign
- Edit Document

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17862
2017-05-10 15:11:49 -07:00
Chad Little
bc4edc946b Add Conduit edit endpoint for Macro
Summary: Opens up the edit endpoint for Macros

Test Plan: Review /conduit/method/macro.edit/

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17866
2017-05-10 14:54:43 -07:00
Austin McKinley
d39758ca7c Migrate Pholio to Frankenstein transactions
Summary: Begins the process of migrating Pholio to Modular Transactions by starting with the mock's description and changing the base class of PholioTransaction. Expect several more of these diffs to quickly follow. Also changes the icon for description changes to `fa-pencil`; previously it was check or ban, depending on the open/closed state. Looks like an accidental switch fallthrough.

Test Plan: made new mocks, edited their descriptions, observed same UI as previously

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D17864
2017-05-10 12:38:49 -07:00
Chad Little
437a843a8e Clean up legalpad transaction copy
Summary: Ref T12685, adds more informative transaction copy, fixes bugs.

Test Plan: Create various documents and edit them.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17860
2017-05-09 10:47:24 -07:00
Chad Little
c4aa0a9b9a Fix FundRefund transaction calls
Summary: This function `renderHandleLink` doesn't exist. Update call.

Test Plan: I'm still lost on how to refund a transaction? Existing?

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17858
2017-05-09 09:59:41 -07:00
Chad Little
b2cbb8f2ee Update Fund for EditEngine commenting
Summary: Ref T12685. Moves Fund to EditEngine commenting.

Test Plan: View an old Fund, leave a comment, add a subscriber.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17857
2017-05-09 09:05:43 -07:00
Chad Little
86f9318052 Update Fund for EditEngine
Summary: Ref T12685, updates fund for edit engine.

Test Plan: Create a Fund, Edit a Fund, wipe out Merchants, check errors for name and missing merchants, back Fund.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17855
2017-05-09 08:46:16 -07:00
Chad Little
f717e4b563 Clean up some Fund language
Summary: Ref T12685. Cleans up various Fund language nits.

Test Plan: Read carefully.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17856
2017-05-08 21:11:45 -07:00
Chad Little
3dae970129 Clean up some rough Macro transaction edges
Summary: Ref T12685, cleans up various macro issues, remove subscribers, fix feed stories, etc.

Test Plan: Create a new macro, see no subscribers, edit various macros.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17848
2017-05-08 16:29:35 -07:00
Chad Little
bf30c07230 Clean up Slowvote transactions a little
Summary: Ref T12685. Moves to `xaction` folder and sets description changes in transaction stories.

Test Plan: Make a poll, edit the description.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17849
2017-05-08 16:15:31 -07:00
epriestley
ff205964cf Make macro audio errors more clear
Summary: Ref T12685. I bamboozled myself.

Test Plan: {F4945786}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17847
2017-05-08 16:01:47 -07:00
Chad Little
32af1d9a9f Restrict PhortuneMerchant creation at EditEngine level
Summary: Ref T12685. Checks merchant capabilities at the edit engine level

Test Plan: Test with and without admin level permissions. Get restricted access if I cheat. Still able to create with admin.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17852
2017-05-08 15:56:07 -07:00
Chad Little
c505876d61 Clean up some Passphrase transaction bugs
Summary: Ref T12685. Makes the description field full remarkup and fixes setting a credential secret after destruction.

Test Plan: Change description a lot, set and destroy credentials.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17851
2017-05-08 15:55:50 -07:00
Chad Little
ade380530f Clean up some Phame transaction edit bugs
Summary: Ref T12685. Sets required on required fields, cleans up mailtags on PhamePost

Test Plan: Create a new blog, post.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12685

Differential Revision: https://secure.phabricator.com/D17850
2017-05-08 15:55:26 -07:00
Chad Little
0a87881e98 Fix file attach bug in Macro
Summary: This was mis-tested by only using one account, which could always see the image. External transaction moved file attachment to the modular transaction for file and audio instead.

Test Plan: Test adding audio and a macro on a pleb account, visit with normal account and see macro fine.

Reviewers: epriestley, amckinley

Reviewed By: amckinley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17836
2017-05-05 19:50:12 -07:00
Chad Little
c8c23af840 Enable Feed stories for Fund
Summary: Not sure when these stopped, also fixed mailtag contants.

Test Plan: Close an initiative, see story, fund initiative, see story.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17835
2017-05-05 11:15:10 -07:00
Chad Little
a9604c4604 Update Fund for modular transactions
Summary: Fixes T12627. Updates FundInitiative and FundBacker with modular transactions.

Test Plan: Create an Initiative, back it with fake monies, close initiative, reopen, edit various fields.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12627

Differential Revision: https://secure.phabricator.com/D17782
2017-05-05 11:03:03 -07:00
Austin McKinley
0dc90b891a Reimplement Slowvote transactions using modular transactions
Summary:
Fixes T12623. Adds new modular transactions to Slowvote. Also converts
the `shuffle` column to `bool` for consistency with other boolean-ish columns.

Test Plan:
Create a new vote, modified everything that could be modified from the web UI,
observed expected timeline.

Example timeline: {F4938843}

Example transaction values in DB: {F4938850}

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T12623

Differential Revision: https://secure.phabricator.com/D17830
2017-05-04 20:20:00 -07:00
epriestley
836819a0e7 Explicitly quote "From" name part when submitting mail to the Mailgun API
Summary:
We are submitting `epriestley (Evan Priestley) <noreply@meta.phacility.com>`, but should be submitting `"epriestley (Evan Priestley)" <noreply@meta.phacility.com>`.

Add the missing quotes.

Test Plan: Locally, this makes the API calls work against the Mailgun sandbox domain.

Reviewers: chad, amckinley

Reviewed By: chad, amckinley

Differential Revision: https://secure.phabricator.com/D17831
2017-05-04 17:03:26 -07:00
Chad Little
d77a3f8760 Update Spaces for modular transactions
Summary: Updates the Spaces application for modular transactions, seemed easy to bang out.

Test Plan: Create a space, edit a space, archive a space. Verify default space works as intended.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17829
2017-05-04 21:48:13 +00:00
Chad Little
705fd11ba1 Update Legalpad to use modular transactions
Summary: Update Legalpad for modular transactions

Test Plan:
- New Document (no sign)
- New Document (individual)
- New Document (corp)
- Require Signature - get prompted to sign before I can do anything.
- Edit Documents
- Sign Documents
- Comment on Documents

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17826
2017-05-04 12:45:02 -07:00
Chad Little
06eae5578b Update Passphrase for modular transactions
Summary: Updates Passphrase for modular transactions.

Test Plan: Create, edit, lock, view, lots of different types of Passphrases. Enable Conduit, Lock Passphrases, Destroy Secrets from the interface and verify from the DB it was eradicated.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17824
2017-05-04 11:31:37 -07:00
Austin McKinley
d34b338f3f Implement modular transactions for application policy changes
Summary: Still needs some cleanup, but ready for review in broad outline form.

Test Plan:
Made lots of policy changes to the Badges application and confirmed expected rows in `application_xactions`, confirmed expected changes to `phabricator.application-settings`.

See example output (not quite working for custom policy objects) here:

{F4922240}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, chad, epriestley

Maniphest Tasks: T11476

Differential Revision: https://secure.phabricator.com/D17757
2017-05-03 17:49:41 -07:00
Chad Little
5307f51170 Update Macro comment form
Summary: Moves over to transaction commenting.

Test Plan: Leave a comment (tested with TYPE_COMMENT still present).

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17823
2017-05-03 12:34:21 -07:00
epriestley
60a19e3646 Allow ApplicationTransactionEditor to figure out whether TYPE_COMMENT is supported or not
Summary: See D17812, etc. We can figure this out by looking at the object carefully. We don't need to go delete all the old TYPE_COMMENT (it doesn't hurt anything) but can nuke it when we see it.

Test Plan:
  - Made a comment in Slowvote (supports commenting).
  - Viewed an Almanac device (does not support commenting).

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17822
2017-05-03 11:20:57 -07:00
Chad Little
d467a9e337 Modernize PonderQuestion with EditEngine
Summary: Just a small touch up to move this to edit engine.

Test Plan:
- Create a question
- Edit a question
- Close question
- Test NUX state

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17812
2017-05-03 11:14:52 -07:00
Chad Little
4aec809b69 Update PhamePost for modular transactions
Summary: Updates PhamePost for modular transactions.

Test Plan:
- Create a post
- Edit a post
- Add a header image
- Delete header image
- Award Token
- Leave comment
- Unpublish post
- Check History page
- Move post
- Archive post

{F4936456}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17818
2017-05-03 11:14:25 -07:00
Chad Little
c878bf5033 Update Macro for EditEngine
Summary: Updates Macro to use EditEngine. Also removes "URL" field for adding a Macro, which I think it's worth pursuing.

Test Plan:
- Create a Macro
- Forget to name it
- Try a PDF
- Use a Macro
- Edit a macro (not working)

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17821
2017-05-03 11:11:23 -07:00
Chad Little
21ac196306 Modernize PhameBlog with modular transactions
Summary: Moves PhameBlog over to the wonderful world of modular transactions and the riches that lay beyond...

Test Plan:
- Create Blog
- Edit Blog
- Set Header
- Delete Header
- Add picture
- Archive blog
- Set incorrect domain values
- Be irresponsible with subtitle length
- Activate blog
- Change description

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17815
2017-05-02 15:39:56 -07:00
Chad Little
a067040024 Update Macro for Modular Transactions
Summary: Overall, seems to work ok.

Test Plan:
- Add a Macro
- Edit Macro
- Use Macro
- Disable Macro
- Re-enable Macro
- Attach Audio
- Set Audio to loop
- Annoy cats

{F4932069}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17813
2017-05-02 08:41:14 -07:00
Chad Little
eaaba278a9 Convert PonderAnswer to modular transactions
Summary: Fixes T12624. Converts PonderAnswer over to modular transactions.

Test Plan:
- Add an answer
- Edit an answer
- Hide an answer
- Comment on an answer

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12624

Differential Revision: https://secure.phabricator.com/D17811
2017-05-01 12:44:30 -07:00
Chad Little
dff028c490 Update PonderQuestion for Modular Transactions
Summary: Ref T12624, this moves PonderQuestion over to modular transactions.

Test Plan:
- Create a question
- Edit a question
- Comment on question
- Answer question
- Edit Answer
- Close Question
- Verify answer count in list views
- Check Question History
- Check Answer History

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12624

Differential Revision: https://secure.phabricator.com/D17810
2017-05-01 11:38:25 -07:00
epriestley
0ebe56be40 Fix two Calendar availability cache issues
Summary:
Fixes T12661.

When changing the start date of an event from some time in the past to some time significantly in the future (more than 24 hours), we'd invalidate only future caches and leave users in an "away" state. Instead, just invalidate all past and future caches (this is simpler than trying to figure out a narrower window, and should not make us do too much extra work).

When uninviting users from events, their caches also didn't get cleared correctly. Instead, clear them.

Test Plan:
  - Changed an event from "Apr 1 - June 1" to "May 15 - June 1", saw availablity clear properly.
  - Uninvited user `@dog` from an ongoing event, saw availability clear properly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12661

Differential Revision: https://secure.phabricator.com/D17809
2017-05-01 10:43:51 -07:00
Chad Little
da6d624fe8 Mark ConpherenceCreate and Update conduit calls as frozen
Summary: T12656, mark these methods as frozen and use conpherence.edit instead.

Test Plan: Visit conduit, check status is displayed.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17808
2017-05-01 09:59:38 -07:00
Josh Cox
f2ca348b3a Set project's ObjectName to its PHID when it doesn't have a hashtag
Summary: Fixes T12659. Previously this would lead to an error when trying to run `arc diff` on a revision that had a milestone as a reviewer (or any non-octothorpe'd Object Name)

Test Plan: Followed repro steps in T12659 and didn't get the error described. Also clicked around and didn't notice any obvious regressions in projects or differential

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, yelirekim

Maniphest Tasks: T12659

Differential Revision: https://secure.phabricator.com/D17807
2017-04-05 06:22:52 -04:00
epriestley
b7c4f60e23 Only recognize "Fixes ..." in main revision content like the Summary or Test Plan
Summary:
Fixes T12642. Currently, writing "Fixes T..." in a comment gets picked up as a formal "fixes".

This is a bit confusing, and can also give you a "no effect" error if you "fixes ..." a task which is already "fixes"'d.

We could make the duplicate action a non-error, but just prevent the text from having an effect instead, which seems cleaner.

Test Plan:
  - Wrote "Fixes ..." in a summary, saw a "fixes" relationship established.
  - Wrote "Fixes ..." in a comment, got a "mention" instead.
  - `var_dump()`'d some stuff as a sanity check, looked reasonable.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12642

Differential Revision: https://secure.phabricator.com/D17805
2017-04-30 13:12:28 -07:00
epriestley
89d0c8e388 Use grey dots for disabled users, even if a user is also unverified
Summary: Fixes T12559.

Test Plan:
{F4929988}

{F4929989}

{F4929990}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12559

Differential Revision: https://secure.phabricator.com/D17806
2017-04-30 13:10:00 -07:00
Chad Little
608dd06151 Add a lipsum generator for Conpherence Rooms
Summary: Builds a basic room generator for Conpherence, picks a random name, adds 10 random users to it, sets view and edit policy to all users.

Test Plan:
`bin/lipsum generate conpherence`

{F4928815}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17699
2017-04-29 20:08:00 -07:00
Chad Little
d6bce34a5d Update Conpherence to use EditEngine
Summary: Moves Conpherence to use EditEngine. This removes the "First Message" field, but I think that's ok until we have direct messaging of some sort, then maybe have built-ins cover that case.

Test Plan:
 - Visit /new/ and /edit/ for creating new rooms.
 - Edit a room in full conpherence
 - Edit a room in durable column
 - grep for METADATA calls

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11729

Differential Revision: https://secure.phabricator.com/D16677
2017-04-29 19:39:12 -07:00
epriestley
d2baa88171 Allow Owners packages to have arbitrarily long names
Summary:
  - Change column type from `sort128` to `sort`.
  - Remove `originalName`. This column is unused. Long ago, we used it to generate a `Thread-Topic` header for mail, but just use PHIDs now (the value just needs to be stable for a given object, users normally don't see it).

Test Plan:
  - Created a package with a beautifully long name. Magnificent!
  - Grepped for `originalName` / `getOriginalName()`, found no Owners hits.
  - Verified that there isn't any name-length validation code to remove.

{F4925637}

Reviewers: chad, amckinley

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17798
2017-04-27 18:04:03 -07:00
Chad Little
7824710522 Add an Owners Package hovercard
Summary: Ref T12600. Basically all the property (not path) information on a hovercard for owner packages.

Test Plan:
Create a package with LOTS OF RULES. Test it as open and archived states.

{F4923441}

{F4923444}

Reviewers: epriestley, jmeador

Reviewed By: jmeador

Subscribers: jmeador, Korvin

Maniphest Tasks: T12600

Differential Revision: https://secure.phabricator.com/D17793
2017-04-26 08:45:39 -07:00
Chad Little
d70d09c02c Fix notice spacing on table views
Summary: When a notice is in a table view in a two column layout, reset the margins.

Test Plan: Visit OwnerDetails with no paths set.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17792
2017-04-25 16:46:55 -07:00
Austin McKinley
6bfaf31c0a Spelling fix
Summary: I can't believe I've been staring at this page for so long without noticing this typo

Test Plan: doitlive

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17791
2017-04-25 12:19:27 -07:00
Chad Little
9509bd1c7f Fix legalpad error display
Summary: Fixes T12636.

Test Plan: Make a from, submit signature without checking box, see correctly aligned error.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12636

Differential Revision: https://secure.phabricator.com/D17789
2017-04-25 09:28:23 -07:00
epriestley
0d5538672c Detect unsynchronizable repositories on multiple cluster hosts
Summary:
Ref T12613. Currently, the SVNTEST and HGTEST repositories are improperly configured on `secure`. These repositories use VCS systems which do not support synchronization, so they can not be served from cluster services with multiple hosts.

However, I've incorrectly configured them the same way as all the Git repositories, which support synchronization. This causes about 50% of requests to randomly fail (when they reach the wrong host).

Detect this issue and warn the user that the configuration is not valid.

It should be exceptionally difficult for normal installs to run into this.

Test Plan:
  - Mostly faked these conditions locally, verified that `secure` really has this configuration.
  - I'll push this, verify that the issue is detected correctly in production, then fix the config which should resolve the intermittent issues with SVNTEST.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12613

Differential Revision: https://secure.phabricator.com/D17774
2017-04-24 10:43:05 -07:00
epriestley
41d9453ece When a user changes to a verified primary address, mark their account as verified
Summary:
Ref T12635. See that task for discussion.

You can currently end up with a verified primary address but no "verified" flag on your account through an unusual sequence of address mutations.

Test Plan:
  - Registered without verifying, using address "A".
  - Added a second email address, address "B".
  - Verified B (most easily with `bin/auth verify`).
  - Changed my primary email to B.
  - Before patch: account not verified.
  - After patch: account verified.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12635

Differential Revision: https://secure.phabricator.com/D17785
2017-04-24 10:24:21 -07:00
Chad Little
b3c8226cbb New hovercard UI for Maniphest Tasks
Summary: Swaps out hovercard boring view for super cool workboard card view. Will have more diffs to add additional information down the road.

Test Plan: {F4921092}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17769
2017-04-23 18:58:08 -07:00
epriestley
d0e6bf831d Add "%I" (instance name) to application log formats
Summary:
Ref T12611. Currently, the HTTP/SSH logs don't have an option to include the instance name.

Add such an option.

Leave it out of the default logs because most installs don't use this.

Test Plan: See next changes.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12611

Differential Revision: https://secure.phabricator.com/D17776
2017-04-23 11:07:19 -07:00
epriestley
bb2b91d28e Add unit tests for file thumbnail generation
Summary: Fixes T12614.

Test Plan:
  - Ran tests, saw them pass.
  - Changed a thumbnail to 800x800, saw tests detect that the default image was missing.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12614

Differential Revision: https://secure.phabricator.com/D17773
2017-04-23 11:02:21 -07:00
epriestley
bc9291b327 Fix Conpherence theme variables for both logged-out and logged-in users
Summary: Ref T12622.

Test Plan: As a logged-out and logged-in user, loaded Conpherence threads.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12622

Differential Revision: https://secure.phabricator.com/D17768
2017-04-22 09:23:39 -07:00
Chad Little
8d398f6f4a Fix variable name for theme_class
Summary: Fixes T12622. This variable is mis-named.

Test Plan: Visit a room I have not joined.

Reviewers: epriestley, amckinley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12622

Differential Revision: https://secure.phabricator.com/D17767
2017-04-22 16:01:10 +00:00
epriestley
03d0e3fdbc Fix fatal in Conpherence NUX state
Summary: Fixes T12619.

Test Plan: Faked `return array()` in ConpherneceThreadQuery, got a NUX instead of fatal.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12619

Differential Revision: https://secure.phabricator.com/D17764
2017-04-21 14:50:29 -07:00
Chad Little
d3546f94c1 Improve diffusion readme layout
Summary: Uses more standard objects and more padding for reading. Removes the ToC, which is visually broken anyways.

Test Plan: Review a README.md in a local repository.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17752
2017-04-21 11:23:26 -07:00
Chad Little
7c61ace086 Attach Diffusion Pagers to their ObjectBoxView
Summary: Adds the ability to set a pager onto an object box directly and pick up appropriate styles.

Test Plan: grep for renderTablePagerBox, test layouts with and without a pager.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12604

Differential Revision: https://secure.phabricator.com/D17754
2017-04-21 11:22:19 -07:00
Chad Little
ed9afa18cc Allow users to choo choo choose a room color
Summary: This adds some basic per user / per room theming for Conpherence, which should hopefully let users identify rooms from just the sidebar color.

Test Plan: Lots of threads with different colors.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17747
2017-04-20 14:30:59 -07:00
epriestley
7a992b5488 When a package or project has been accepted or rejected, show who did it ("Accepted (by dog)")
Summary: Makes it more clear whose authority actions have been taken under.

Test Plan: {F4916376}

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17741
2017-04-20 13:07:08 -07:00
Chad Little
6969e9389f Move room preferences into own controller
Summary: Ref T12591. These preferences are per user and we need to reload the whole UI on a change anyways. Simplifies future expansion.

Test Plan: Option click into new window, set preferences. View in Conpherence, see saved settings, change sound. Hear new sound.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12591

Differential Revision: https://secure.phabricator.com/D17740
2017-04-19 16:48:36 -07:00
epriestley
95dd9dbf43 Make Applications extend LiskDAO
Summary:
Ref T11476. This is a bit hacky, but makes `Application` extend `LiskDAO` so we can apply transactions to it with an `Editor` class.

Also fixes schema stuff so builds should produce a clean bill of health again.

This might only get you slightly further, yell if you run into more trouble.

Test Plan:
  - Ran `bin/storage upgrade -f` and got no warnings.
  - Browsed around, nothing exploded?

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11476

Differential Revision: https://secure.phabricator.com/D17738
2017-04-19 16:06:14 -07:00
Austin McKinley
d58b808f04 Fixing copy/paste mistake
Test Plan: doitlive

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17737
2017-04-19 15:48:59 -07:00
Austin McKinley
febd68039f Add initial infrastructure for adding ModularTransaction support to Application config changes
Summary: Part of the groundwork for T11476.

Test Plan: ran `./bin/storage upgrade` and observed expected DB tables

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11476

Differential Revision: https://secure.phabricator.com/D17736
2017-04-19 15:44:57 -07:00
Chad Little
db60af7ea5 Set all room settings at once
Summary: Sets notification and sound preferences in a single array()

Test Plan: Change email preference, save, set sound preference, save. Email preference still OK.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17735
2017-04-19 14:39:17 -07:00
Chad Little
51485a1f82 Add participants ModularTransactions to Conpherence
Summary: Moves participants over to ModularTransactions, simplified a lot of the code. Fixes T12550

Test Plan:
Create a new room with just myself and myself + fake accounts.
Remove a person.
Remove myself.
Edit a room, topic.
Type some messages.
???
Profit

Reviewers: chad

Reviewed By: chad

Subscribers: Korvin

Maniphest Tasks: T12550

Differential Revision: https://secure.phabricator.com/D17685
2017-04-19 14:01:15 -07:00
epriestley
b9868f4f05 Don't require a transaction to mark a participant up-to-date
Summary:
Pathway to D17685. We no longer have "behindTransactionPHID", so we no longer need the latest transaction.

This allows some code to be removed.

Test Plan:
  - Grepped for callsites to `markUpToDate()` and variables used in the calls.
  - Sent messages in a couple threads, viewed them, saw unread counts go away.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17733
2017-04-19 14:00:55 -07:00
epriestley
f46263903c Fix improper filtering behavior in ConpherenceParticipantQuery
Summary:
Pathway to D17685. This fixes an issue idenified in D17731: if any caller ever queried for more than one participant, some results could get thrown away by re-keying the results on thread PHID: two different participants can be members of the same thread!

This also fixes an issue from D17683, where a `needParticipantCache()` callsite was overlooked.

Test Plan:
  - Viewed Conpherence dropdown.
  - Sent messages, saw unread count / thread order still work properly.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17732
2017-04-19 13:59:47 -07:00
epriestley
76d0b67d91 Remove "dateTouched" from ConpherenceParticipant
Summary:
Pathway to D17685. This column is (mostly) a denormalization of `dateModified` on the thread.

Just use a JOIN instead.

This isn't //exactly// the same: we'll bump threads to the top now for non-message changes (e.g., a topic or title change). That seems fine, but we could put a `lastMessageDate` on Thread later if we want to refine it.

Also got rid of a lot of other unused stuff. There's a big garbage TODO here, I'll fix that in the next change.

Test Plan:
  - Grepped for `dateTouched`.
  - Grepped for `participantCursor`.
  - Grepped for `ConpherenceParticipantQuery::LIMIT`.
  - Looked for callsites to `setOrder()`, found none.
  - Added a message to an older thread, saw it bump up to the top.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17731
2017-04-19 13:58:54 -07:00
epriestley
0a335f91cd Remove "participationStatus" from ConpherenceParticipant
Summary:
Pathway to D17685. This column is a very complicated cache of: is participant.messageCount equal to thread.messageCount?

We can just ask this question with a JOIN instead and simplify things dramatically.

Test Plan:
  - Ran migration.
  - Browsed around.
  - Sent a message, saw unread count go up.
  - Read the message, saw unread count go down.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17730
2017-04-19 13:58:42 -07:00
epriestley
c96e17f5ea Remove "behindTransactionPHID" from ConpherenceParticipant
Summary: Pathway to D17685. Nothing reads this field and it has no use or value.

Test Plan:
  - Ran migration.
  - Grepped for `behindTransactionPHID`.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17729
2017-04-19 13:58:29 -07:00
Chad Little
15e7624a17 Add a few more sounds
Summary: A few more mp3s to choose from for Conpherence.

Test Plan: Test each sound in a new room.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17734
2017-04-19 13:47:23 -07:00
Chad Little
8e813f4f1f Add some basic sound preferences
Summary: Ref T7567. This adds some constants (for adding new sounds), global setting for turning on and off sound (setting) and per thread preference for sound choice. Also specc'd out Mentions, if added.

Test Plan: I tested all the preference wiring, but need to set up notifications locally to verify if this works. Feel free to test.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: amckinley, Korvin

Maniphest Tasks: T7567

Differential Revision: https://secure.phabricator.com/D17726
2017-04-19 13:04:02 -07:00
epriestley
f880000eb0 Stem fulltext tokens before filtering them for stopwords
Summary:
Fixes T12596. A query for a token (like "having") which stems to a stopword (like "have") currently survives filtering. Stem it first so it gets caught.

Also, for InnoDB, a custom stopword table can be configured. If it is, read that instead of the default stopword list (I configured it locally, but the default list is reasonable so we never formally recommended installs configure it).

Test Plan:
Queried for words that stem to stopwords, saw them filtered:

{F4915843}

Queried for the original problem query and saw "having" caught with "have" in the stopword list:

{F4915844}

Fiddled with local InnoDB stopword table config and saw the stopword list get loaded correctly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12596

Differential Revision: https://secure.phabricator.com/D17728
2017-04-19 10:02:21 -07:00
epriestley
b479941ceb Make "Range" HTTP header work for Celerity static resource requests
Summary:
Ref T7567. In T8266 I fixed a bunch of obscure "Range" issues, but only for file downloads -- not for Celerity.

Extend all that stuff to Celerity, which is fortunately much easier.

I believe this will fix Conpherence sounds in Safari.

Test Plan:
  - Wrote out an HTTP request in a text file with `Range: bytes=0-1` and similar, piped it to localhost with `cat request.txt | nc localhost 80`, saw server return appropriate range responses consistent with file behavior after T8266, which all seems to work.
  - Also did that for files to try to make sure I wasn't breaking anything.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T7567

Differential Revision: https://secure.phabricator.com/D17724
2017-04-18 13:46:27 -07:00
Austin McKinley
ece9579d25 Switch File deletion to use ModularTransactions
Summary: Fixes T12587. Adds a new `PhabricatorFileDeleteTransaction` that enqueues `File` delete tasks.

Test Plan:
  - hack `PhabricatorFileQuery` to ignore isDeleted state
  - stop daemons
  - upload a file, delete it from the UI
  - check that the DB has updated isDeleted = 1
  - check timeline rendering in `File` detail view
  - start daemons
  - confirm rows are deleted from DB

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, thoughtpolice

Maniphest Tasks: T12587

Differential Revision: https://secure.phabricator.com/D17723
2017-04-18 13:01:51 -07:00
epriestley
ab2aa74d6e Fix several duplication/replay behaviors in Aphlict
Summary:
Ref T12566. Ref T12563. This fixes three bugs with Aphlict replay stuff:

First, Conphernece would try to repaint the UI even if no thread was open. Only repaint when a thread is open.

Second, although we deduplicate JX.Leader messages, we didn't deduplicate actual notification messages. If you browsed the leader window, then it re-elected itelf as a leader and replayed history, it could rebroadcast notifications and other windows could show doubles. Deduplicate notifications to prevent this.

Third, we always replayed the last 60 seconds of history. When you browsed the leader window, whichever window became the new leader (possibly the one you just browsed) could replay messages from before it had opened, leading to duplicate messages. Particularly, after receiving a message and then browsing you could see that message again. Instead, only replay history as far back as when the window first opened.

Test Plan:
  - Clicked "Repaint" with a thread open, saw a repaint. Clicked "Repaint" with Conpherence open but no thread, no repaint and no 404 request to `/update/null/`.
  - In browser A, opened three windows. In browser B, sent a notification. In browser A, browsed the leader window away twice in a row. Observed that the window which never became a leader doesn't duplicate notifications.
  - In browser A, opened three windows. In browser B, sent a notification. In browser A, browsed the leader window away over and over again. Observed that replay requests issued with appropriate history windows.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12566, T12563

Differential Revision: https://secure.phabricator.com/D17722
2017-04-18 12:10:12 -07:00
epriestley
5d55804e3f Play a sound when receiving a new chat message
Summary:
Ref T7567. Nothing fancy yet, just getting this working. Sound is lightly edited version of "Pop 6":

https://www.freesound.org/people/greenvwbeetle/sounds/244656/

Test Plan: Sent chat, heard sounds.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7567

Differential Revision: https://secure.phabricator.com/D17721
2017-04-18 11:34:17 -07:00
Austin McKinley
be00264ae7 Make daemons perform file deletion
Summary:
Deletion is a possibly time-intensive process, especially with large
files that are backed by high-latency, chunked storage (such as
S3). Even ~200mb objects take minutes to delete, which makes for an
unhappy experience. Fixes T10828.

Test Plan:
Delete a large file, and stare in awe of the swiftness with
which I am redirected to the main file application.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: thoughtpolice, Korvin

Maniphest Tasks: T10828

Differential Revision: https://secure.phabricator.com/D15743
2017-04-18 11:09:41 -07:00
epriestley
8377bb3637 Raise a tailored error message on "show-outbound --id cat"
Summary: Fixes T12579. Unclear why the user ran this command.

Test Plan: Ran with `--id cat`. Ran with `--id 123`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12579

Differential Revision: https://secure.phabricator.com/D17719
2017-04-18 09:51:26 -07:00
Austin McKinley
b54adc6161 Kick off indexing for File objects on creation
Summary: Ensures that newly-made `File` objects get indexed into the new ngrams index. Fixes T8788.

Test Plan:
  - uploaded a file with daemons stopped; confirmed no new rows in ngrams table
  - started daemons; confirmed indexing of previously-uploaded files happened
  - uploaded a new file with daemons running; confirmed it got added to the index

Not sure how to test the changes to `PhabricatorFileUploadSource->writeChunkedFile()` and `PhabricatorChunkedFileStorageEngine->allocateChunks()`. I spent a few minutes trying to find their callers, but the first looks like it requires a Diffusion repo and the 2nd is only accessible via Conduit. I can test that stuff if necessary, but it's such a small change that I'm not worried about it.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T8788

Differential Revision: https://secure.phabricator.com/D17718
2017-04-18 08:38:34 -07:00
Austin McKinley
976fbee877 Implement ngram search for File objects
Summary: Follows the outline in D15656 for implementing ngram search for names of File objects. Also created FileFullTextEngine, because without implementing `PhabricatorFulltextInterface`, `./bin/search` complains that `File` is not an indexable type.

Test Plan:
  - ran `./bin/storage upgrade` to apply the schema change
  - confirmed the presence of a new `file_filename_ngrams` table
  - added a couple file objects
  - ran `bin/search index --type file --force`
  - confirmed the presence of rows in `file_filename_ngrams`
  - did a few keyword searches and saw expected results

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T8788

Differential Revision: https://secure.phabricator.com/D17702
2017-04-17 17:37:20 -07:00
Chad Little
c98be54bf4 Don't show tag when no topic is set
Summary: Check the strlen of topic before adding a tag to the header in Conpherence.

Test Plan: Remove a topic, no longer see indigo bubble.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17715
2017-04-17 16:15:17 -07:00
epriestley
ffed156981 After a reconnect, repaint Conpherence thread state
Summary: Ref T12566. When we reconnect, refresh the current thread even if we replayed notifications.

Test Plan:
  - Clicked the "Repaint" button, saw the thread refresh.
  - Clicked the "Reconnect" button, saw the thread reresh.
  - Launched `aphlict debug`, killed it, restarted it, saw the thread refresh after reconnect.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12566

Differential Revision: https://secure.phabricator.com/D17713
2017-04-17 16:00:32 -07:00
epriestley
eaecf35324 Deduplicate application-level notifications from Aphlict
Summary:
Fixes T12564. We already had some code which seems to deal with this properly, it just wasn't getting used.

Assign each application-level notification a unique ID, then ignore messages with duplicate IDs.

Test Plan:
  - In browser A, loaded `/T123`.
  - In browser B, loaded `/T123`.
  - Made a comment as B.
  - Saw notification as A.
  - Mashed "Replay" a bunch.
  - Before patch: piles of duplicate notifications.
  - After patch: no duplicates.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12564

Differential Revision: https://secure.phabricator.com/D17710
2017-04-17 15:55:38 -07:00
epriestley
02194f0fc8 After Aphlict reconnects, ask the server to replay recent messages
Summary:
Fixes T12563. If we've ever seen an "open", mark all future connections as reconnects. When we reconnect, replay recent history.

(Until duplicate messages (T12564) are handled better this may cause some notification duplication.)

Also emit a reconnect event (for T12566) but don't use it yet.

Test Plan: {F4912044}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12563

Differential Revision: https://secure.phabricator.com/D17708
2017-04-17 15:54:51 -07:00
epriestley
88157a9442 Hold recent messages in Aphlict so they can be replayed after clients reconnect
Summary:
Ref T12563. Before broadcasting messages from the server, store them in a history buffer.

A future change will let clients retrieve them.

Test Plan:
  - Used the web frontend to look at the buffer, reloaded over time, sent messages. Saw buffer size go up as I sent messages and fall after 60 seconds.
  - Set size to 4 messages, sent a bunch of messages, saw the buffer size max out at 4 messages.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12563

Differential Revision: https://secure.phabricator.com/D17707
2017-04-17 15:53:58 -07:00
epriestley
1212047843 Add a "Reconnect" debugging action and show reconnect delays in the console
Summary: Ref T12568. Ref T12567. Allows you to force a reconnect, and shows the reconnect delay on connection close/failure.

Test Plan: {F4911879}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12568, T12567

Differential Revision: https://secure.phabricator.com/D17705
2017-04-17 15:51:24 -07:00
epriestley
f394fefe6f Add a very basic "Realtime" log to DarkConsole
Summary: Ref T12568. This begins building toward a more useful realtime debugging console for Leader/Aphlict/general realtime stuff.

Test Plan: {F4911521}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12568

Differential Revision: https://secure.phabricator.com/D17701
2017-04-17 15:46:31 -07:00
epriestley
6052bc1933 Extend "fulltext" and "ngrams" interfaces from "indexable" interface
Summary: Ref T8788. See D17702. This allows `bin/search index` to index stuff which only implements `Ngrams`, not `Fulltext`.

Test Plan: Kinda poked around `bin/search index` a bit, yell if you hit more issues deeper down the stack?

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T8788

Differential Revision: https://secure.phabricator.com/D17704
2017-04-17 12:59:41 -07:00
Chad Little
2d00f56837 Use PHUIListItemView in ConpherenceThreadList
Summary: Fixes T12556 Uses more common components in ConpherenceThreadList by moving to PHUIListItemView. Reduces clutter by moving privacy into the header. Gets ride of "See More" double interchanges.

Test Plan:
I need to test this more, doesn't seem to auto-select top room any more, also might build a lipsum generator.

 - Create lots of rooms with various policies
 - Test clicking on policy object
 - Click on different rooms
 - Post in rooms
 - Load up second account, see room numbers
 - Clear room message count by clicking on room

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12556

Differential Revision: https://secure.phabricator.com/D17698
2017-04-17 11:21:49 -07:00
Austin McKinley
f801c7ae29 Change PhabricatorPhurlURLViewController to use EditEngine for commenting
Test Plan: Created a phurl, added some comments, confirmed that "Change Subscribers" and "Change Project Tags" are now available in the comment form.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: chad, Korvin

Maniphest Tasks: T11661

Differential Revision: https://secure.phabricator.com/D17686
2017-04-17 10:19:21 -07:00
Chad Little
a56f9a1a55 Clean up remove participant language in Conpherence
Summary: Updates the language to use "Remove Participant" instead of "Banish User"

Test Plan: Read through the various cases, test them by removing myself or others

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17697
2017-04-15 16:07:54 +00:00
Chad Little
c1e8b394cc Fix join and remove policy checks for Conpherence
Summary:
I think these got munged when I removed CAN_JOIN.

 - If you can view the room, you can join it.
 - ~~If you can view the room, you can add others to it.~~ This rule adjustment was removed, see discussion on the revision.
 - If you are a participant in the room, you can remove yourself.
 - If you can edit a room, you can remove anyone.

Test Plan:
Normal feature set:

 - Create a new room that only I can edit, viewable by all users.
 - Leave room (bye k thx)
 - Create another room, myself only
 - Join room from second account
 - See ability to only remove myself
 - Remove myself
 - Rejoin
 - Add third account
 - Log into first account
 - Boot off randos
 - Test joining by green button, message, and by + sign.

Policy consistency:

  - As a user who can not edit the room, tried to add other members. Received policy exception. The `+` button is currently visible and enabled for all users (even users who have not joined the room) but this is pre-existing.

Reviewers: chad

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17696
2017-04-15 06:16:45 -07:00
epriestley
aec19d2acf Reduce code duplication in Phortune account controllers
Summary:
Ref T12451. This is a GREAT comment (A++) but we only need one copy of it.

This uses a pattern similar to Projects, which is a little weird but works well enough.

Test Plan:
  - Viewed all four tabs of an account.
  - Viewed a page with a bad account ID which 404'd properly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12451

Differential Revision: https://secure.phabricator.com/D17694
2017-04-14 10:24:56 -07:00
epriestley
7fbb5f2d92 Reduce some code duplication in PhortuneLandingController
Summary: Ref T12451. This code is the same as the other code.

Test Plan: Went through the default-account case with this code, worked the same as the other code.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12451

Differential Revision: https://secure.phabricator.com/D17693
2017-04-14 10:24:32 -07:00
epriestley
505b1d8379 Fix member edit transaction validation so it works for both implicit and explicit account creation
Summary:
Ref T12451. Ref T12484. This should deal with all the `+` / `-` / `=` cases correctly, I think.

Also makes sure that members are real users, not commits or tokens or whatever. And expands the creation test case to make some other basic sanity checks.

Test Plan:
  - Went through implicit first-time creation flow.
  - Went through explicit second-time creation flow.
  - Unit test now passes.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12484, T12451

Differential Revision: https://secure.phabricator.com/D17692
2017-04-14 10:24:15 -07:00
epriestley
71d933d496 Add a failing test case for new Phortune account initialization
Summary:
Ref T12451. Ref T12484. I think D17657 fixed this, but caused the bug in D17690. The fix for that causes this bug again.

Put a unit test on it. This test currently fails; I'll correct the bug in the next change.

Test Plan: Ran `arc unit`, saw a failure.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12484, T12451

Differential Revision: https://secure.phabricator.com/D17691
2017-04-14 10:23:10 -07:00
epriestley
e1a8b5d3e9 Fix a bug where Phortune accounts created via "Create Account" would not have the viewer added as a member
Summary:
Ref T12451. When you explicitly created a second or third account or whatever, you wouldn't be added as a member.

(The editor sees that you're "already a member", so it doesn't add you.)

Test Plan:
  - Go to `/phortune/`.
  - Click "Switch Accounts".
  - Click "Create Account".
  - Create an account.
  - Before patch: unable to view it since you don't get added as a member.
  - After patch: account created with you as member.
  - Also created an accont with multiple members.
  - Tried to create an account with no members.
  - Tried to create an account with just someone else as a member.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12451

Differential Revision: https://secure.phabricator.com/D17690
2017-04-14 10:22:54 -07:00
epriestley
7274e4857c Fix a bug where Phortune could fatal while building crumbs
Summary: Ref T12451. `$this->getAccount()` may not return an account.

Test Plan:
  - Visit `/phortune/X/`, where `X` is the ID of an account you don't have permission to view.
  - Before patch: fatal.
  - After patch: normal policy exception page.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12451

Differential Revision: https://secure.phabricator.com/D17689
2017-04-14 10:22:42 -07:00
epriestley
1e43d57c81 When closing tasks with "Fixes xxx", try to act more authentically as the acting user
Summary:
Via HackerOne (<https://hackerone.com/reports/220909>). When we close commits in response to "Fixes Txxx", we currently act as the omnipotent user. This allows users to close tasks they can't see by pushing commits with "Fixes Txxx" in the message.

However, we can't actually tell who authored or committed a change: we're just using the "Author" and "Committer" values from Git in most cases, and anyone can forge those. So we can't really get this right, in a security sense.

(We can tell who //pushed// a change if we host it, but that's often not the right user. If GPG signing was more prevalent, we could use that. In the future, we could use side channels like having `arc land` tell Phabrcator who was pushing changes.)

Since I think the impact of this is fairly minor and this isn't //really// a security issue (more of a confusion/abuse/product issue) I think the behavior is okay more-or-less as-is, but we can do better when we do identify an author: drop permissions, and use their privileges to load the tasks which the commit "fixes".

This effectively implements this rule:

> If we identify the author of a commit as user X, that commit can only affect tasks which user X can see and edit.

Note that:

  - Commits which we can't identify the author for can still affect any task.
  - Any user can forge any other user's identity (or an invalid identity) and affect any task.

So this is just a guard rail to prevent mistakes by good-faith users who type the wrong task IDs, not a real security measure.

Also note that to perform this "attack" you must already have commit access to a repository (or permission to create a repository).

Test Plan:
  - Used `bin/repository reparse --message <commit> --force-autoclose` to run the relevant code.
  - Made the code `throw` before it actually applied the edit.
  - Verified that the edit was rejected if the author was recognized and can not see or could not edit the task.
  - Verified that the edit is accepted if the author can see+edit the task.
  - Verified that the edit is accepted if we can't figure out who the author is.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17688
2017-04-14 08:03:46 -07:00
epriestley
69053a40f9 Dirty the SSH key cache when usernames change
Summary:
Fixes T12554. The SSH key cache contains usernames, but is not currently dirtied on username changes.

An alternative solution would be to use user PHIDs instead of usernames in the file, which would make this unnecessary, but that would make debugging a bit harder. For now, I think this small added complexity is worth the easier debugging, but we could look at this again if cache management gets harder in the future.

Test Plan:
  - Added a key as `ducksey`, ran `bin/ssh-auth`, saw key immediately.
  - Renamed `ducksey` to `ducker`, ran `bin/ssh-auth`, saw username change immediately.
  - Added another key as `ducker`, ran `bin/ssh-auth`, saw key immediately.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12554

Differential Revision: https://secure.phabricator.com/D17687
2017-04-14 08:03:00 -07:00
Austin McKinley
980d6cb70b Add validation for config settings of type regex
Summary: Also fixes insufficiently-escaped regex examples

Test Plan: Made several changes to http://local.phacility.com/config/edit/syntax.filemap/ and observed validation failures on malformed regexes, and success on well-formed regexes.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12532

Differential Revision: https://secure.phabricator.com/D17684
2017-04-13 13:57:03 -07:00