1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-11 08:06:13 +01:00
Commit graph

12667 commits

Author SHA1 Message Date
epriestley
509fbb6c20 When building audit queries, prefilter possible "authorPHID" values
Summary:
Ref T13244. See PHI1057. Currently, if you're a member of a lot of projects/packages, you can end up with a very large `commit.authorPHID IN (...)` clause in part of the "Active Audits" query, since your `alice` token in "Responsible Users: alice" expands into every package and project you can audit on behalf of.

It's impossible for a commit to be authored by anything but a user, and evidence in PHI1057 suggests this giant `IN (...)` list can prevent MySQL from making effective utilization of the `<authorPHID, auditStatus, ...>` key on the table.

Prefilter the list of PHIDs to only PHIDs which can possibly author a commit.

(We'll also eventually need to convert the `authorPHIDs` into `identityPHIDs` anyway, for T12164, and this moves us slightly toward that.)

Test Plan: Loaded "Active Audits" before and after change, saw a more streamlined and sensible `authorPHID IN (...)` clause afterwards.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13244

Differential Revision: https://secure.phabricator.com/D20129
2019-02-07 15:36:56 -08:00
epriestley
a4bab60ad0 Don't show "registration might be too open" warnings unless an auth provider actually allows registration
Summary:
Depends on D20118. Fixes T5351. We possibly raise some warnings about registration (approval queue, email domains), but they aren't relevant if no one can register.

Hide these warnings if no providers actually support registration.

Test Plan: Viewed the Auth provider list with registration providers and with no registration providers, saw more tailored guidance.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T5351

Differential Revision: https://secure.phabricator.com/D20119
2019-02-07 15:32:42 -08:00
epriestley
7469075a83 Allow users to be approved from the profile "Manage" page, alongside other similar actions
Summary:
Depends on D20122. Fixes T8029. Adds an "Approve User" action to the "Manage" page.

Users are normally approved from the "Approval Queue", but if you click into a user's profile to check them out in more detail it kind of dead ends you right now. I've occasionally hit this myself, and think this workflow is generally reasonable enough to support upstream.

Test Plan: {F6193742}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T8029

Differential Revision: https://secure.phabricator.com/D20123
2019-02-07 15:04:23 -08:00
epriestley
949afb02fd On login forms, autofocus the "username" field
Summary: Depends on D20120. Fixes T8907. I thought this needed some Javascript nonsense but Safari, Firefox and Chrome all support an `autofocus` attribute.

Test Plan: Loaded login page with password auth enabled in Safari, Firefox, and Chrome; saw username field automatically gain focus.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T8907

Differential Revision: https://secure.phabricator.com/D20122
2019-02-07 15:03:43 -08:00
epriestley
8054f7d191 Convert a manual query against external accounts into a modern Query
Summary: Depends on D20108. Ref T6703. Update this outdated callsite to a more modern appraoch.

Test Plan: Destroyed a user with `bin/remove destroy`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T6703

Differential Revision: https://secure.phabricator.com/D20109
2019-02-07 15:02:00 -08:00
epriestley
f0364eef8a Remove weird integration between Legalpad and the ExternalAccount table
Summary:
Depends on D20107. Ref T6703. Legalpad currently inserts "email" records into the external account table, but they're never used for anything and nothing else references them.

They also aren't necessary for anything important to work, and the only effect they have is making the UI say "External Account" instead of "None" under the "Account" column. In particular, the signatures still record the actual email address.

Stop doing this, remove all the references, and destroy all the rows.

(Long ago, Maniphest may also have done this, but no longer does. Nuance/Gatekeeper use a more modern and more suitable "ExternalObject" thing that I initially started adapting here before realizing that Legalpad doesn't actually care about this data.)

Test Plan: Signed documents with an email address, saw signature reflected properly in UI. Grepped for other callsites.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T6703

Differential Revision: https://secure.phabricator.com/D20108
2019-02-07 15:00:00 -08:00
epriestley
9f5e6bee90 Make the default behavior of getApplicationTransactionCommentObject() "return null" instead of "throw"
Summary:
Depends on D20115. See <https://discourse.phabricator-community.org/t/transaction-search-endpoint-does-not-work-on-differential-diffs/2369/>.

Currently, `getApplicationTransactionCommentObject()` throws by default. Subclasses must override it to `return null` to indicate that they don't support comments.

This is silly, and leads to a bunch of code that does a `try / catch` around it, and at least some code (here, `transaction.search`) which doesn't `try / catch` and gets the wrong behavior as a result.

Just make it `return null` by default, meaning "no support for comments". Then remove the `try / catch` stuff and all the `return null` implementations.

Test Plan:
  - Grepped for `getApplicationTransactionCommentObject()`, fixed each callsite / definition.
  - Called `transaction.search` on a diff with transactions (i.e., not a sourced-from-commit diff).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: jbrownEP

Differential Revision: https://secure.phabricator.com/D20121
2019-02-07 14:56:38 -08:00
epriestley
4fa5a2421e Add formal setup guidance warning that "feed.http-hooks" will be removed in a future version of Phabricator
Summary: Depends on D20114. This is on the way out, so make that explicitly clear.

Test Plan: Read setup issue and configuration option.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20115
2019-02-07 14:54:09 -08:00
epriestley
e25dc2dfe2 Revert "feed.http-hooks" HTTP request construction to use "http_build_query()" so nested "storyData" is handled correctly
Summary:
See <https://discourse.phabricator-community.org/t/storydata-is-blank-in-outgoing-requests-to-the-configured-feed-http-hooks/2366/>.

This behavior was changed by D20049. I think it's generally good that we not accept/encode nested values in a PHP-specific way, but retain `feed.http-hooks` compatibility for now.

Test Plan: {F6190681}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20114
2019-02-07 14:53:03 -08:00
epriestley
26081594e2 Fix two very, very minor correctness issues in Slowvote
Summary:
See <https://hackerone.com/reports/492525> and <https://hackerone.com/reports/489531>. I previously awarded a bounty for <https://hackerone.com/reports/434116> so Slowvote is getting "researched" a lot.

  - Prevent users from undoing their vote by submitting the form with nothing selected.
  - Prevent users from racing between the `delete()` and `save()` to vote for multiple options in a plurality poll.

Test Plan:
  - Clicked the vote button with nothing selected in plurality and approval polls, got an error now.
  - Added a `sleep(5)` between `delete()` and `save()`. Submitted different plurality votes in different windows. Before: votes raced, invalid end state. After: votes waited on the lock, arrived in a valid end state.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20125
2019-02-07 12:45:11 -08:00
Austin McKinley
f2236eb061 Autofocus form control for adding TOTP codes
Summary: Ref D20122. This is something I wanted in a bunch of places. Looks like at some point the most-annoying one (autofocus for entering TOTOP codes) already got fixed at some point.

Test Plan: Loaded the form, got autofocus as expected.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D20128
2019-02-07 11:56:49 -08:00
epriestley
fc3b90e1d1 Allow users to unlink their last external account with a warning, instead of preventing the action
Summary:
Depends on D20105. Fixes T7732. T7732 describes a case where a user had their Google credentials swapped and had trouble regaining access to their account.

Since we now allow email login even if password auth is disabled, it's okay to let users unlink their final account, and it's even reasonable for users to unlink their final account if it is mis-linked.

Just give them a warning that what they're doing is a little sketchy, rather than preventing the workflow.

Test Plan: Unlinked my only login account, got a stern warning instead of a dead end.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T7732

Differential Revision: https://secure.phabricator.com/D20106
2019-02-06 17:07:41 -08:00
epriestley
d6f691cf5d In "External Accounts", replace hard-to-find tiny "link" icon with a nice button with text on it
Summary:
Ref T6703. Replaces the small "link" icon with a more obvious "Link External Account" button.

Moves us toward operating against `$config` objects instead of against `$provider` objects, which is more modern and will some day allow us to resolve T6703.

Test Plan: Viewed page, saw a more obvious button. Linked an external account.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T6703

Differential Revision: https://secure.phabricator.com/D20105
2019-02-06 16:07:16 -08:00
epriestley
55286d49e8 Clarify "metamta.default-address" instructions and lock the option
Summary: This option no longer needs to be configured if you configure inbound mail (and that's the easiest setup approach in a lot of cases), so stop telling users they have to set it up.

Test Plan: Read documentation and configuration help.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20104
2019-02-06 16:03:49 -08:00
epriestley
113a2773dd Remove one-time login from username change email
Summary:
Depends on D20100. Ref T7732. Ref T13244. This is a bit of an adventure.

Long ago, passwords were digested with usernames as part of the salt. This was a mistake: it meant that your password becomes invalid if your username is changed.

(I think very very long ago, some other hashing may also have used usernames -- perhaps session hashing or CSRF hashing?)

To work around this, the "username change" email included a one-time login link and some language about resetting your password.

This flaw was fixed when passwords were moved to shared infrastructure (they're now salted more cleanly on a per-digest basis), and since D18908 (about a year ago) we've transparently upgraded password digests on use.

Although it's still technically possible that a username change could invalidate your password, it requires:

  - You set the password on a version of Phabricator earlier than ~2018 Week 5 (about a year ago).
  - You haven't logged into a version of Phabricator newer than that using your password since then.
  - Your username is changed.

This probably affects more than zero users, but I suspect not //many// more than zero. These users can always use "Forgot password?" to recover account access.

Since the value of this is almost certainly very near zero now and declining over time, just get rid of it. Also move the actual mail out of `PhabricatorUser`, ala the similar recent change to welcome mail in D19989.

Test Plan: Changed a user's username, reviewed resulting mail with `bin/mail show-outbound`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13244, T7732

Differential Revision: https://secure.phabricator.com/D20102
2019-02-05 16:01:53 -08:00
epriestley
9632c704c6 Always allow users to login via email link, even if an install does not use passwords
Summary:
Depends on D20099. Ref T13244. See PHI774. When password auth is enabled, we support a standard email-based account recovery mechanism with "Forgot password?".

When password auth is not enabled, we disable the self-serve version of this mechanism. You can still get email account login links via "Send Welcome Mail" or "bin/auth recover".

There's no real technical, product, or security reason not to let everyone do email login all the time. On the technical front, these links already work and are used in other contexts. On the product front, we just need to tweak a couple of strings.

On the security front, there's some argument that this mechanism provides more overall surface area for an attacker, but if we find that argument compelling we should probably provide a way to disable the self-serve pathway in all cases, rather than coupling it to which providers are enabled.

Also, inch toward having things iterate over configurations (saved database objects) instead of providers (abstract implementations) so we can some day live in a world where we support multiple configurations of the same provider type (T6703).

Test Plan:
  - With password auth enabled, reset password.
  - Without password auth enabled, did an email login recovery.

{F6184910}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13244

Differential Revision: https://secure.phabricator.com/D20100
2019-02-05 16:00:55 -08:00
epriestley
99e5ef84fc Remove obsolete "PhabricatorAuthLoginHandler"
Summary: Depends on D20096. Reverts D14057. This was added for Phacility use cases in D14057 but never used. It is obsoleted by {nav Auth > Customize Messages} for non-Phacility use cases.

Test Plan: Grepped for removed symbol.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20099
2019-02-05 14:20:14 -08:00
epriestley
4fcb38a2a9 Move the Auth Provider edit flow toward a more modern layout
Summary:
Depends on D20095. Ref T13244. Currently, auth providers have a list item view and a single gigantic edit screen complete with a timeline, piles of instructions, supplemental information, etc.

As a step toward making this stuff easier to use and more modern, give them a separate view UI with normal actions, similar to basically every other type of object. Move the timeline and "Disable/Enable" to the view page (from the edit page and the list page, respectively).

Test Plan: Created, edited, and viewed auth providers.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13244

Differential Revision: https://secure.phabricator.com/D20096
2019-02-05 14:19:26 -08:00
epriestley
8c8d56dc56 Replace "Add Auth Provider" radio buttons with a more modern "click to select" UI
Summary:
Depends on D20094. Ref T13244. Ref T6703. See PHI774. Currently, we use an older-style radio-button UI to choose an auth provider type (Google, Password, LDAP, etc).

Instead, use a more modern click-to-select UI.

Test Plan: {F6184343}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13244, T6703

Differential Revision: https://secure.phabricator.com/D20095
2019-02-05 14:18:16 -08:00
epriestley
6f3bd13cf5 Begin adding more guidance to the "One-Time Login" flow
Summary:
Ref T13244. See PHI774. If an install does not use password auth, the "one-time login" flow (via "Welcome" email or "bin/auth recover") is pretty rough. Current behavior:

  - If an install uses passwords, the user is prompted to set a password.
  - If an install does not use passwords, you're dumped to `/settings/external/` to link an external account. This is pretty sketchy and this UI does not make it clear what users are expected to do (link an account) or why (so they can log in).

Instead, improve this flow:

  - Password reset flow is fine.
  - (Future Change) If there are external linkable accounts (like Google) and the user doesn't have any linked, I want to give users a flow like a password reset flow that says "link to an external account".
  - (This Change) If you're an administrator and there are no providers at all, go to "/auth/" so you can set something up.
  - (This Change) If we don't hit on any other rules, just go home?

This may be tweaked a bit as we go, but basically I want to refine the "/settings/external/" case into a more useful flow which gives users more of a chance of surviving it.

Test Plan: Logged in with passwords enabled (got password reset), with nothing enabled as an admin (got sent to Auth), and with something other than passwords enabled (got sent home).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13244

Differential Revision: https://secure.phabricator.com/D20094
2019-02-05 14:17:25 -08:00
epriestley
03eb989fd8 Give Duo MFA a stronger hint if users continue without answering the challenge
Summary: See PHI912. Also, clean up some leftover copy/pastey code here.

Test Plan: {F6182333}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20088
2019-02-05 14:14:41 -08:00
epriestley
ab467d52f4 Improve feed rendering of user rename story
Summary:
Ref T13244. This story publishes to the feed (and I think that's reasonable and desirable) but doesn't render as nicely as it could.

Improve the rendering.

(See T9233 for some context on why we render stories like this one in this way.)

Test Plan: {F6184490}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13244

Differential Revision: https://secure.phabricator.com/D20097
2019-02-05 14:13:44 -08:00
epriestley
5dc6502293 Make the mobile menu available in "/mail/"
Summary: Ref T13244. See <https://discourse.phabricator-community.org/t/left-hand-menu-not-responsive-on-mobile/2358>.

Test Plan: {F6184160}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13244

Differential Revision: https://secure.phabricator.com/D20093
2019-02-05 14:10:57 -08:00
epriestley
ee24eb60b7 In Owners Packages, make the API representation of the "Auditing" field more consistent
Summary:
Ref T13244. See PHI1047. A while ago, the "Review" field changed from "yes/no" to 20 flavors of "Non-Owner Blocking Under A Full Moon". The sky didn't fall, so we'll probably do this to "Audit" eventually too.

The "owners.search" API method anticipates this and returns "none" or "audit" to describe package audit statuses, so it can begin returning "audit-non-owner-reviewers" or whatever in the future.

However, the "owners.edit" API method doesn't work the same way, and takes strings, and the strings have to be numbers. This is goofy and confusing and generally bad.

Make "owners.edit" take the same strings that "owners.search" emits. For now, continue accepting the old values of "0" and "1".

Test Plan:
  - Edited audit status of packages via API using "none", "audit", "0", "1" (worked), and invalid values like "quack" (helpful error).
  - Edited audit status of packages via web UI.
  - Used `owners.search` to retrieve package information.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13244

Differential Revision: https://secure.phabricator.com/D20091
2019-02-05 14:07:23 -08:00
Leopold Schabel
b8fe991ba2 Improve description text in the "Create Diff" form
Summary: The textarea is, in fact, above the description!

Test Plan: Description text changed.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D20092
2019-02-05 13:47:53 +00:00
epriestley
db1e123706 Fix an issue where Duo validation could incorrectly apply to other factor types
See <https://discourse.phabricator-community.org/t/configuring-mfa-provider-totp-fails-for-missing-duo-only-options/2355>.

Test Plan: Created a TOTP provider; created a Duo provider (with missing and supplied values).
2019-02-03 06:36:49 -08:00
epriestley
7c795ab757 Let omnipotent actors skip MFA transactions
Summary: This seems generally reasonable, but is also a narrow fix to "Phacility scripts try to move instances into 'up', but the daemons can't MFA".

Test Plan: Launched a new instance locally, no more "daemons can't MFA" error.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20081
2019-02-01 13:32:09 -08:00
epriestley
942d6d60d5 Don't load unnecessary handle data on "transaction.search"
Summary: Ref T13242. Currently, the transaction query loads handles by default (this is unusual). We don't need them, so turn them off.

Test Plan: No apparent behavioral change, will compare production profiles.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13242

Differential Revision: https://secure.phabricator.com/D20068
2019-02-01 08:14:26 -08:00
epriestley
7dd55a728f Make repository daemons periodically check for out-of-sync repositories
Summary:
See PHI1015. If you add new repository nodes to a cluster, we may not actually sync some repositories for up to 6 hours (if they've had no commits for 30 days).

Add an explicit check for out-of-sync repositories to trigger background sync.

Test Plan:
  - Ran `bin/phd debug pullocal`.
  - Fiddled with the `repository_workingcopy` version table to put the local node in and out of sync with the cluster.
  - Saw appropriate responses in the daemon (sync; wait if the last sync trigger was too recent).

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20078
2019-01-31 22:12:39 -08:00
epriestley
f3e154eb02 Allow "inactive" repositories to be read over SSH for cluster sync
Summary:
Fixes T13192. See PHI1015. When you deactivate a repository, we currently stop serving it.

This creates a problem for intracluster sync, since new nodes can't sync it. If nothing else, this means that if you "ship of theseus" your cluster and turn nodes over one at a time, you will eventually lose the entire repository. Since that's clearly a bad outcome, support sync.

Test Plan:
Testing this requires a "real" cluster, so I mostly used `secure`.

I deactivated rGITTEST and ran this on `secure002`:

```
./bin/repository thaw --demote secure002.phacility.net --force GITTEST && ./bin/repository update GITTEST
```

Before the patch, this failed:

```
[2019-01-31 19:40:37] EXCEPTION: (CommandException) Command failed with error #128!
COMMAND
git fetch --prune -- 'ssh://172.30.0.64:22/diffusion/GITTEST/' '+refs/*:refs/*'

STDOUT
(empty)

STDERR
Warning: Permanently added '172.30.0.64' (RSA) to the list of known hosts.
phabricator-ssh-exec: This repository ("rGITTEST") is not available over SSH.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

After applying (a similar patch to) this patch to `secure001`, the sync worked.

I'll repeat this test with the actual patch once this deploys to `secure`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13192

Differential Revision: https://secure.phabricator.com/D20077
2019-01-31 22:12:13 -08:00
epriestley
bc61d09f55 Allow parent and child revisions to be modified via Conduit
Summary: See PHI1048. We have similar transactions elsewhere already (particularly, on tasks) and these edges don't have any special properties (like "Closes Txx As Wontfix" edges do) so this doesn't create any sort of peril.

Test Plan: {F6170556}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20075
2019-01-31 22:11:17 -08:00
epriestley
b33333ab8e Fix method name typo in new modular transaction text/html mail methods
See PHI1039. See D20057.
2019-01-31 08:52:27 -08:00
epriestley
138c07f32c Allow modular transactions to override transaction title and body text in mail
Summary:
Ref T12921. I'm moving Instances to modular transactions, and we have an "Alert" transaction type used to send notifications ("Your instance is going to be suspended for nonpayment.").

Currently, there's no way to specifically customize mail rendering under modular transactions. Add crude support for it.

Note that (per comment) this is fairly aspirational right now, since we actually always render everything as text (see T12921). But this API should (?) mostly survive intact when I fix this properly, and allows Instances to move to modular transactions so I can fix some more pressing issues in the meantime.

Test Plan: See next diff for Instances.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12921

Differential Revision: https://secure.phabricator.com/D20057
2019-01-30 19:48:30 -08:00
epriestley
87b0ef8839 Remove "iconv" PHP extension dependency
Summary: Depends on D20069. Ref T13232. This is a very, very weak dependency and we can reasonably polyfill it.

Test Plan: Grepped for `iconv` in libphutil, arcanist, and Phabricator.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13232

Differential Revision: https://secure.phabricator.com/D20070
2019-01-30 19:46:58 -08:00
epriestley
48a3760814 Correct a bug where milestone "spacePHID" columns could become desynchronized
Summary:
Depends on D20041. See PHI1046. If you do this:

- Create a parent project called "Crab" in Space 1.
- Create a milestone called "Left Claw".
- Shift "Crab" to Space 2.
- Create a milestone called "Right Claw".

...you currently end up with "Left Claw" in the wrong `spacePHID` in the database. At the application level it's in the correct space, but when we `WHERE ... AND spacePHID IN (...)` we can incorrectly filter it out.

Test Plan:
  - Did the above setup.
  - Saved "Crab", saw the space fix itself.
  - Put things back in the broken state.
  - Ran the migration script, saw things fix themselves again.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: aeiser, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D20063
2019-01-30 19:41:49 -08:00
epriestley
e9b2d667ee Improve handling of "Deny" responses from Duo
Summary:
Ref T13231. See <https://discourse.phabricator-community.org/t/duo-integration-crashes-if-user-is-not-enrolled-and-enrollment-is-disabled/2340/5>

(There's an actual bug here, although I'm not sure exactly what's going on on the Duo side in the report.)

Test Plan:
To reproduce this, I was only able to actually "Deny" my account explicitly in Duo.

  - With "Deny", tried to add a factor. Got a nice helpful error message.
  - Undenied, added a factor, re-denied, tried to pass an MFA gate. Got another nice helpful error message.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13231

Differential Revision: https://secure.phabricator.com/D20065
2019-01-30 19:33:15 -08:00
epriestley
93b512b63c Update a factor query in TransactionEditor for providers
Summary: This query didn't get updated and could let you through an explicit "Sign with MFA" action if you have only disabled factors on your account.

Test Plan:
  - Disabled all factors.
  - Used explicit "Sign With MFA".
    - Before: Went through.
    - After: Sensible error.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20072
2019-01-30 19:27:37 -08:00
epriestley
612e9c6e09 Hide "Signed with MFA" stories from feed
Summary:
See <https://discourse.phabricator-community.org/t/sign-with-mfa-action-on-differential-revisions-creates-strange-feed-entries/2346>.

If you "Sign with MFA" and only add a comment, we can produce a weird notification and feed story. Hide these stories from feed since they're broadly not interesting.

Test Plan:
  - Used "sign with MFA" and only posted a comment.
  - Observed feed.
    - Before: Janky "untitled story".
    - After: Sensible "added a comment" story.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20071
2019-01-30 19:26:33 -08:00
epriestley
6bcfc212eb Allow diff change detection to complete for Mercurial changes which remove a binary file
Summary:
See PHI1044. When you push a commit, we try to detect if the diff is the same as the last diff in Differential. This is a soft/heuristic check and only used to provide a hint in email ("CHANGED SINCE LAST DIFF").

For some changes, we can end up on this pathway with a binary changeset for a deleted file in the commit, and try to fetch the file data. This will fail since the file has been deleted. I'm not //entirely// sure why this hasn't cropped up before and didn't try to truly build an end-to-end test case, but it's a valid state that we shouldn't fatal in.

When we get here in this state, just detect a change. This is safe, even if it isn't always correct.

(This will be revisited in the future so I'm favoring fixing the broken behavior for now.)

Test Plan: This required some rigging, but I modified `bin/differential extract` to run the `isDiffChangedBeforeCommit()` code, then rigged a commit/diff to hit this case and reproduced the reported error. After the change, the comparison worked cleanly.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20056
2019-01-30 19:17:03 -08:00
epriestley
1767b80654 Replace manual query string construction with "phutil_build_http_querystring()"
Summary: Now that we have a nice function for this, use it to simplify some code.

Test Plan: Ran through the Duo enroll workflow to make sure signing still works.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20053
2019-01-30 19:14:57 -08:00
epriestley
a24e6deb57 Read "$_POST" before hooking the profiler, and remove "aphront.default-application-configuration-class"
Summary:
Ref T4369. Ref T12297. Ref T13242. Ref PHI1010. I want to take a quick look at `transaction.search` and see if there's anything quick and obvious we can do to improve performance.

On `secure`, the `__profile__` flag does not survive POST like it's supposed to: when you profile a page and then submit a form on the page, the result is supposed to be profiled. The intent is to make it easier to profile Conduit calls.

I believe this is because we're hooking the profiler, then rebuilding POST data a little later -- so `$_POST['__profile__']` isn't set yet when the profiler checks.

Move the POST rebuild a little earlier to fix this.

Also, remove the very ancient "aphront.default-application-configuration-class". I believe this was only used by Facebook to do CIDR checks against corpnet or something like that. It is poorly named and long-obsolete now, and `AphrontSite` does everything we might reasonably have wanted it to do.

Test Plan: Poked around locally without any issues. Will check if this fixes the issue on `secure`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13242, T12297, T4369

Differential Revision: https://secure.phabricator.com/D20046
2019-01-30 06:22:41 -08:00
epriestley
70b474e550 Allow MFA enrollment guidance to be customized
Summary: Depends on D20039. Ref T13242. If installs want users to install a specific application, reference particular help, etc., let them customize the MFA enrollment message so they can make it say "if you have issues, see this walkthrough on the corporate wiki" or whatever.

Test Plan:
{F6164340}

{F6164341}

{F6164342}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13242

Differential Revision: https://secure.phabricator.com/D20043
2019-01-30 06:21:58 -08:00
epriestley
2374c92544 Add a warning about MFA requirements to edit forms
Summary:
Depends on D20044. Ref T13242. Similar to D20044, add reminder text to edit forms.

It would be nice to "workflow" these so the MFA flow happens inline, but Maniphest's inline edit behavior currently conflicts with this. Set it aside for now since the next workboards iteration (triggers) is probably a good opportunity to revisit it.

Test Plan: {F6164496}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13242

Differential Revision: https://secure.phabricator.com/D20045
2019-01-30 06:21:25 -08:00
epriestley
13c5b427d6 Warn users about MFA requirements when interacting with "MFA Required" objects via the comment form
Summary:
Ref T13242. Warn user that they'll need to MFA (so they can go dig their phone out of their bag first or whatever, or don't type a giant comment on mobile if their U2F key is back at the office) on the comment form.

Also, when they'll need MFA and won't be able to provide it (no MFA on account), stop them from typing up a big comment that they can't actually submit: point them at MFA setup first.

Test Plan:
{F6164448}

{F6164449}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13242

Differential Revision: https://secure.phabricator.com/D20044
2019-01-30 06:20:52 -08:00
epriestley
f7e8fa0764 Provide an Editor extension point for transaction validation
Summary:
Depends on D20040. Ref T13242. See PHI1039. See PHI873. Two reasonable cases have arisen recently where extending validation rules would help solve problems.

We can do this in a pretty straightforward way with a standard extension pattern.

Test Plan:
Used this extension to keep ducks away from projects:

```lang=php
<?php

final class NoDucksEditorExtension
  extends PhabricatorEditorExtension {

  const EXTENSIONKEY = 'no.ducks';

  public function getExtensionName() {
    return pht('No Ducks!');
  }

  public function supportsObject(
    PhabricatorApplicationTransactionEditor $editor,
    PhabricatorApplicationTransactionInterface $object) {
    return ($object instanceof PhabricatorProject);
  }

  public function validateTransactions($object, array $xactions) {
    $errors = array();

    $name_type = PhabricatorProjectNameTransaction::TRANSACTIONTYPE;

    $old_value = $object->getName();
    foreach ($xactions as $xaction) {
      if ($xaction->getTransactionType() !== $name_type) {
        continue;
      }

      $new_value = $xaction->getNewValue();
      if ($old_value === $new_value) {
        continue;
      }

      if (preg_match('/duck/i', $new_value)) {
        $errors[] = $this->newInvalidTransactionError(
          $xaction,
          pht('Project names must not contain the substring "duck".'));
      }
    }

    return $errors;
  }

}
```

{F6162585}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13242

Differential Revision: https://secure.phabricator.com/D20041
2019-01-30 06:18:41 -08:00
epriestley
c9760e8d64 Support subtypes in Projects
Summary:
Ref T13242. See PHI1039. Maniphest subtypes generally seem to be working well. I designed them as a general capability that might be extended to other `EditEngine` objects later, and PHI1039 describes a situation where extending subtypes to projects would give us some reasonable tools.

(Some installs also already use icons/colors as a sort of lightweight version of subtypes, so I believe this is generally useful capability.)

Some of this is a little bit copy-pasted and could probably be shared, but I'd like to wait a bit longer before merging it. For example, both configs have exactly the same structure right now, but Projects should possibly have some different flags (for example: to disable creating subprojects / milestones).

This implementation is pretty basic for now: notably, subprojects/milestones don't get the nice "choose from among subtype forms" treatment that tasks do. If this ends up being part of a solution to PHI1039, I'd plan to fill that in later on.

Test Plan: Defined multiple subtypes, created subtype forms, created projects with appropriate subtypes. Filtered them by subtype. Saw subtype information on list/detail views.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13242

Differential Revision: https://secure.phabricator.com/D20040
2019-01-30 06:17:55 -08:00
epriestley
d254c1f8b1 Use "null", not "-1", as a local "no version" marker when performing intracluster repository sync
Summary:
Ref T13242. See <https://discourse.phabricator-community.org/t/out-of-range-value-for-column-deviceversion/2218>.

The synchronization log column is `uint32?` and `-1` doesn't go into that column.

Since we're only using `-1` for convenience to cheat through `$max_version > $this_version` checks, use `null` instead and make the checks more explicit.

Test Plan: Reproducing this is a bit tricky and I cheated fairly heavily to force the code down this pathway without actually building a multi-device cluster, but I did reproduce the original exception, apply the patch, and observe that it fixed things.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13242

Differential Revision: https://secure.phabricator.com/D20047
2019-01-28 18:50:01 -08:00
epriestley
9fd8343704 Bring Duo MFA upstream
Summary: Depends on D20038. Ref T13231. Although I planned to keep this out of the upstream (see T13229) it ended up having enough pieces that I imagine it may need more fixes/updates than we can reasonably manage by copy/pasting stuff around. Until T5055, we don't really have good tools for managing this. Make my life easier by just upstreaming this.

Test Plan: See T13231 for a bunch of workflow discussion.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13231

Differential Revision: https://secure.phabricator.com/D20039
2019-01-28 18:26:45 -08:00
epriestley
d8d4efe89e Require MFA to edit MFA providers
Summary: Depends on D20037. Ref T13222. Ref T7667. Although administrators can now disable MFA from the web UI, at least require that they survive MFA gates to do so. T7667 (`bin/auth lock`) should provide a sturdier approach here in the long term.

Test Plan: Created and edited MFA providers, was prompted for MFA.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T7667

Differential Revision: https://secure.phabricator.com/D20038
2019-01-28 09:44:39 -08:00
epriestley
44a0b3e83d Replace "Show Secret" in Passphrase with one-shot MFA
Summary: Depends on D20036. Ref T13222. Now that we support one-shot MFA, swap this from session MFA to one-shot MFA.

Test Plan: Revealed a credential, was no longer left in high-security mode.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20037
2019-01-28 09:44:08 -08:00
epriestley
d24e66724d Convert "Rename User" from session MFA to one-shot MFA
Summary:
Depends on D20035. Ref T13222.

  - Allow individual transactions to request one-shot MFA if available.
  - Make "change username" request MFA.

Test Plan:
  - Renamed a user, got prompted for MFA, provided it.
  - Saw that I no longer remain in high-security after performing the edit.
  - Grepped for other uses of `PhabricatorUserUsernameTransaction`, found none.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20036
2019-01-28 09:41:10 -08:00
epriestley
29b4fad941 Get rid of "throwResult()" for control flow in MFA factors
Summary: Depends on D20034. Ref T13222. This is just cleanup -- I thought we'd have like two of these, but we ended up having a whole lot in Duo and a decent number in SMS. Just let factors return a result explicitly if they can make a decision early. I think using `instanceof` for control flow is a lesser evil than using `catch`, on the balance.

Test Plan: `grep`, went through enroll/gate flows on SMS and Duo.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20035
2019-01-28 09:40:28 -08:00
epriestley
bce44385e1 Add more factor details to the Settings factor list
Summary:
Depends on D20033. Ref T13222. Flesh this UI out a bit, and provide bit-strength information for TOTP.

Also, stop users from adding multiple SMS factors since this is pointless (they all always text your primary contact number).

Test Plan:
{F6156245}

{F6156246}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20034
2019-01-28 09:40:00 -08:00
epriestley
8e5d9c6f0e Allow MFA providers to be deprecated or disabled
Summary: Ref T13222. Providers can now be deprecated (existing factors still work, but users can't add new factors for the provider) or disabled (factors stop working, also can't add new ones).

Test Plan:
  - Enabled, deprecated, and disabled some providers.
  - Viewed provider detail, provider list.
  - Viewed MFA settings list.
  - Verified that I'm prompted for enabled + deprecated only at gates.
  - Tried to disable final provider, got an error.
  - Hit the MFA setup gate by enabling "Require MFA" with no providers, got a more useful message.
  - Immediately forced a user to the "MFA Setup Gate" by disabling their only active provider with another provider enabled ("We no longer support TOTP, you HAVE to finish Duo enrollment to continue starting Monday.").

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20031
2019-01-28 09:29:27 -08:00
epriestley
c9ff6ce390 Add CSRF to SMS challenges, and pave the way for more MFA types (including Duo)
Summary:
Depends on D20026. Ref T13222. Ref T13231. The primary change here is that we'll no longer send you an SMS if you hit an MFA gate without CSRF tokens.

Then there's a lot of support for genralizing into Duo (and other push factors, potentially), I'll annotate things inline.

Test Plan: Implemented Duo, elsewhere.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13231, T13222

Differential Revision: https://secure.phabricator.com/D20028
2019-01-24 15:10:57 -08:00
epriestley
069160404f Add a Duo API future
Summary: Depends on D20025. Ref T13231. Although I'm not currently planning to actually upstream a Duo MFA provider, it's probably easiest to put most of the support pieces in the upstream until T5055.

Test Plan: Used a test script to make some (mostly trivial) API calls and got valid results back, so I think the parameter signing is correct.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13231

Differential Revision: https://secure.phabricator.com/D20026
2019-01-24 15:10:17 -08:00
epriestley
98c4cdc5be Make the "PHP 7" setup warning more explicit about what it means
Summary: See <https://discourse.phabricator-community.org/t/minimum-php-versions-supported-by-latest-stable-phabricator/2314/3>.

Test Plan: o~o

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20027
2019-01-24 15:09:00 -08:00
epriestley
5cfcef7f53 Fix bad "$this" references in "Must Encrypt" mail after MailEngine changes
Summary: See PHI1038. I missed these when pulling the code out.

Test Plan: Sent "Must encrypt" mail, verified it made it through the queue in one piece.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20029
2019-01-24 15:07:42 -08:00
Austin McKinley
e72684a4ba Add infrastructure for sending SMS via AWS SNS
Summary:
Ref T920. Ref T13235. This adds a `Future`, similar to `TwilioFuture`, for interacting with Amazon's SNS service.

Also updates the documentation.

Also makes the code consistent with the documentation by accepting a `media` argument.

Test Plan: Clicked the "send test message" button from the Settings UI.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T13235, T920

Differential Revision: https://secure.phabricator.com/D19982
2019-01-23 16:29:50 -08:00
epriestley
ab2cbbd9f9 Add a "test message" action for contact numbers
Summary: Depends on D20024. See D20022. Put something in place temporarily until we build out validation at some point.

Test Plan: Sent myself a test message.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20025
2019-01-23 14:22:27 -08:00
epriestley
587e9cea19 Always require MFA to edit contact numbers
Summary:
Depends on D20023. Ref T13222. Although I think this isn't strictly necessary from a pure security perspective (since you can't modify the primary number while you have MFA SMS), it seems like a generally good idea.

This adds a slightly new MFA mode, where we want MFA if it's available but don't strictly require it.

Test Plan: Disabled, enabled, primaried, unprimaried, and edited contact numbers. With MFA enabled, got prompted for MFA. With no MFA, no prompts.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20024
2019-01-23 14:19:56 -08:00
epriestley
7805b217ad Prevent users from editing, disabling, or swapping their primary contact number while they have SMS MFA
Summary:
Depends on D20022. Ref T13222. Since you can easily lock yourself out of your account by swapping to a bad number, prevent contact number edits while "contact number" MFA (today, always SMS) is enabled.

(Another approach would be to bind factors to specific contact numbers, and then prevent that number from being edited or disabled while SMS MFA was attached to it. However, I think that's a bit more complicated and a little more unwieldy, and ends up in about the same place as this. I'd consider it more strongly in the future if we had like 20 users say "I have 9 phones" but I doubt this is a real use case.)

Test Plan:
  - With SMS MFA, tried to edit my primary contact number, disable it, and promote another number to become primary. Got a sensible error message in all cases.
  - After removing SMS MFA, did all that stuff with no issues.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20023
2019-01-23 14:18:33 -08:00
epriestley
ada8a56bb7 Implement SMS MFA
Summary:
Depends on D20021. Ref T13222. This has a few rough edges, including:

  - The challenges theselves are CSRF-able.
  - You can go disable/edit your contact number after setting up SMS MFA and lock yourself out of your account.
  - SMS doesn't require MFA so an attacker can just swap your number to their number.

...but mostly works.

Test Plan:
  - Added SMS MFA to my account.
  - Typed in the number I was texted.
  - Typed in some other different numbers (didn't work).
  - Cancelled/resumed the workflow, used SMS in conjunction with other factors, tried old codes, etc.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20022
2019-01-23 14:17:38 -08:00
epriestley
6c11f37396 Add a pre-enroll step for MFA, primarily as a CSRF gate
Summary:
Depends on D20020. Ref T13222. This puts another step in the MFA enrollment flow: pick a provider; read text and click "Continue"; actually enroll.

This is primarily to stop CSRF attacks, since otherwise an attacker can put `<img src="phabricator.com/auth/settings/enroll/?providerPHID=xyz" />` on `cute-cat-pix.com` and get you to send yourself some SMS enrollment text messages, which would be mildly annoying.

We could skip this step if we already have a valid CSRF token (and we often will), but I think there's some value in doing it anyway. In particular:

  - For SMS/Duo, it seems nice to have an explicit "we're about to hit your phone" button.
  - We could let installs customize this text and give users a smoother onboard.
  - It allows the relatively wordy enroll form to be a little less wordy.
  - For tokens which can expire (SMS, Duo) it might save you from answering too slowly if you have to go dig your phone out of your bag downstairs or something.

Test Plan: Added factors, read text. Tried to CSRF the endpoint, got a dialog instead of a live challenge generation.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20021
2019-01-23 14:16:57 -08:00
epriestley
f3340c6335 Allow different MFA factor types (SMS, TOTP, Duo, ...) to share "sync" tokens when enrolling new factors
Summary:
Depends on D20019. Ref T13222. Currently, TOTP uses a temporary token to make sure you've set up the app on your phone properly and that you're providing an answer to a secret which we generated (not an attacker-generated secret).

However, most factor types need some kind of sync token. SMS needs to send you a code; Duo needs to store a transaction ID. Turn this "TOTP" token into an "MFA Sync" token and lift the implementation up to the base class.

Also, slightly simplify some of the HTTP form gymnastics.

Test Plan:
  - Hit the TOTP enroll screen.
  - Reloaded it, got new secrets.
  - Reloaded it more than 10 times, got told to stop generating new challenges.
  - Answered a challenge properly, got a new TOTP factor.
  - Grepped for removed class name.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20020
2019-01-23 14:13:50 -08:00
epriestley
7c1d1c13f4 Add a rate limit for enroll attempts when adding new MFA configurations
Summary:
Depends on D20018. Ref T13222. When you add a new MFA configuration, you can technically (?) guess your way through it with brute force. It's not clear why this would ever really be useful (if an attacker can get here and wants to add TOTP, they can just add TOTP!) but it's probably bad, so don't let users do it.

This limit is fairly generous because I don't think this actually part of any real attack, at least today with factors we're considering.

Test Plan:
  - Added TOTP, guessed wrong a ton of times, got rate limited.
  - Added TOTP, guessed right, got a TOTP factor configuration added to my account.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20019
2019-01-23 14:12:19 -08:00
epriestley
e91bc26da6 Don't rate limit users clicking "Wait Patiently" at an MFA gate even if they typed some text earlier
Summary:
Depends on D20017. Ref T13222. Currently, if you:

  - type some text at a TOTP gate;
  - wait ~60 seconds for the challenge to expire;
  - submit the form into a "Wait patiently" message; and
  - mash that wait button over and over again very patiently

...you still rack up rate limiting points, because the hidden text from your original request is preserved and triggers the "is the user responding to a challenge" test. Only perform this test if we haven't already decided that we're going to make them wait.

Test Plan:
  - Did the above; before patch: rate limited; after patch: not rate limited.
  - Intentionally typed a bunch of bad answers which were actually evaluated: rate limited properly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20018
2019-01-23 14:11:24 -08:00
epriestley
bb20c13651 Allow MFA factors to provide more guidance text on create workflows
Summary:
Depends on D20016. Ref T920. This does nothing interesting on its own since the TOTP provider has no guidance/warnings, but landing it separately helps to simplify an upcoming SMS diff.

SMS will have these guidance messages:

  - "Administrator: you haven't configured any mailer which can send SMS, like Twilio."
  - "Administrator: SMS is weak."
  - "User: you haven't configured a contact number."

Test Plan: {F6151283} {F6151284}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D20017
2019-01-23 14:10:16 -08:00
epriestley
ee7b03bdf7 Correct an issue where the default "Settings" screen could show the wrong settings
Summary:
Depends on D20013. Recently, I renamed the "Account" panel to "Language".

When you land on "Settings" and the first panel is an "EditEngine" panel ("Account/Langauge", "Date and Time", and "Conpherence" are all "EditEngine" panels), the engine shows the controls for the first panel.

However, the "first panel" according to EditEngine and the "first panel" in the menu are currently different: the menu groups panels into topics.

When I renamed "Account" to "Language", it went from conicidentally being the first panel in both lists to being the second panel in the grouped menu list and the, uh, like 12th panel in the ungrouped raw list.

This made landing on "Settings" show you the right chrome, but show you a different panel's controls ("Conpherence", now alphabetically first).

Instead, use the same order in both places.

(This was also a pre-existing bug if you use a language which translates the panel names such that "Account" is not alphabetically first.)

Test Plan: Visited "Settings", saw "Date & Time" form controls instead of "Conpherence" form controls on the default screen with "Date & Time" selected in the menu.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20016
2019-01-23 14:09:03 -08:00
epriestley
f69fbf5ea6 Make the "Test" adapter support both SMS and email
Summary:
Depends on D20012. Ref T920. If you have a test adapter configured, it should swallow messages and prevent them from ever hitting a lower-priority adapter.

Make the test adapter support SMS so this actually happens.

Test Plan: Ran `bin/mail send-test --type sms ...` with a test adapter (first) and a Twilio adapter (second). Got SMS swallowed by test adapter instead of live SMS messages.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D20013
2019-01-23 14:07:07 -08:00
epriestley
af71c51f0a Give "MetaMTAMail" a "message type" and support SMS
Summary:
Depends on D20011. Ref T920. This change lets a "MetaMTAMail" storage object represent various different types of messages, and  makes "all" the `bin/mail` stuff "totally work" with messages of non-email types.

In practice, a lot of the related tooling needs some polish/refinement, but the basics work.

Test Plan: Used `echo beep boop | bin/mail send-test --to epriestley --type sms` to send myself SMS.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D20012
2019-01-23 14:05:46 -08:00
epriestley
596435b35e Support designating a contact number as "primary"
Summary:
Depends on D20010. Ref T920. Allow users to designate which contact number is "primary": the number we'll actually send stuff to.

Since this interacts in weird ways with "disable", just do a "when any number is touched, put all of the user's rows into the right state" sort of thing.

Test Plan:
  - Added numbers, made numbers primary, disabled a primary number, un-disabled a number with no primaries. Got sensible behavior in all cases.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D20011
2019-01-23 14:03:08 -08:00
epriestley
12203762b7 Allow contact numbers to be enabled and disabled
Summary: Depends on D20008. Ref T920. Continue fleshing out contact number behaviors.

Test Plan:
  - Enabled and disabled a contact number.
  - Saw list, detail views reflect change.
  - Added number X, disabled it, added it again (allowed), enabled the disabled one ("already in use" exception).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D20010
2019-01-23 13:59:55 -08:00
epriestley
c4244aa177 Allow users to access some settings at the "Add MFA" account setup roadblock
Summary:
Depends on D20006. Ref T13222. Currently, the "MFA Is Required" gate doesn't let you do anything else, but you'll need to be able to access "Contact Numbers" if an install provides SMS MFA.

Tweak this UI to give users limited access to settings, so they can set up contact numbers and change their language.

(This is a little bit fiddly, and I'm doing it early on partly so it can get more testing as these changes move forward.)

Test Plan: {F6146136}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D20008
2019-01-23 13:43:28 -08:00
epriestley
d6d93dd658 Add icons to Settings
Summary: Depends on D20005. I love icons.

Test Plan: {F6145996}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20006
2019-01-23 13:41:41 -08:00
epriestley
f713fa1fd7 Expand "Settings" UI to full-width
Summary:
Depends on D19988. See D19826 for the last UI expansion. I don't have an especially strong product rationale for un-fixed-width'ing Settings since it doesn't suffer from the "mystery meat actions" issues that other fixed-width UIs do, but I like the full-width UI better and the other other fixed-width UIs all (?) have some actual rationale (e.g., large tables, multiple actions on subpanels), so "consistency" is an argument here.

Also rename "account" to "language" since both settings are language-related.

This moves away from the direction in D18436.

Test Plan:
Clicked each Settings panel, saw sensible rendering at full-width.

{F6145944}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20005
2019-01-23 13:40:34 -08:00
epriestley
f0c6ee4823 Add "Contact Numbers" so we can send users SMS mesages
Summary:
Ref T920. To send you SMS messages, we need to know your phone number.

This adds bare-bone basics (transactions, storage, editor, etc).

From here:

**Disabling Numbers**: I'll let you disable numbers in an upcoming diff.

**Primary Number**: I think I'm just going to let you pick a number as "primary", similar to how email works. We could imagine a world where you have one "MFA" number and one "notifications" number, but this seems unlikely-ish?

**Publishing Numbers (Profile / API)**: At some point, we could let you say that a number is public / "show on my profile" and provide API access / directory features. Not planning to touch this for now.

**Non-Phone Numbers**: Eventually this could be a list of other similar contact mechanisms (APNS/GCM devices, Whatsapp numbers, ICQ number, twitter handle so MFA can slide into your DM's?). Not planning to touch this for now, but the path should be straightforward when we get there. This is why it's called "Contact Number", not "Phone Number".

**MFA-Required + SMS**: Right now, if the only MFA provider is SMS and MFA is required on the install, you can't actually get into Settings to add a contact number to configure SMS. I'll look at the best way to deal with this in an upcoming diff -- likely, giving you partial access to more of Setings before you get thorugh the MFA gate. Conceptually, it seems reasonable to let you adjust some other settings, like "Language" and "Accessibility", before you set up MFA, so if the "you need to add MFA" portal was more like a partial Settings screen, maybe that's pretty reasonable.

**Verifying Numbers**: We'll probably need to tackle this eventually, but I'm not planning to worry about it for now.

Test Plan: {F6137174}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: avivey, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19988
2019-01-23 13:39:56 -08:00
epriestley
aa48373889 Update bin/auth MFA commands for the new "MFA Provider" indirection layer
Summary:
Ref T13222. This updates the CLI tools and documentation for the changes in D19975.

The flags `--type` and `--all-types` retain their current meaning. In most cases, `bin/auth strip --type totp` is sufficient and you don't need to bother looking up the relevant provider PHID. The existing `bin/auth list-factors` is also unchanged.

The new `--provider` flag allows you to select configs from a particular provider in a more granular way. The new `bin/auth list-mfa-providers` provides an easy way to get PHIDs.

(In the Phacility cluster, the "Strip MFA" action just reaches into the database and deletes rows manually, so this isn't terribly important. I verified that the code should still work properly.)

Test Plan:
  - Ran `bin/auth list-mfa-providers`.
  - Stripped by user / type / provider.
  - Grepped for `list-factors` and `auth strip`.
  - Hit all (?) of the various possible error cases.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19976
2019-01-23 13:38:44 -08:00
epriestley
0fcff78253 Convert user MFA factors to point at configurable "MFA Providers", not raw "MFA Factors"
Summary:
Ref T13222. Users configure "Factor Configs", which say "I have an entry on my phone for TOTP secret key XYZ".

Currently, these point at raw implementations -- always "TOTP" in practice.

To support configuring available MFA types (like "no MFA") and adding MFA types that need some options set (like "Duo", which needs API keys), bind "Factor Configs" to a "Factor Provider" instead.

In the future, several "Factors" will be available (TOTP, SMS, Duo, Postal Mail, ...). Administrators configure zero or more "MFA Providers" they want to use (e.g., "Duo" + here's my API key). Then users can add configs for these providers (e.g., "here's my Duo account").

Upshot:

  - Factor: a PHP subclass, implements the technical details of a type of MFA factor (TOTP, SMS, Duo, etc).
  - FactorProvider: a storage object, owned by administrators, configuration of a Factor that says "this should be available on this install", plus provides API keys, a human-readable name, etc.
  - FactorConfig: a storage object, owned by a user, says "I have a factor for provider X on my phone/whatever with secret key Q / my duo account is X / my address is Y".

Couple of things not covered here:

  - Statuses for providers ("Disabled", "Deprecated") don't do anything yet, but you can't edit them anyway.
  - Some `bin/auth` tools need to be updated.
  - When no providers are configured, the MFA panel should probably vanish.
  - Documentation.

Test Plan:
  - Ran migration with providers, saw configs point at the first provider.
  - Ran migration without providers, saw a provider created and configs pointed at it.
  - Added/removed factors and providers. Passed MFA gates. Spot-checked database for general sanity.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19975
2019-01-23 13:37:43 -08:00
Austin McKinley
6138d5885d Update documentation to reflect bin/auth changes
Summary: See https://secure.phabricator.com/D18901#249481. Update the docs and a warning string to reflect the new reality that `bin/auth recover` is now able to recover any account, not just administrators.

Test Plan: Mk 1 eyeball

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D20007
2019-01-21 12:19:54 -08:00
epriestley
881d79c1ea When dirtying repository cluster routing caches after an Almanac edit, discover linked bindings from devices
Summary:
See PHI1030. When you edit an Almanac object, we attempt to discover all the related objects so we can dirty the repository cluster routing cache: if you modify a Device or Service that's part of a clustered repository, we need to blow away our cached view of the layout.

Currently, we don't correctly find linked Bindings when editing a Device, so we may miss Services which have keys that need to be disabled. Instead, discover these linked objects.

See D17000 for the original implementation and more context.

Test Plan:
  - Used `var_dump()` to dump out the discovered objects and dirtied cache keys.
  - Before change: editing a Service dirties repository routing keys (this is correct), but editing a Device does not.
  - After change: editing a Device now correctly dirties repository routing keys.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20003
2019-01-21 10:32:48 -08:00
epriestley
afd2ace0dc Apply inverse edge edits after committing primary object edits
Summary:
Fixes T13082. When you create a revision (say, `D111`) with `Ref T222` in the body, we write a `D111 -> T222` edge ("revision 111 references task 222") and an inverse `T222 -> D111` edge ("task 222 is referenced by revision 111").

We also apply a transaction to `D111` ("alice added a task: Txxx.") and an inverse transaction to `T222` ("alice added a revision: Dxxx").

Currently, it appears that the inverse transaction can sometimes generate mail faster than `D111` actually commits its (database) transactions, so the mail says "alice added a revision: Unknown Object (Differential Revision)". See T13082 for evidence that this is true, and a reproduction case.

To fix this, apply the inverse transaction (to `T222`) after we commit the main object (here, `D111`).

This is tricky because when we apply transactions, the transaction editor automatically "fixes" them to be consistent with the database state. For example, if a task already has title "XYZ" and you set the title to "XYZ" (same title), we just no-op the transaction.

It also fixes edge edits. The old sequence was:

  - Open (database) transaction.
  - Apply our transaction ("alice added a task").
  - Apply the inverse transaction ("alice added a revision").
  - Write the edges to the database.
  - Commit (database) transaction.

Under this sequence, the inverse transaction was "correct" and didn't need to be fixed, so the fixing step didn't touch it.

The new sequence is:

  - Open (database) transaction.
  - Apply our transaction ("alice added a task").
  - Write the edges.
  - Commit (database) transaction.
  - Apply the inverse transaction ("alice added a revision").

Since the inverse transaction now happens after the database edge write, the fixing step detects that it's a no-op and throws it away if we do this naively.

Instead, add some special cases around inverse edits to skip the correction/fixing logic, and just pass the "right" values in the first place.

Test Plan:
Added and removed related tasks from revisions, saw appropriate transactions render on both objects.

(It's hard to be certain this completely fixes the issue since it only happened occasionally in the first place, but we can see if it happens any more on `secure`.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13082, T222

Differential Revision: https://secure.phabricator.com/D19969
2019-01-20 21:03:20 -08:00
epriestley
0db29e624c Provide a richer error when an intracluster request can not be satisfied by the target node
Summary: See PHI1030. When installs hit this error, provide more details about which node we ended up on and what's going on.

Test Plan:
```
$ git pull
phabricator-ssh-exec: This repository request (for repository "spellbook") has been incorrectly routed to a cluster host (with device name "local.phacility.net", and hostname "orbital-3.local") which can not serve the request.

The Almanac device address for the correct device may improperly point at this host, or the "device.id" configuration file on this host may be incorrect.

Requests routed within the cluster by Phabricator are always expected to be sent to a node which can serve the request. To prevent loops, this request will not be proxied again.

(This is a read request.)
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20002
2019-01-20 17:26:06 -08:00
epriestley
be8b7c9eba Fix "Welcome Mail" check for a message when no message exists
Summary: Fixes T13239. See that task for discussion.

Test Plan: Tried to send welcome mail with no "Welcome" message.

Maniphest Tasks: T13239

Differential Revision: https://secure.phabricator.com/D20001
2019-01-20 07:00:33 -08:00
epriestley
755c40221d Temporarily disable transaction story links in HTML mail for the deploy
Ref T12921. See that task for discussion. Behavioral revert of D19968.
2019-01-19 05:10:02 -08:00
epriestley
e6ca2b998f Allow Conduit method call logs to be exported with the standard export pipeline
Summary:
See PHI1026. Allow installs to export Conduit call logs to a flat format.

Also, add date range queries.

Test Plan:
  - Exported some call logs.
  - Filtered logs by date.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19996
2019-01-18 20:09:57 -08:00
epriestley
6bb31de305 Use the customizable "Welcome Mail" message in welcome mail
Summary:
Depends on D19994. See PHI1027. If an install has customized the "Welcome Mail" message, include it in welcome mail. A special custom message from the profile screen overrides it, if provided.

(I fiddled with putting the custom message as "placeholder" text in the remarkup area as a hint, but newlines in "placeholder" text appear to have issues in Safari and Firefox. I think this is probably reasonably clear as-is.)

Make both render remarkup-into-text so things like links work properly, as it's reasonably likely that installs will want to link to things.

Test Plan:
  - With custom "Welcome Mail" text, sent mail with no custom override (got custom text) and a custom override (got overridden text).
  - Linked to some stuff, got sensible links in the mail (`bin/mail show-outbound`).

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19995
2019-01-18 19:55:44 -08:00
epriestley
22ad1ff2c5 Show the customized "Login" message on the login screen
Summary: Depends on D19992. Ref T13222. If administrators provide a custom login message, show it on the login screen.

Test Plan:
{F6137930}

  - Viewed login screen with and without a custom message.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19994
2019-01-18 19:54:02 -08:00
epriestley
2c713b2d25 Add "Auth Messages" to support customizing onboarding/welcome flows
Summary:
Ref T13222. Long ago, we had a Config option (`welcome.html`) to let you dump HTML onto the login screen, but this was relatively hard to use and not good from a security perspective.

In some cases this was obsoleted by Dashboards, but there's at least some remaining set of use cases for actual login instructions on the login screen. For example, WMF has some guidance on //which// SSO mechanism to use based on what types of account you have. On `secure`, users assume they can register by clicking "Log In With GitHub" or whatever, and it might reduce frustration to tell them upfront that registration is closed.

Some other types of auth messaging could also either use customization or defaults (e.g., the invite/welcome/approve mail).

We could do this with a bunch of Config options, but I'd generally like to move to a world where there's less stuff in Config and more configuration is contextual. I think it tends to be easier to use, and we get a lot of fringe benefits (granular permissions, API, normal transaction logs, more abililty to customize workflows and provide contextual help/hints, etc). Here, for example, we can provide a remarkup preview, which would be trickier with Config.

This does not actually do anything yet.

Test Plan: {F6137541}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19992
2019-01-18 19:53:19 -08:00
epriestley
ab7aceaabf Allow administrators to provide custom welcome text when welcoming users on the profile workflow
Summary:
See PHI1027. Currently, we allow you to customize invite email, but not most other types of email (approve, welcome). As a step forward, also allow welcome email to be customized with a message.

I considered separating the custom text from the main text with something heavyhanded ("alice added this custom message:") or a beautiful ASCII art divider like one of these:

https://www.asciiart.eu/art-and-design/dividers

...but nothing truly sung to me.

This only works on the profile flow for now. I'm planning to let you set a default message. I may or may not let you customize from "Create New User", seems like the default message probably covers most of that. Probably won't touch `scripts/user/add_user.php` since that's not really exactly super supported.

Test Plan:
Sent mail with and without custom messages, reviewed it with `bin/mail show-outbound`.

{F6137410}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19991
2019-01-18 19:52:40 -08:00
epriestley
7f950f520b When password auth is not enabled, don't tell users to set a password in welcome email
Summary:
See PHI1027. Currently, the "Welcome" mail always tells users to set a password. This definitely isn't helpful if an install doesn't have password auth enabled.

We can't necessarily guess what they're supposed to do, so just give them generic instructions ("set up your account"). Upcoming changes will give administrators more control over the mail content.

Test Plan: Sent both versions of the mail, used `bin/mail show-outbound` to inspect them for correctness.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19990
2019-01-18 19:51:16 -08:00
epriestley
5537e29ee8 Move "Welcome" mail generation out of PhabricatorUser
Summary:
Ref PHI1027. Currently, `PhabricatorUser` has a couple of mail-related methods which shouldn't really be there in the long term. Immediately, I want to make some adjusments to the welcome email.

Move "Welcome" mail generation to a separate class and consolidate all the error handling. (Eventually, "invite" and "verify address" email should move to similar subclasses, too.) Previously, a bunch of errors/conditions got checked in multiple places.

The only functional change is that we no longer allow you to send welcome mail to disabled users.

Test Plan:
  - Used "Send Welcome Mail" from profile pages to send mail.
  - Hit "not admin", "disabled user", "bot/mailing list" errors.
  - Used `scripts/user/add_user.php` to send welcome mail.
  - Used "Create New User" to send welcome mail.
  - Verified mail with `bin/mail show-outbound`. (Cleaned up a couple of minor display issues here.)

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19989
2019-01-18 19:50:35 -08:00
epriestley
98bf3a950d Add setup warnings for "local_infile" (MySQL Server) and "mysql[i].allow_local_infile" (PHP Client)
Summary: Ref T13238. Warn users about these horrible options and encourage them to defuse them.

Test Plan: Hit both warnings, fixed the issues, issues went away.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13238

Differential Revision: https://secure.phabricator.com/D19999
2019-01-18 19:49:48 -08:00
epriestley
310ad7f8f4 Put a hard limit on password login attempts from the same remote address
Summary:
Ref T13222. Currently, if a remote address fails a few login attempts (5) in a short period of time (15 minutes) we require a CAPTCHA for each additional attempt.

This relies on:

  - Administrators configuring ReCAPTCHA, which they may just not bother with.
  - Administrators being comfortable with Google running arbitrary trusted Javascript, which they may not be comfortable with.
  - ReCAPTCHA actually being effective, which seems likely true for unsophisticated attackers but perhaps less true for more sophisticated attackers (see <https://github.com/ecthros/uncaptcha2>, for example).

(For unsophisticated attackers and researchers, "Rumola" has been the standard CAPTCHA bypass tool for some time. This is an extension that pays humans to solve CAPTCHAs for you. This is not practical at "brute force a strong password" scale. Google appears to have removed it from the Chrome store. The "submit the captcha back to Google's APIs" trick probably isn't practical at brute-force-scale either, but it's easier to imagine weaponizing that than weaponizing human solvers.)

Add a hard gate behind the CAPTHCA wall so that we fail into a secure state if there's no CAPTCHA or the attacker can defeat CAPTCHAs at a very low cost.

The big downside to this is that an attacker who controls your remote address (e.g., is behind the same NAT device you're behind on corpnet) can lock you out of your account. However:

  - That //should// be a lot of access (although maybe this isn't that high of a barrier in many cases, since compromising a "smart fridge" or "smart water glass" or whatever might be good enough).
  - You can still do "Forgot password?" and login via email link, although this may not be obvious.

Test Plan:
  - Logged in normally.
  - Failed many many login attempts, got hard gated.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19997
2019-01-18 19:48:42 -08:00
epriestley
c125ab7a42 Remove "metamta.*.subject-prefix" options
Summary:
In ~2012, the first of these options was added because someone who hates dogs and works at Asana also hated `[Differential]` in the subject line. The use case there was actually //removing// the text, not changing it, but I made the prefix editable since it seemed like slightly less of a one-off.

These options are among the dumbest and most useless config options we have and very rarely used, see T11760. A very small number of instances have configured one of these options.

Newer applications stopped providing these options and no one has complained.

You can get the same effect with `translation.override`. Although I'm not sure we'll keep that around forever, it's a reasonable replacement today. I'll call out an example in the changelog to help installs that want to preserve this option.

If we did want to provide this, it should just be in {nav Applications > Settings} for each application, but I think it's wildly-low-value and "hack via translations" or "local patch" are entirely reasonable if you really want to change these strings.

Test Plan: Grepped for `subject-prefix`.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19993
2019-01-17 19:18:50 -08:00
epriestley
ff220acae6 Don't bounce mail messages if any recipient was reserved
Summary:
Ref T13222. If we receive a message and nothing processes it, we normally try to send the user an error message like "hey, nothing handled this, maybe you got the address wrong".

Just skip the "send them an error message" part if any recipient was reserved, so if you "Reply All" to a message that is "From: noreply@phabricator" you don't get a relatively unhelpful error.

This also makes sure that the "void" address doesn't generate bounces if the "From" is a valid user email address (e.g., with `metamta.can-send-as-user`). That is:

  - Phabricator needs to send a mail with only "CC" users.
  - Phabricator puts the "void" address in "To" as a placeholder.
  - The "void" address happens to route back to Phabricator.

We don't want that mail to bounce to anywhere. Normally, it won't:

  - From is usually "noreply@phabricator", which isn't a user, so we won't send anything back: we only send mail to verified user email addresses.
  - The message will have "X-Phabricator-Sent-This-Message: true" so we won't process it at all.

...but this is another layer of certainty.

Test Plan: Used `bin/mail receive-test` to receive mail to an invalid, unreserved address (bounce/error email) and an invalid, reserved address (no bounce/error email).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19987
2019-01-17 19:17:37 -08:00
epriestley
6b6c991ad4 Allow Phortune accounts to customize their billing address and name
Summary:
See PHI1023. Ref T7607. Occasionally, companies need their billing address (or some other custom text) to appear on invoices to satisfy process or compliance requirements.

Allow accounts to have a custom "Billing Name" and a custom "Billing Address" which appear on invoices.

Test Plan: {F6134707}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T7607

Differential Revision: https://secure.phabricator.com/D19979
2019-01-16 16:16:27 -08:00
epriestley
a1516fefb6 Fix an issue where "Import Columns" could fail on a board for a project with milestones
Summary:
See PHI1025. When you "Import Columns", we test if you're trying to import into a board that already has columns. However, this test is too broad (it incorrectly detects "proxy" columns for milestones as columns) and not user-friendly (it returns 400 instead of a readable error).

Correct these issues, and refine some of the logic around proxy columns.

Test Plan:
  - Created a project, A.
  - Created a milestone under that project.
  - Imported another project's columns to A's workboard.
    - Before change: Unhelpful 400.
    - After change: import worked fine.
  - Also, hit the new error dialogs and read through them.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19978
2019-01-16 16:15:45 -08:00
epriestley
c5f446defb Prevent application email addresses from shadowing user email addresses
Summary:
Fixes T13234. Don't let application email addresses be configured with user addresses. This might prevent an unlikely bit of mischief where someone does this intentionally, detailed in T13234.

(Possibly, these tables should just be merged some day, similar to how the "Password" table is now a shared resource that's modular enough for multiple applications to use it.)

Test Plan: {F6132259}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13234

Differential Revision: https://secure.phabricator.com/D19974
2019-01-16 13:28:08 -08:00
epriestley
dc4d7f1f3e Reorder "Merge" transaction to make "Close as Duplicate" produce a "[Merged]" email subject
Summary:
Fixes T11782. When you "Close as Duplicate", generate a "[Merged]" email by making the merge the first transaction.

(There are other, more-deterministic ways to do this with action strength, but this is much simpler and I believe it suffices.)

Test Plan: Used "Close as Duplicate", got a "[Merged]" email out of it.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11782

Differential Revision: https://secure.phabricator.com/D19972
2019-01-16 13:27:10 -08:00
epriestley
35f0e31ed3 Add a Twilio SMS message adapter
Summary: Ref T920. Adds a "phone number" object, an "SMS" message type, and Twilio glue.

Test Plan:
Used this test script to send myself some text messages after configuring Twilio in `cluster.mailers`.

```
<?php

require_once 'scripts/init/init-script.php';

if ($argc < 3) {
  throw new Exception('usage: test.php <number> <body>');
}
$to_number = $argv[1];
$text_body = $argv[2];

$mailers = PhabricatorMetaMTAMail::newMailers(
  array(
    'outbound' => true,
    'media' => array(
      PhabricatorMailSMSMessage::MESSAGETYPE,
    ),
  ));
if (!$mailers) {
  return new Aphront404Response();
}
$mailer = head($mailers);

$message = id(new PhabricatorMailSMSMessage())
  ->setToNumber(new PhabricatorPhoneNumber($to_number))
  ->setTextBody($text_body);

$mailer->sendMessage($message);
```

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19971
2019-01-16 13:25:59 -08:00
epriestley
96d3e73eed Fix an issue where "CC"-only email improperly wiped CC addresses
Summary: Ref T920. See <https://discourse.phabricator-community.org/t/ccd-emails-not-working-with-sendgrid-since-2019-week-1-update/2294>.

Test Plan: Used `bin/mail send-test --cc ...` without `--to`, got email.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19970
2019-01-16 13:22:43 -08:00
epriestley
0c0cbb1c09 Fix an issue where transactions in mail were always rendered as text
Summary:
Fixes T12921. Currently, we call `getTitleForHTMLMail()`, but that calls `getTitleForMail()` which forces us into text rendering mode.

Instead, have `getTitleForHTML/TextMail()` force the rendering mode, then call `getTitleForMail()` with the desired rendering mode.

This causes stories like "epriestely added dependent tasks: x, y." to appear as links in email instead of plain text.

Test Plan: Used `bin/mail show-outbound --id ... --dump-html > out.html` to verify HTML mail.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12921

Differential Revision: https://secure.phabricator.com/D19968
2019-01-16 13:21:05 -08:00
epriestley
c3cafffed7 Update the "SES" and "sendmail" mailers for the new API; remove "encoding"
Summary: Ref T13222. Ref T920. This is the last of the upstream adapter updates.

Test Plan:
  - Sent mail with SES.
  - Sent mail with "sendmail". I don't have sendmail actually configured to an upstream MTA so I'm not 100% sure this worked, but the `sendmail` binary didn't complain and almost all of the code is shared with SES, so I'm reasonably confident this actually works.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T920

Differential Revision: https://secure.phabricator.com/D19965
2019-01-16 13:18:55 -08:00
epriestley
43a6f34e7f Update the SMTP (PHPMailer) adapter for the new mail API; remove "encoding" and "mailer"
Summary:
Ref T920. Ref T12404.

  - Update to the new "$message" API.
  - Remove "encoding". I believe "base64" is always the best value for this since we stopped seeing issues once we changed the default.
  - Remove "mailer". This is a legacy option that makes little sense given how configuration now works.
  - Rename to "SMTP". This doesn't affect users anymore since this mailer has been configured as `smtp` for about a year.
  - This does NOT add a timeout since the SMTP code is inside PHPMailer (see T12404).

Test Plan: Sent messages with many mail features via GMail SMTP and SendGrid SMTP.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12404, T920

Differential Revision: https://secure.phabricator.com/D19961
2019-01-16 13:10:57 -08:00
epriestley
966a93334c Don't require "CAN_EDIT" to watch/unwatch a project
Summary:
See T1024. When "CAN_EDIT" became default in T13186, this was missed as an exception.

Watching shouldn't require "CAN_EDIT", so exempt it.

Test Plan:
  - Before change: tried to watch a project I could not edit, got a policy error.
  - After change: watched/unwatched a project I could not edit.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19977
2019-01-16 13:09:59 -08:00
epriestley
64e3296fe6 Upgrade Sendgrid to the modern mailer API; removes "api-user" option
Summary:
Ref T920. Ref T5969.

  - Update to the new "$message" API.
  - Update to Sendgrid v3.
  - Add a timeout.
  - This removes the "api-user" option, which Sendgrid no longer seems to use.

Test Plan: Sent Sendgrid messages with `bin/mail send-test ...` using subject/headers/attachments/html/to/cc.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: jbrownEP

Maniphest Tasks: T5969, T920

Differential Revision: https://secure.phabricator.com/D19960
2019-01-16 13:09:29 -08:00
epriestley
d7da3560ec Update Mailgun adapter for the new mail adapter API
Summary: Ref T920. Ref T5969. Update the Mailgun adapter for the API changes and add a timeout.

Test Plan: Configured Mailgun as a mailer, sent mail with subject/to/cc/headers/html/attachments using `bin/mail send-test`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T5969, T920

Differential Revision: https://secure.phabricator.com/D19959
2019-01-16 13:02:04 -08:00
epriestley
bc97a7d755 Update Mail test adapter for the newer adapter API and make all tests pass
Summary: Depends on D19956. Ref T920. Move the TestAdapter to the new API and adjust a couple of tests for the changes.

Test Plan: All tests now pass.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19957
2019-01-16 13:01:25 -08:00
epriestley
a8657e6ab6 Update Postmark adapter for multiple mail media
Summary:
Depends on D19955. Ref T920. Ref T5969. Update Postmark to accept new Message objects. Also:

  - Update the inbound whitelist.
  - Add a little support for `media` configuration.
  - Add a service call timeout (see T5969).
  - Drop the needless word "Implementation" from the Adapter class tree. I could call these "Mailers" instead of "Adapters", but then we get "PhabricatorMailMailer" which feels questionable.

Test Plan: Used `bin/mail send-test` to send mail via Postmark with various options (mulitple recipients, text vs html, attachments).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T5969, T920

Differential Revision: https://secure.phabricator.com/D19956
2019-01-16 13:00:34 -08:00
epriestley
b5797ce60a Refactor mail to produce an intermediate "bag of strings" object in preparation for SMS
Summary:
Depends on D19954. Ref T920. This is a step toward a world where "Mailers" are generic and may send messages over a broader array of channels (email, SMS, postal mail).

There are a few major parts here:

  - Instead of calling `$mailer->setSubject()`, `$mailer->setFrom()`, etc., build in intermediate `$message` object first, then pass that to the mailer.
    - This breaks every mailer! This change on its own does not fix them. I plan to fix them in a series of "update mailer X", "update mailer Y" followups.
    - This generally makes the API easier to change in the far future, and in the near future supports mailers accepting different types of `$message` objects with the same API.
  - Pull the "build an email" stuff out into a `PhabricatorMailEmailEngine`. `MetaMTAMail` is already a huge object without also doing this translation step. This is just a separation/simplification change, but also tries to fight against `MetaMTAMail` getting 5K lines of email/sms/whatsapp/postal-mail code.
  - Try to rewrite the "build an email" stuff to be a bit more straightforward while making it generate objects. Prior to this change, it had this weird flow:

```lang=php
foreach ($properties as $key => $prop) {
  switch ($key) {
    case 'xyz':
      // ...
  }
}
```

This is just inherently somewhat hard to puzzle out, and it means that processing order depends on internal property order, which is quite surprising.

Test Plan: This breaks everything on its own; adapters must be updated to use the new API. See followups.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19955
2019-01-16 12:58:29 -08:00
epriestley
a37b28ef79 Prevent inbound processing of the "void/placeholder" address and other reserved addresses
Summary:
Depends on D19952. Ref T13222. Never process mail targets if they match:

  - The "default" address which we send mail "From".
  - The "void" address which we use as a placholder "To" when we only have "CC" addresses.
  - Any address from a list of reserved/administrative names.

The first two prevent loops. The third one prevents abuse.

There's a reasonably well-annotated list of reservations and reasons here:

https://webmasters.stackexchange.com/questions/104811/is-there-any-list-of-email-addresses-reserved-because-of-security-concerns-for-a

Stuff like `support@` seems fine; stuff like `ssladmin@` might let you get SSL certs issued for a domain you don't control.

Also, forbid users from creating application emails with these reserved addresses.

Finally, build the default and void addresses somewhat more cleverly.

Test Plan: Added unit tests, tried to configured reserved addresses, hit the default/void cases manually with `bin/mail receive-test`.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: olexiy.myronenko

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19953
2019-01-16 12:28:53 -08:00
epriestley
e3aa043a02 Allow multiple mail receivers to react to an individual email
Summary:
Fixes T7477. Fixes T13066. Currently, inbound mail is processed by the first receiver that matches any "To:" address. "Cc" addresses are ignored.

**To, CC, and Multiple Receivers**

Some users would like to be able to "Cc" addresses like `bugs@` instead of having to "To" the address, which makes perfect sense. That's the driving use case behind T7477.

Since users can To/Cc multiple "create object" or "update object" addresses, I also wanted to make the behavior more general. For example, if you email `bugs@` and also `paste@`, your mail might reasonably make both a Task and a Paste. Is this useful? I'm not sure. But it seems like it's pretty clearly the best match for user intent, and the least-surprising behavior we can have. There's also no good rule for picking which address "wins" when two or more match -- we ended up with "address order", which is pretty arbitrary since "To" and "Cc" are not really ordered fields.

One part of this change is removing `phabricator.allow-email-users`. In practice, this option only controlled whether users were allowed to send mail to "Application Email" addresses with a configured default author, and it's unlikely that we'll expand it since I think the future of external/grey users is Nuance, not richer interaction with Maniphest/Differential/etc. Since this option only made "Default Author" work and "Default Author" is optional, we can simplify behavior by making the rule work like this:

  - If an address specifies a default author, it allows public email.
  - If an address does not, it doesn't.

That's basically how it worked already, except that you could intentionally "break" the behavior by not configuring `phabricator.allow-email-users`. This is a backwards compatility change with possible security implications (it might allow email in that was previously blocked by configuration) that I'll call out in the changelog, but I suspect that no installs are really impacted and this new behavior is generally more intuitive.

A somewhat related change here is that each receiver is allowed to react to each individual email address, instead of firing once. This allows you to configure `bugs-a@` and `bugs-b@` and CC them both and get two tasks. Useful? Maybe not, but seems like the best execution of intent.

**Sender vs Author**

Adjacently, T13066 described an improvement to error handling behavior here: we did not distinguish between "sender" (the user matching the email "From" address) and "actor" (the user we're actually acting as in the application). These are different when you're some internet rando and send to `bugs@`, which has a default author. Then the "sender" is `null` and the "author" is `@bugs-robot` or whatever (some user account you've configured).

This refines "Sender" vs "Author". This is mostly a purity/correctness change, but it means that we won't send random email error messages to `@bugs-robot`.

Since receivers are now allowed to process mail with no "sender" if they have some default "actor" they would rather use instead, it's not an error to send from an invalid address unless nothing processes the mail.

**Other**

This removes the "abundant receivers" error since this is no longer an error.

This always sets "external user" mail recipients to be unverified. As far as I can tell, there's no pathway by which we send them email anyway (before or after this change), although it's possible I'm missing something somewhere.

Test Plan:
I did most of this with `bin/mail receive-test`. I rigged the workflow slightly for some of it since it doesn't support multiple addresses or explicit "CC" and adding either would be a bit tricky.

These could also be tested with `scripts/mail/mail_handler.php`, but I don't currently have the MIME parser extension installed locally after a recent upgrade to Mojave and suspect T13232 makes it tricky to install.

- Ran unit tests, which provide significant coverage of this flow.
- Sent mail to multiple Maniphest application emails, got multiple tasks.
- Sent mail to a Maniphest and a Paste application email, got a task and a paste.
- Sent mail to a task.
  - Saw original email recorded on tasks. This is a behavior particular to tasks.
- Sent mail to a paste.
- Sent mail to a mock.
- Sent mail to a Phame blog post.
- Sent mail to a Legalpad document.
- Sent mail to a Conpherence thread.
- Sent mail to a poll.
- This isn't every type of supported object but it's enough of them that I'm pretty confident I didn't break the whole flow.
- Sent mail to an object I could not view (got an error).
- As a non-user, sent mail to several "create an object..." addresses.
  - Addresses with a default user worked (e.g., created a task).
  - Addresses without a default user did not work.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13066, T7477

Differential Revision: https://secure.phabricator.com/D19952
2019-01-16 12:28:02 -08:00
epriestley
a62f334d95 Add a skeleton for configurable MFA provider types
Summary:
Ref T13222. Ref T13231. See PHI912. I'm planning to turn MFA providers into concrete objects, so you can disable and configure them.

Currently, we only support TOTP, which doesn't require any configuration, but other provider types (like Duo or Yubikey OTP) do require some configuration (server URIs, API keys, etc). TOTP //could// also have some configuration, like "bits of entropy" or "allowed window size" or whatever, if we want.

Add concrete objects for this and standard transaction / policy / query support. These objects don't do anything interesting yet and don't actually interact with MFA, this is just skeleton code for now.

Test Plan:
{F6090444}

{F6090445}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13231, T13222

Differential Revision: https://secure.phabricator.com/D19935
2019-01-16 12:27:23 -08:00
Austin McKinley
b98d46ce7d Resurrect setup check for cluster.mailers
Summary:
D19940 removed this file entirely, which has led to at least one user who was unsure how to proceed now that `cluster.mailers` is required for outbound mail: https://discourse.phabricator-community.org/t/invalid-argument-supplied-for-foreach-phabricatormetamtamail-php/2287

This isn't //always// a setup issue for installs that don't care about sending mail, but this at least this gives a sporting chance to users who don't follow the changelogs.

Also, I'm not sure if there's a way to use `pht()` to generate links; right now the phurl is just in plain text.

Test Plan: Removed `cluster.mailers` config; observed expected setup issue.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19964
2019-01-16 12:14:36 -08:00
epriestley
3b94b3e812 Correct a zero-based month tooltip on burnup charts
Summary: See PHI1017. This is a trivial fix even though these burnups are headed toward a grisly fate.

Test Plan: Moused over some January datapoints, saw "1" instead of "0".

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19967
2019-01-15 18:09:18 -08:00
epriestley
0b8f24dfd3 Fix bad "SMTP" and "cluster.mailers" default value
Summary: See note in D19964.

Test Plan: O_o

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19966
2019-01-14 13:10:33 -08:00
epriestley
73e3057c52 Rename "MetaMTA" mail attachments and add more mail message objects
Summary:
Depends on D19953. Ref T9141. We have a "MetaMTAAttachment" object, rename it to "MailAttachment".

Also add a "Header" object and an "EmailMessage" object. Currently, mail adapters have a large number of methods like `setSubject()`, `addTo()`, etc, that I would like to remove.

I'd like the API to be more like `sendMessage(PhabricatorMailExternalMessage $message)`. This is likely a significant simplification anyway, since the implementations of all these methods are just copy/pasted boilerplate anyway (lots of `$this->subject = $subject;`) and this will let Adapters support other message media (SMS, APNS, Whatsapp, etc.)

That's a larger change, but move toward a world where we can build a concrete `$message` object for "email" or "sms".

The `PhabricatorMailEmailMessage` object is just a dumb, flat object representation of the information an adapter needs to actually send mail. The existing `PhabricatorMetaMTAMail` is a much more complex object and has a lot of rich data (delivery status, related object PHIDs, etc) and is a storage object.

The new flow will be something like:

  - `PhabricatorMetaMTAMail` (possibly renamed) is the storage object for any outbound message on this channel. It tracks message content, acceptable delivery media (SMS vs email), delivery status, related objects, has a PHID, and has a daemon worker associated with delivering it.
  - It builds a `PhabricatorMailExternalMessage`, which is a simple, flat description of the message it wants to send. The subclass of this object depends on the message medium. For email, this will be an `EmailMessage`. This is just a "bag of strings" sort of object, with appropriate flattened values for the adapter to work with (e.g., Email has email addresses, SMS has phone numbers).
  - It passes the `ExternalMessage` (which is a `MailMessage` or `SMSMessage` or whatever) to the adapter.
  - The adapter reads the nice flat properties off it and turns them into an API request, SMTP call, etc.

This is sort of how things work today anyway. The major change is that I want to hand off a "bag of strings" object instead of calling `setX()`, `setY()`, `setZ()` with each individual value.

Test Plan: Grepped for `MetaMTAAttachment`. This doesn't change any behavior yet.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T9141

Differential Revision: https://secure.phabricator.com/D19954
2019-01-04 15:23:44 -08:00
epriestley
dda3ff89e0 Consolidate some application email receiver code in preparation for API changes
Summary:
Ref T7477. The various "create a new X via email" applications (Paste, Differential, Maniphest, etc) all have a bunch of duplicate code.

The inheritance stack here is generally a little weird. Extend these from a shared parent to reduce the number of callsites I need to change when this API is adjusted for T7477.

Test Plan: Ran unit tests. This will get more thorough testing once more pieces are in place.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T7477

Differential Revision: https://secure.phabricator.com/D19950
2019-01-04 15:21:50 -08:00
epriestley
e48c36697a Make blame UI recover gracefully if Identities haven't been built yet for a commit
Summary:
See PHI1014. We may not have Identities if you race the import pipeline, or in some other cases which are more "bug" / "missing migration"-flavored.

Load the commit data so we can fall back to it if we don't have identities.

Test Plan:
  - Wiped out all my identities with `UPDATE ... SET authorIdentityPHID = NULL WHERE ...`.
  - Before change: blame fataled with `Attempting to access attached data on PhabricatorRepositoryCommit (via getCommitData()), but the data is not actually attached.`.
  - After change: blame falls back gracefully.
  - Restored identities with `bin/repository rebuild-identities`, checked blame again.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19958
2019-01-04 15:15:12 -08:00
epriestley
84f94994ad Remove "metamta.insecure-auth-with-reply-to" Config option
Summary:
Ref T7477. This option was added in D842 in 2011, to support a specific narrow use case at Quora with community moderators using some kind of weird Gmail config.

I don't recall it ever coming up since then, and a survey of a subset of hosted instances (see T11760) reveals that no instances are using this option today. Presumably, even Quora has completed the onboarding discussed in D842, if they still use Phabricator. This option generally does not seem very useful outside of very unusual/narrow cases like the one Quora had.

This would be relatively easy to restore as a local patch if installs //do// need it, but I suspect this has no use cases anywhere.

Test Plan: Grepped for option, blame-delved to figure out why we added it in the first place, surveyed instances for usage.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T7477

Differential Revision: https://secure.phabricator.com/D19949
2019-01-04 13:56:39 -08:00
epriestley
57b3619181 Extract some email address utility code from the receiver stack
Summary:
Ref T7477. We have some address normalization code in the reciever stack that is really shared code. I want to introduce some new callsites elsewhere but don't want to put a lot of static calls to other random objects all over the place.

This technically "solves" T7477 (it changes "to" to "to + cc" for finding receivers) but doesn't yet implement proper behavior (with multiple receivers, for example).

Test Plan: Ran unit tests, which cover this pretty well. Additional changes will vet this more thoroughly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T7477

Differential Revision: https://secure.phabricator.com/D19948
2019-01-04 13:55:49 -08:00
epriestley
e2f0571104 Drop empty inbound mail at the beginning of the receive workflow, not inside object handlers
Summary:
Ref T920. Ref T7477. We currently drop empty mail only once it reaches the `ReplyHandler` layer.

I think no plausible receiver can ever do anything useful with this kind of mail, so we can safely drop it earlier and simplify some of the logic. After T7477, we'd end up throwing multiple exceptions if you sent empty mail to several valid receivers.

(I also want to move away from APIs oriented around raw addresses in more specialized layers, and this is one of the few callsites for raw mail address information.)

This requires updating some unit tests to actually have message bodies, since they failed with this error before hitting the other errors otherwise.

Test Plan: Used `bin/mail receive-test` to send empty mail, got appropriate "err:empty" out of it.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T7477, T920

Differential Revision: https://secure.phabricator.com/D19947
2019-01-04 13:50:21 -08:00
epriestley
1f4cf23455 Remove "phabricator.csrf-key" and upgrade CSRF hashing to SHA256
Summary:
Ref T12509.

  - Remove the "phabricator.csrf-key" configuration option in favor of automatically generating an HMAC key.
  - Upgrade two hasher callsites (one in CSRF itself, one in providing a CSRF secret for logged-out users) to SHA256.
  - Extract the CSRF logic from `PhabricatorUser` to a standalone engine.

I was originally going to do this as two changes (extract logic, then upgrade hashes) but the logic had a couple of very silly pieces to it that made faithful extraction a little silly.

For example, it computed `time_block = (epoch + (offset * cycle_frequency)) / cycle_frequency` instead of `time_block = (epoch / cycle_frequency) + offset`. These are equivalent but the former was kind of silly.

It also computed `substr(hmac(substr(hmac(secret)).salt))` instead of `substr(hmac(secret.salt))`. These have the same overall effect but the former is, again, kind of silly (and a little bit materially worse, in this case).

This will cause a one-time compatibility break: pages loaded before the upgrade won't be able to submit contained forms after the upgrade, unless they're open for long enough for the Javascript to refresh the CSRF token (an hour, I think?). I'll note this in the changelog.

Test Plan:
  - As a logged-in user, submitted forms normally (worked).
  - As a logged-in user, submitted forms with a bad CSRF value (error, as expected).
  - As a logged-out user, hit the success and error cases.
  - Visually inspected tokens for correct format.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D19946
2019-01-04 13:49:47 -08:00
epriestley
93e6dc1c1d Upgrade object reply addresses to SHA256 and remove "phabricator.mail-key"
Summary:
Ref T12509.

  - Upgrade an old SHA1 to SHA256.
  - Replace an old manually configurable HMAC key with an automatically generated one.

This is generally both simpler (less configuration) and more secure (you now get a unique value automatically).

This causes a one-time compatibility break that invalidates old "Reply-To" addresses. I'll note this in the changelog.

If you leaked a bunch of addresses, you could force a change here by mucking around with `phabricator_auth.auth_hmackey`, but AFAIK no one has ever used this value to react to any sort of security issue.

(I'll note the possibility that we might want to provide/document this "manually force HMAC keys to regenerate" stuff some day in T6994.)

Test Plan: Grepped for removed config. I'll vet this pathway more heavily in upcoming changes.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D19945
2019-01-04 13:47:35 -08:00
epriestley
a0668df75a Remove "metamta.domain" and "metamta.placeholder-to-recipient" config options
Summary:
Ref T920. This simplifies mail configuration.

The "metamta.domain" option is only used to generate Thread-ID values, and we just need something that looks like a bit like a domain in order to make GMail happy. Just use the install domain. In most cases, this is almost certainly the configured value anyway. In some cases, this may cause a one-time threading break for existing threads; I'll call this out in the changelog.

The "metamta.placeholder-to-recipient" is used to put some null value in "To:" when a mail only has CCs. This is so that if you write a local client mail rule like "when I'm in CC, burn the message in a fire" it works even if all the "to" addresses have elected not to receive the mail. Instead: just send it to an unlikely address at our own domain.

I'll add some additional handling for the possiblity that we may receive this email ourselves in the next change, but it overlaps with T7477.

Test Plan: Grepped for these configuration values.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19942
2019-01-04 13:45:15 -08:00
epriestley
afa69eedd1 Remove an old digest in Celerity code and some obsolete configuration options
Summary:
Ref T12509. This upgrades a `weakDigest()` callsite to SHA256-HMAC and removes three config options:

  - `celerity.resource-hash`: Now hard-coded, since the use case for ever adjusting it was very weak.
  - `celerity.enable-deflate`: Intended to make cache inspection easier, but we haven't needed to inspect caches in ~forever.
  - `celerity.minify`: Intended to make debugging minification easier, but we haven't needed to debug this in ~forever.

In the latter two cases, the options were purely developer-focused, and it's easy to go add an `&& false` somewhere in the code if we need to disable these features to debug something, but the relevant parts of the code basically work properly and never need debugging. These options were excessively paranoid, based on the static resource enviroment at Facebook being far more perilous.

The first case theoretically had end-user utility for fixing stuck content caches. In modern Phabricator, it's not intuitive that you'd go adjust a Config option to fix this. I don't recall any users ever actually running into problems here, though.

(An earlier version of this change did more magic with `celerity.resource-hash`, but this ended up with a more substantial simplification.)

Test Plan: Grepped for removed configuration options.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D19941
2019-01-04 13:43:38 -08:00
epriestley
7e87d254ab Add a parameterized Future for Twilio API calls
Summary: Ref T920. We currently embed the Twilio PHP API, but can replace it with about 100 lines of code and get a future-oriented interface as a bonus. Add a Future so we can move toward a simpler calling convention for the API.

Test Plan: Used this future to send SMS messages via the Twilio API.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19937
2019-01-04 13:42:52 -08:00
epriestley
3963c86ad5 Pass timeline view data to comment previews, restoring Differential comment previews
Summary:
Ref T13222. In D19918, I refactored how timelines get "view data". Today, this is always additional data about which images/changesets/diffs are visible on the current revision/commit/mock, so we can tell if inline comments should be linked to a `#anchor` on the same page (if the inline is rendered there somewhere) or to a `/D123?id=1&vs=2` full link on a different page (if it isn't), but in general this could be any sort of state information about the current page that affects how the timeline should render.

Previously, comment previews did not use any specialized object code and always rendered a "generic" timeline story. This was actually a bug, but none of the code we have today cares about this (since it's all inline related, and inlines render separately) so it never impacted anything.

After the `TimelineEngine` change, the preview renders with Differential-specific code. This is more correct, but we were not passing the preview the "view data" so it broke.

This preview doesn't actually need the view data and we could just make it bail out if it isn't present, but pass it through for consistency and so this works like we'd expect if we do something fancier with view data in the future.

Test Plan: Viewed comment and inline comment previews in Differential, saw old behavior restored.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19943
2019-01-03 13:06:54 -08:00
Austin McKinley
65e89c239e Add status to PhabricatorProjectQuery->getPagingValueMap()
Summary: Probably fixes https://discourse.phabricator-community.org/t/unhandled-exception-for-certain-fields-in-maniphest-search/2263. Couldn't repro this locally, but this is almost certainly the correct fix.

Test Plan: doitlive

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19951
2019-01-03 11:34:24 -08:00
epriestley
9d5b933ed5 Remove all legacy configuration options for mailers
Summary:
Ref T920. About a year ago (in 2018 Week 6, see D19003) we moved from individually configured mailers to `cluster.mailers`, primarily to support fallback across multiple mail providers.

Since this has been stable for quite a while, drop support for the older options.

Test Plan: Grepped for all removed options.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19940
2019-01-03 04:09:21 -08:00
epriestley
cfcd35d8a3 Remove standalone SMS support in favor of a "Mail, SMS, and other media are mostly the same thing" approach
Summary:
Ref T920. Over time, mail has become much more complex and I think considering "mail", "sms", "postcards", "whatsapp", etc., to be mostly-the-same is now a more promising avenue than building separate stacks for each one.

Throw away all the standalone SMS code, including the Twilio config options. I have a separate diff that adds Twilio as a mail adapter and functions correctly, but it needs some more work to bring upstream.

This permanently destroys the `sms` table, which no real reachable code ever wrote to. I'll call this out in the changelog.

Test Plan:
  - Grepped for `SMS` and `Twilio`.
  - Ran storage upgrade.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19939
2019-01-03 04:05:20 -08:00
Austin McKinley
05a9474138 Raise warning when accidentally submitting Conduit parameters as a JSON-encoded body
Summary: See T12447 for discussion. It is reasonably intuitive to try and pass Conduit parameters via a JSON-encoded HTTP body, but if you do so, you'll get an unhelpful messsage about how method so-and-so does not accept a parameter named "your_entire_json_body". Instead, detect this mistake and advise developers to use form-encoded parameters.

Test Plan:
Got a better error when attempting to make Conduit calls from React code. Tested the following additional invocations of Conduit and got the expected results without an error:

* From the Conduit UI
* With cURL:
```
~ $ curl http://local.phacility.com:8080/api/conpherence.querythread \
>     -d api.token=api-tvv2zb565zrtueab5ddprmpxvrwb \
>     -d ids[0]=1
```
* With `arc call-conduit`:
```
~ $ echo '{
>   "ids": [
>     1
>   ]
> }' | arc call-conduit --conduit-uri http://local.phacility.com:8080/ --conduit-token api-tvv2zb565zrtueab5ddprmpxvrwb conpherence.querythread
```

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19944
2019-01-02 17:31:16 -08:00
epriestley
ea8be11add Fix a qsprintf() issue in mail queries
Summary: Ref T920. Bumped into this while looking at SMS support.

Test Plan: Loaded `/mail/`, no more `qsprintf()` warning.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T920

Differential Revision: https://secure.phabricator.com/D19936
2019-01-02 14:39:31 -08:00
epriestley
106e90dcf0 Remove the "willApplyTransactions()" hook from ApplicationTransactionEditor
Summary:
Depends on D19908. Ref T13222. In D19897, I reordered some transaction code and affected the call order of `willApplyTransactions()`.

It turns out that we use this call for only one thing, and that thing is pretty silly: naming the raw paste data file when editing paste content.

This is only user-visible in the URL when you click "View Raw Paste" and seems exceptionally low-value, so remove the hook and pick a consistent name for the paste datafiles. (We could retain the name behavior in other ways, but it hardly seems worthwhile.)

Test Plan:
  - Created and edited a paste.
  - Grepped for `willApplyTransactions()`.

Note that `EditEngine` (vs `ApplicationTransacitonEditor`) still has a `willApplyTransactions()`, which has one callsite in Phabricator (in Calendar) and a couple in Instances. That's untouched and still works.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19909
2018-12-28 00:19:38 -08:00
epriestley
1729e7b467 Improve UI for "wait" and "answered" MFA challenges
Summary:
Depends on D19906. Ref T13222. This isn't going to win any design awards, but make the "wait" and "answered" elements a little more clear.

Ideally, the icon parts could be animated Google Authenticator-style timers (but I think we'd need to draw them in a `<canvas />` unless there's some clever trick that I don't know) or maybe we could just have the background be like a "water level" that empties out. Not sure I'm going to actually write the JS for either of those, but the UI at least looks a little more intentional.

Test Plan:
{F6070914}

{F6070915}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19908
2018-12-28 00:18:53 -08:00
epriestley
918f4ebcd8 Fix a double-prompt for MFA when recovering a password account
Summary:
Depends on D19905. Ref T13222. In D19843, I refactored this stuff but `$jump_into_hisec` was dropped.

This is a hint to keep the upgraded session in hisec mode, which we need to do a password reset when using a recovery link. Without it, we double prompt you for MFA: first to upgrade to a full session, then to change your password.

Pass this into the engine properly to avoid the double-prompt.

Test Plan:
  - Used `bin/auth recover` to get a partial session with MFA enabled and a password provider.
  - Before: double MFA prompt.
  - After: session stays upgraded when it becomes full, no second prompt.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19906
2018-12-28 00:17:47 -08:00
epriestley
ca39be6091 Make partial sessions expire after 30 minutes, and do not extend them
Summary:
Depends on D19904. Ref T13226. Ref T13222. Currently, partial sessions (where you've provided a primary auth factor like a password, but not yet provided MFA) work like normal sessions: they're good for 30 days and extend indefinitely under regular use.

This behavior is convenient for full sessions, but normal users don't ever spend 30 minutes answering MFA, so there's no real reason to do it for partial sessions. If we add login alerts in the future, limiting partial sessions to a short lifetime will make them more useful, since an attacker can't get one partial session and keep extending it forever while waiting for an opportunity to get past your MFA.

Test Plan:
  - Did a partial login (to the MFA prompt), checked database, saw a ~29 minute partial session.
  - Did a full login, saw session extend to ~30 days.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13226, T13222

Differential Revision: https://secure.phabricator.com/D19905
2018-12-28 00:17:01 -08:00
epriestley
38c48ae7d0 Remove support for the "TYPE_AUTH_WILLLOGIN" event
Summary:
Depends on D19903. Ref T13222. This was a Facebook-specific thing from D6202 that I believe no other install ever used, and I'm generally trying to move away from the old "event" system (the more modern modular/engine patterns generally replace it).

Just drop support for this. Since the constant is being removed, anything that's actually using it should break in an obvious way, and I'll note this in the changelog.

There's no explicit replacement but I don't think this hook is useful for anything except "being Facebook in 2013".

Test Plan:
  - Grepped for `TYPE_AUTH_WILLLOGIN`.
  - Logged in.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19904
2018-12-28 00:16:22 -08:00
epriestley
ff49d1ef77 Allow "bin/auth recover" to generate a link which forces a full login session
Summary:
Depends on D19902. Ref T13222. This is mostly a "while I'm in here..." change since MFA is getting touched so much anyway.

Doing cluster support, I sometimes need to log into user accounts on instances that have MFA. I currently accomplish this by doing `bin/auth recover`, getting a parital session, and then forcing it into a full session in the database. This is inconvenient and somewhat dangerous.

Instead, allow `bin/auth recover` to generate a link that skips the "partial session" stage: adding required MFA, providing MFA, and signing legalpad documents.

Anyone who can run `bin/auth recover` can do this anyway, this just reduces the chance I accidentally bypass MFA on the wrong session when doing support stuff.

Test Plan:
  - Logged in with `bin/auth recover`, was prompted for MFA.
  - Logged in with `bin/auth recover --force-full-session`, was not prompted for MFA.
  - Did a password reset, followed reset link, was prompted for MFA.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19903
2018-12-28 00:15:36 -08:00
epriestley
6a6db0ac8e Allow tokens to be awarded to MFA-required objects
Summary:
Depends on D19901. Ref T13222. See PHI873. Currently, the MFA code and the (older, not-really-transactional) token code don't play nicely.

In particular, if the Editor throws we tend to get half an effect applied.

For now, just make this work. Some day it could become more modern so that the transaction actually applies the write.

Test Plan: Awarded and rescinded tokens from an MFA-required object.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19902
2018-12-28 00:14:48 -08:00
epriestley
efb01bf34f Allow "MFA Required" objects to be edited without MFA if the edit is only creating inverse edges
Summary:
Depends on D19900. Ref T13222. See PHI873. When an object requires MFA, we currently require MFA for every transaction.

This includes some ambiguous cases like "unsubscribe", but also includes "mention", which seems like clearly bad behavior.

Allow an "MFA" object to be the target of mentions, "edit child tasks", etc.

Test Plan:
  - Mentioned an MFA object elsewhere (no MFA prompt).
  - Made an MFA object a subtask of a non-MFA object (no MFA prompt).
  - Tried to edit an MFA object normally (still got an MFA prompt).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19901
2018-12-28 00:12:49 -08:00
epriestley
1c89b3175f Improve UI messaging around "one-shot" vs "session upgrade" MFA
Summary:
Depends on D19899. Ref T13222. When we prompt you for one-shot MFA, we currently give you a lot of misleading text about your session staying in "high security mode".

Differentiate between one-shot and session upgrade MFA, and give the user appropriate cues and explanatory text.

Test Plan:
  - Hit one-shot MFA on an "mfa" task in Maniphest.
  - Hit session upgrade MFA in Settings > Multi-Factor.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19900
2018-12-28 00:11:36 -08:00
epriestley
d3c325c4fc Allow objects to be put in an "MFA required for all interactions" mode, and support "MFA required" statuses in Maniphest
Summary:
Depends on D19898. Ref T13222. See PHI873. Allow objects to opt into an "MFA is required for all edits" mode.

Put tasks in this mode if they're in a status that specifies it is an `mfa` status.

This is still a little rough for now:

  - There's no UI hint that you'll have to MFA. I'll likely add some hinting in a followup.
  - All edits currently require MFA, even subscribe/unsubscribe. We could maybe relax this if it's an issue.

Test Plan:
  - Edited an MFA-required object via comments, edit forms, and most/all of the extensions. These prompted for MFA, then worked correctly.
  - Tried to edit via Conduit, failed with a reasonably comprehensible error.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19899
2018-12-28 00:10:54 -08:00
epriestley
3da9844564 Tighten some MFA/TOTP parameters to improve resistance to brute force attacks
Summary:
Depends on D19897. Ref T13222. See some discussion in D19890.

  - Only rate limit users if they're actually answering a challenge, not if they're just clicking "Wait Patiently".
  - Reduce the number of allowed attempts per hour from 100 back to 10.
  - Reduce the TOTP window from +/- 2 timesteps (allowing ~60 seconds of skew) to +/- 1 timestep (allowing ~30 seconds of skew).
  - Change the window where a TOTP response remains valid to a flat 60 seconds instead of a calculation based on windows and timesteps.

Test Plan:
  - Hit an MFA prompt.
  - Without typing in any codes, mashed "submit" as much as I wanted (>>10 times / hour).
  - Answered prompt correctly.
  - Mashed "Wait Patiently" as much as I wanted (>>10 times / hour).
  - Guessed random numbers, was rate limited after 10 attempts.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19898
2018-12-28 00:10:13 -08:00
epriestley
543f2b6bf1 Allow any transaction group to be signed with a one-shot "Sign With MFA" action
Summary:
Depends on D19896. Ref T13222. See PHI873. Add a core "Sign With MFA" transaction type which prompts you for MFA and marks your transactions as MFA'd.

This is a one-shot gate and does not keep you in MFA.

Test Plan:
  - Used "Sign with MFA", got prompted for MFA, answered MFA, saw transactions apply with MFA metadata and markers.
  - Tried to sign alone, got appropriate errors.
  - Tried to sign no-op changes, got appropriate errors.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19897
2018-12-28 00:09:30 -08:00
epriestley
10db27833b Remove old Phrequent propery rendering code and show "Time Spent" in higher precision
Summary:
See <https://discourse.phabricator-community.org/t/how-to-get-total-time-spent-on-a-task-in-minutes-or-hours/2241>.

Phrequent has two nearly-identical copies of its rendering code: one for old "property event" objects and one for newer "curtain" objects. In the upstream, both trackable object types (tasks and revisions) use curtains, so throw away the old code since it isn't reachable. Third-party trackable objects can update to the curtain UI, but it's unlikely they exist.

Render the remaining curtain UI with more precision, so we show "Time Spent: 2d, 11h, 49m" instead of "Time Spent: 2d".

Test Plan: {F6074404}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19927
2018-12-28 00:07:25 -08:00
epriestley
15df57f1c8 In Webhooks, give errors human-readable labels and show reminder text for "Silent Mode"
Summary:
Depends on D19928. See <https://discourse.phabricator-community.org/t/firehose-webhook-not-working-with-self-hosted-requestbin-instance/2240/>.

Currently, we report "hook" and "silent", which are raw internal codes.

Instead, report human-readable labels so the user gets a better hint about what's going on ("In Silent Mode").

Also, render a "hey, you're in silent mode so none of this will work" reminder banner in this UI.

Test Plan:
{F6074421}

Note:

  - New warning banner.
  - Table has more human-readable text ("In Silent Mode").

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19929
2018-12-28 00:05:46 -08:00
epriestley
3c65601285 Implement "@{config:...}" as a real Remarkup rule
Summary:
See <https://discourse.phabricator-community.org/t/firehose-webhook-not-working-with-self-hosted-requestbin-instance/2240/>. I want to make it easier to link to configuration from system text.

We currently use this weird hack with `{{...}}` that only works within Config itself. Instead, use `@{config:...}`, which is already used by Diviner for `@{class:...}`, etc., so it shouldn't conflict with anything.

Test Plan: Viewed config options, clicked links to other config options.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19928
2018-12-28 00:05:09 -08:00
epriestley
e98ee73602 Fix the last remaining (?) continue inside switch
Summary: See <https://discourse.phabricator-community.org/t/error-on-project-creation-or-edition-with-php7-3/2236/>. This is the only remaining case that the linter rule in D19931 detected in libphutil, arcanist, or Phabricator.

Test Plan: Ran `arc lint --everything ...` in all three repositories, only hit this one.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19932
2018-12-28 00:04:25 -08:00
epriestley
b3faa2874c Restore a Mock key to Pholio Images
Summary: Ref T11351. We only query for images by PHID or by Mock, so the only key we need for now is `<mockPHID>`.

Test Plan: Ran `bin/storage upgrade`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19934
2018-12-28 00:03:48 -08:00
epriestley
e469f8594e Implement Pholio file add/remove transactions without "applyInitialEffects"
Summary:
Depends on D19924. Ref T11351. Like in D19924, apply these transactions by accepting PHIDs instead of objects so we don't need to juggle the `Image` objects down to PHIDs in `applyInitialEffects`.

(Validation is a little light here for now, but only first-party code can reach this, and you can't violate policies or do anything truly bad even if you could pick values to feed in here.)

Test Plan: Created and edited Mocks; added, removed, and reordered images in a Pholio Mock.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19926
2018-12-28 00:02:55 -08:00
epriestley
741fb747db Implement "replace" transactions in Pholio without "applyInitialEffects"
Summary:
Depends on D19923. Ref T11351. Currently, this transaction takes an `Image` as the `newValue` and uses some magic to reduce it into PHIDs by the time we're done.

This creates some problems today where I'd like to get rid of `applyInitialEffects` for MFA code. In the future, it creates a problem becuase there's no way to pass an entire `Image` object over the API.

Instead, create the `Image` first, then just provide the PHID. This is generally simpler, will work well with the API in the future, and stops us from needing any `applyInitialEffects` stuff.

Test Plan: Replaced images in a Pholio mock.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19924
2018-12-28 00:02:21 -08:00
epriestley
b1e7e3a10e Reduce the amount of weird "static" and "cache" behavior in Pholio query classes
Summary: Depends on D19922. Ref T11351. These query classes have some slightly weird behavior, including `public static function loadImages(...)`. Convert all this stuff into more standard query patterns.

Test Plan: Grepped for callsites, browsed around in Pholio.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19923
2018-12-20 15:32:13 -08:00
epriestley
1e2bc7775b Remove the onboard "mailKey" from Pholio Mocks
Summary: Depends on D19921. Ref T11351. Ref T13065. Update Pholio to use the shared mail infrastructure. See D19670 for a previous change in this vein.

Test Plan: Ran upgrade, spot-checked that everything made it into the new table alive.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13065, T11351

Differential Revision: https://secure.phabricator.com/D19922
2018-12-20 15:30:02 -08:00
epriestley
28989ac231 Make the Pholio Mock "getImages" / "getAllImages" API more clear
Summary:
Depends on D19920. Ref T11351. Currently, "images" and "all images" are attached to Mocks separately, and `getImages()` gets you only some images.

Clean this up slightly:

  - One attach method; attach everything.
  - Two getters, one for "images" (returns all images); one for "active images" (returns active images).

Test Plan: Browsed around Pholio without any apparent behavioral changes.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19921
2018-12-20 15:29:04 -08:00
epriestley
11cf8f05b1 Remove "getApplicationTransactionObject()" from ApplicationTransactionInterface
Summary:
Depends on D19919. Ref T11351. This method appeared in D8802 (note that "get...Object" was renamed to "get...Transaction" there, so this method was actually "new" even though a method of the same name had existed before).

The goal at the time was to let Harbormaster post build results to Diffs and have them end up on Revisions, but this eventually got a better implementation (see below) where the Harbormaster-specific code can just specify a "publishable object" where build results should go.

The new `get...Object` semantics ultimately broke some stuff, and the actual implementation in Differential was removed in D10911, so this method hasn't really served a purpose since December 2014. I think that broke the Harbormaster thing by accident and we just lived with it for a bit, then Harbormaster got some more work and D17139 introduced "publishable" objects which was a better approach. This was later refined by D19281.

So: the original problem (sending build results to the right place) has a good solution now, this method hasn't done anything for 4 years, and it was probably a bad idea in the first place since it's pretty weird/surprising/fragile.

Note that `Comment` objects still have an unrelated method with the same name. In that case, the method ties the `Comment` storage object to the related `Transaction` storage object.

Test Plan: Grepped for `getApplicationTransactionObject`, verified that all remaining callsites are related to `Comment` objects.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19920
2018-12-20 15:16:19 -08:00
epriestley
937e88c399 Remove obsolete, no-op implementations of "willRenderTimeline()"
Summary:
Depends on D19918. Ref T11351. In D19918, I removed all calls to this method. Now, remove all implementations.

All of these implementations just `return $timeline`, only the three sites in D19918 did anything interesting.

Test Plan: Used `grep willRenderTimeline` to find callsites, found none.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19919
2018-12-20 15:04:49 -08:00
epriestley
6c43d1d52c Remove "willRenderTimeline()" from ApplicationTransactionInterface
Summary:
Depends on D19914. Ref T11351. Some of the Phoilo rabbit holes go very deep.

`PhabricatorApplicationTransactionInterface` currently requires you to implement `willRenderTimeline()`. Almost every object just implements this as `return $timeline`; only Pholio, Diffusion, and Differential specialize it. In all cases, they are specializing it mostly to render inline comments.

The actual implementations are a bit of a weird mess and the way the data is threaded through the call stack is weird and not very modern.

Try to clean this up:

  - Stop requiring `willRenderTimeline()` to be implemented.
  - Stop requiring `getApplicationTransactionViewObject()` to be implemented (only the three above, plus Legalpad, implement this, and Legalpad's implementation is a no-op). These two methods are inherently pretty coupled for almost any reasonable thing you might want to do with the timeline.
  - Simplify the handling of "renderdata" and call it "View Data". This is additional information about the current view of the transaction timeline that is required to render it correctly. This is only used in Differential, to decide if we can link an inline comment to an anchor on the same page or should link it to another page. We could perhaps do this on the client instead, but having this data doesn't seem inherently bad to me.
  - If objects want to customize timeline rendering, they now implement `PhabricatorTimelineInterface` and provide a `TimelineEngine` which gets a nice formal stack.

This leaves a lot of empty `willRenderTimeline()` implementations hanging around. I'll remove these in the next change, it's just going to be deleting a couple dozen copies of an identical empty method implementation.

Test Plan:
  - Viewed audits, revisions, and mocks with inline comments.
  - Used "Show Older" to page a revision back in history (this is relevant for "View Data").
  - Grepped for symbols: willRenderTimeline, getApplicationTransactionViewObject, Legalpad classes.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19918
2018-12-20 14:55:07 -08:00
epriestley
21f07bf6f7 Make Images in Pholio refer to mocks by PHID instead of ID
Summary:
Ref T11351. In Pholio, we currently use a `mockID`, but a `mockPHID` is generally preferable / more modern / more flexible. In particular, we need PHIDs to load handles and prefer PHIDs when exposing information to the API, and using PHIDs internally makes a bunch of things easier/better/faster and ~nothing harder/worse/slower.

I'll add some inlines about a few things.

Test Plan: Ran migrations, spot-checked database for sanity. Loaded Pholio, saw data unchanged. Created and edited images.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19914
2018-12-20 14:54:25 -08:00
epriestley
961fd7e849 In Legalpad, prompt for MFA at the end of the workflow instead of the beginning
Summary: Depends on D19895. Ref T13222. This is a simple behavioral improvement for the current MFA implementation in Legalpad: don't MFA the user and //then// realize that they forgot to actually check the box.

Test Plan:
  - Submitted form without the box checked, got an error saying "check the box" instead of MFA.
  - Submitted the form with the box checked, got an MFA prompt.
  - Passed the MFA gate, got a signed form.
  - Tried to sign another form, hit MFA timed lockout.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19896
2018-12-20 14:51:25 -08:00
epriestley
b63783c067 Carry MFA responses which have been "answered" but not "completed" through the MFA workflow
Summary:
Depends on D19894. Ref T13222. See PHI873. When you provide a correct response to an MFA challenge, we mark it as "answered".

Currently, we never let you reuse an "answered" token. That's usually fine, but if you have 2+ factors on your account and get one or more (but fewer than all of them) right when you submit the form, you need to answer them all again, possibly after waiting for a lockout period. This is needless.

When you answer a challenge correctly, add a hidden input with a code proving you got it right so you don't need to provide another answer for a little while.

Why not just put your response in a form input, e.g. `<input type="hidden" name="totp-response" value="123456" />`?

  - We may allow the "answered" response to be valid for a different amount of time than the actual answer. For TOTP, we currently allow a response to remain valid for 60 seconds, but the actual code you entered might expire sooner.
  - In some cases, there's no response we can provide (with push + approve MFA, you don't enter a code, you just tap "yes, allow this" on your phone). Conceivably, we may not be able to re-verify a push+approve code if the remote implements one-shot answers.
  - The "responseToken" stuff may end up embedded in normal forms in some cases in the future, and this approach just generally reduces the amount of plaintext MFA we have floating around.

Test Plan:
  - Added 2 MFA tokens to my account.
  - Hit the MFA prompt.
  - Provided one good response and one bad response.
  - Submitted the form.
  - Old behavior: good response gets locked out for ~120 seconds.
  - New behavior: good response is marked "answered", fixing the other response lets me submit the form.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19895
2018-12-20 14:46:45 -08:00
epriestley
ce953ea447 Explicitly mark MFA challenges as "answered" and "completed"
Summary:
Depends on D19893. Ref T13222. See PHI873. A challenge is "answered" if you provide a valid response. A challenge is "completed" if we let you through the MFA check and do whatever actual action the check is protecting.

If you only have one MFA factor, challenges will be "completed" immediately after they are "answered". However, if you have two or more factors, it's possible to "answer" one or more prompts, but fewer than all of the prompts, and end up with "answered" challenges that are not "completed".

In the future, it may also be possible to answer all the challenges but then have an error occur before they are marked "completed" (for example, a unique key collision in the transaction code). For now, nothing interesting happens between "answered" and "completed". This would take the form of the caller explicitly providing flags like "wait to mark the challenges as completed until I do something" and "okay, mark the challenges as completed now".

This change prevents all token reuse, even on the same workflow. Future changes will let the answered challenges "stick" to the client form so you don't have to re-answer challenges for a short period of time if you hit a unique key collision.

Test Plan:
  - Used a token to get through an MFA gate.
  - Tried to go through another gate, was told to wait for a long time for the next challenge window.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19894
2018-12-20 14:45:22 -08:00
epriestley
657f3c3806 When accepting a TOTP response, require it respond explicitly to a specific challenge
Summary:
Depends on D19890. Ref T13222. See PHI873. Currently, we only validate TOTP responses against the current (realtime) timestep. Instead, also validate them against a specific challenge.

This mostly just moves us toward more specifically preventing responses from being reused, and supporting flows which must look more like this (SMS/push).

One rough edge here is that during the T+3 and T+4 windows (you request a prompt, then wait 60-120 seconds to respond) only past responses actually work (the current code on your device won't). For example:

  - At T+0, you request MFA. We issue a T+0 challenge that accepts codes T-2, T-1, T+0, T+1, and T+2. The challenge locks out T+3 and T+4 to prevent the window from overlapping with the next challenge we may issue (see D19890).
  - If you wait 60 seconds until T+3 to actually submit a code, the realtime valid responses are T+1, T+2, T+3, T+4, T+5. The challenge valid responses are T-2, T-1, T+0, T+1, and T+2. Only T+1 and T+2 are in the intersection. Your device is showing T+3 if the clock is right, so if you type in what's shown on your device it won't be accepted.
  - This //may// get refined in future changes, but, in the worst case, it's probably fine if it doesn't. Beyond 120s you'll get a new challenge and a full [-2, ..., +2] window to respond, so this lockout is temporary even if you manage to hit it.
  - If this //doesn't// get refined, I'll change the UI to say "This factor recently issued a challenge which has expired, wait N seconds." to smooth this over a bit.

Test Plan:
  - Went through MFA.
  - Added a new TOTP factor.
  - Hit some error cases on purpose.
  - Tried to use an old code a moment after it expired, got rejected.
  - Waited 60+ seconds, tried to use the current displayed factor, got rejected (this isn't great, but currently expected).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19893
2018-12-20 14:44:35 -08:00
epriestley
0673e79d6d Simplify and correct some challenge TTL lockout code
Summary:
Depends on D19889. Ref T13222. Some of this logic is either not-quite-right or a little more complicated than it needs to be.

Currently, we TTL TOTP challenges after three timesteps -- once the current code could no longer be used. But we actually have to TTL it after five timesteps -- once the most-future acceptable code could no longer be used. Otherwise, you can enter the most-future code now (perhaps the attacker compromises NTP and skews the server clock back by 75 seconds) and then an attacker can re-use it in three timesteps.

Generally, simplify things a bit and trust TTLs more. This also makes the "wait" dialog friendlier since we can give users an exact number of seconds.

The overall behavior here is still a little odd because we don't actually require you to respond to the challenge you were issued (right now, we check that the response is valid whenever you submit it, not that it's a valid response to the challenge we issued), but that will change in a future diff. This is just moving us generally in the right direction, and doesn't yet lock everything down properly.

Test Plan:
  - Added a little snippet to the control caption to list all the valid codes to make this easier:

```
    $key = new PhutilOpaqueEnvelope($config->getFactorSecret());
    $valid = array();
    foreach ($this->getAllowedTimesteps() as $step) {
      $valid[] = self::getTOTPCode($key, $step);
    }

    $control->setCaption(
      pht(
        'Valid Codes: '.implode(', ', $valid)));
```

  - Used the most-future code to sign `L3`.
  - Verified that `L4` did not unlock until the code for `L3` left the activation window.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19890
2018-12-20 14:44:07 -08:00
Austin McKinley
2de632d4fe Update continue/break for php 7.3
Summary:
Fixes https://discourse.phabricator-community.org/t/error-on-project-creation-or-edition-with-php7-3/2236

I didn't actually repro this because I don't have php 7.3 installed. I'm also not sure if the `break; break` was intentional or not, since I'm not sure you could ever reach two consecutive break statements.

Test Plan: Created some projects. Didn't actually try to hit the code that fires if you're making a project both a subproject and a milestone.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19925
2018-12-20 14:12:35 -08:00
Austin McKinley
c72d29f401 Cleanup some clustering rough edges
Summary: Suppress an unhelpful Almanac transaction and document the location of the secret clustering management capability. I thought maybe implementing `shouldHide` and checking for `isCreate` would work, but the binding apparently gets created before an interface is bound to it.

Test Plan: Looked at a fresh binding and didn't see "Unknown Object(??)", ran bin/diviner and saw expected output.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19917
2018-12-20 11:19:19 -08:00
Austin McKinley
979187132d Update accountadmin to use new admin empowerment code
Summary: Fixes https://discourse.phabricator-community.org/t/admin-account-creation-fails-call-to-undefined-method-phabricatorusereditor-makeadminuser/2227. This callsite got skipped when updating the EmpowerController to use the new transactional admin approval code.

Test Plan: Invoked `accountadmin` to promote a user, no longer got an exception.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19915
2018-12-19 12:00:53 -08:00
epriestley
aa3b2ec5dc Give Pholio Images an authorPHID and use ExtendedPolicies to implement policy behavior
Summary:
Depends on D19912. Ref T11351. Images currently use `getMock()->getPolicy()` stuff to define policies. This causes bugs with object policies like "Subscribers", since the policy engine tries to evaluate the subscribers //for the image// when the intent is to evaluate the subscribers for the mock.

Move this to ExtendedPolicies to fix the behavior, and give Images sensible policy behavior when they aren't attached to a mock (specifically: only the user who created the image can see it).

Test Plan: Applied migrations, created and edited mocks and images without anything blowing up. Set mock visibility to "Subscribers", everything worked great.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19913
2018-12-19 10:50:52 -08:00
epriestley
c4c5d8a210 Un-implement MarkupInterface from Mocks and Images in Pholio
Summary: Depends on D19911. Ref T11351. `MarkupInterface` has mostly been replaced with `PHUIRemarkupView`, and isn't really doing anything for us here. Get rid of it to simplify the code.

Test Plan: Viewed various mocks with descriptions and image descriptions, saw remarkup presented properly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19912
2018-12-19 10:47:27 -08:00
epriestley
1d84b5b86b Give Pholio images a more modern initializer method
Summary: Depends on D19910. Ref T11351. Minor changes to make this behave in a more modern way.

Test Plan:
  - Destroyed a mock.
  - Lipsum'd a mock.
  - Poked around, edited/viewed mocks.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19911
2018-12-19 10:47:02 -08:00
epriestley
38083f6f9e Slightly modernize PholioImageQuery
Summary:
Ref T11351. My end goal is to remove `applyInitialEffects()` from Pholio to clear the way for D19897.

Start with some query modernization.

Test Plan: Browsed Pholio, nothing appeared to have changed.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11351

Differential Revision: https://secure.phabricator.com/D19910
2018-12-19 10:46:10 -08:00
Austin McKinley
95ea4f11b9 Fix sorting bug in ProjectDatasource
Summary:
See https://discourse.phabricator-community.org/t/typeahead-returning-only-archived-results/2220. Ref T12538.

If a user has more than 100 disabled projects matching their search term, only disabled projects will be returned in the typeahead search results.

Test Plan: Harcoded hard limit in `PhabricatorTypeaheadModularDatasourceController` to force truncation of search results, observed active project on top of results as expected.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12538

Differential Revision: https://secure.phabricator.com/D19907
2018-12-18 12:17:52 -08:00
epriestley
46052878b1 Bind MFA challenges to particular workflows, like signing a specific Legalpad document
Summary:
Depends on D19888. Ref T13222. When we issue an MFA challenge, prevent the user from responding to it in the context of a different workflow: if you ask for MFA to do something minor (award a token) you can't use the same challenge to do something more serious (launch nukes).

This defuses highly-hypothetical attacks where the attacker:

  - already controls the user's session (since the challenge is already bound to the session); and
  - can observe MFA codes.

One version of this attack is the "spill coffee on the victim when the code is shown on their phone, then grab their phone" attack. This whole vector really strains the bounds of plausibility, but it's easy to lock challenges to a workflow and it's possible that there's some more clever version of the "spill coffee" attack available to more sophisticated social engineers or with future MFA factors which we don't yet support.

The "spill coffee" attack, in detail, is:

  - Go over to the victim's desk.
  - Ask them to do something safe and nonsuspicious that requires MFA (sign `L123 Best Friendship Agreement`).
  - When they unlock their phone, spill coffee all over them.
  - Urge them to go to the bathroom to clean up immediately, leaving their phone and computer in your custody.
  - Type the MFA code shown on the phone into a dangerous MFA prompt (sign `L345 Eternal Declaration of War`).
  - When they return, they may not suspect anything (it would be normal for the MFA token to have expired), or you can spill more coffee on their computer now to destroy it, and blame it on the earlier spill.

Test Plan:
  - Triggered signatures for two different documents.
  - Got prompted in one, got a "wait" in the other.
  - Backed out of the good prompt, returned, still prompted.
  - Answered the good prompt.
  - Waited for the bad prompt to expire.
  - Went through the bad prompt again, got an actual prompt this time.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19889
2018-12-18 12:06:16 -08:00
Austin McKinley
b6999c7ef4 Move admin promotions to modular transactions
Summary: Continue clean up of super-old code. I am pretty proud of "defrocked", but would also consider "dethroned", "ousted", "unseated", "unmade", or "disenfranchised". I feel like there's a word for being kicked out of Hogwarts and having your wizarding powers revoked, but it is not leaping to mind.

Test Plan: Promoted/demoted users to/from admin, attempted to demote myself and observed preserved witty text, checked user timelines, checked feed, checked DB for sanity, including `user_logs`. I didn't test exposing this via Conduit to attempt promoting a user without having admin access.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19891
2018-12-18 10:15:47 -08:00
epriestley
5e94343c7d Add a garbage collector for MFA challenges
Summary:
Depends on D19886. Ref T13222. Clean up MFA challenges after they expire.

(There's maybe some argument to keeping these around for a little while for debugging/forensics, but I suspect it would never actually be valuable and figure we can cross that bridge if we come to it.)

Test Plan:
  - Ran `bin/garbage collect --collector ...` and saw old MFA challenges collected.
  - Triggered a new challenge, GC'd again, saw it survive GC while still active.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19888
2018-12-17 07:00:55 -08:00
epriestley
b8cbfda07c Track MFA "challenges" so we can bind challenges to sessions and support SMS and other push MFA
Summary:
Ref T13222. See PHI873. Ref T9770.

Currently, we support only TOTP MFA. For some MFA (SMS and "push-to-app"-style MFA) we may need to keep track of MFA details (e.g., the code we SMS'd you). There isn't much support for that yet.

We also currently allow free reuse of TOTP responses across sessions and workflows. This hypothetically enables some "spyglass" attacks where you look at someone's phone and type the code in before they do. T9770 discusses this in more detail, but is focused on an attack window starting when the user submits the form. I claim the attack window opens when the TOTP code is shown on their phone, and the window between the code being shown and being submitted is //much// more interesting than the window after it is submitted.

To address both of these cases, start tracking MFA "Challenges". These are basically a record that we asked you to give us MFA credentials.

For TOTP, the challenge binds a particular timestep to a given session, so an attacker can't look at your phone and type the code into their browser before (or after) you do -- they have a different session. For now, this means that codes are reusable in the same session, but that will be refined in the future.

For SMS / push, the "Challenge" would store the code we sent you so we could validate it.

This is mostly a step on the way toward one-shot MFA, ad-hoc MFA in comment action stacks, and figuring out what's going on with Duo.

Test Plan:
  - Passed MFA normally.
  - Passed MFA normally, simultaneously, as two different users.
  - With two different sessions for the same user:
    - Opened MFA in A, opened MFA in B. B got a "wait".
    - Submitted MFA in A.
    - Clicked "Wait" a bunch in B.
    - Submitted MFA in B when prompted.
  - Passed MFA normally, then passed MFA normally again with the same code in the same session. (This change does not prevent code reuse.)

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222, T9770

Differential Revision: https://secure.phabricator.com/D19886
2018-12-17 07:00:21 -08:00
epriestley
c731508d74 Require MFA implementations to return a formal result object when validating factors
Summary:
Ref T13222. See PHI873. Currently, MFA implementations return this weird sort of ad-hoc dictionary from validation, which is later used to render form/control stuff.

I want to make this more formal to handle token reuse / session binding cases, and let MFA factors share more code around challenges. Formalize this into a proper object instead of an ad-hoc bundle of properties.

Test Plan:
  - Answered a TOTP MFA prompt wrong (nothing, bad value).
  - Answered a TOTP MFA prompt properly.
  - Added new TOTP MFA, survived enrollment.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19885
2018-12-17 06:59:46 -08:00
epriestley
54b952df5d Fix weird gap/spacing on user "Manage" page
Summary: I added this recently for debugging test notifications, but goofed up the markup, thought it was just some weird layout issue, and never got back to it.

Test Plan: {F6063455}

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19892
2018-12-14 15:40:26 -08:00
Austin McKinley
d23cc4b862 Move user renames to modular transactions
Summary: Cleaning up more super-old code from `PhabricatorUserEditor`. Also fix user logging in approve transactions. I'm not sure how it worked at all previously.

Test Plan: Created new users, renamed them, checked DB for sanity. Entered invalid names, duplicate names, and empty names, got appropriate error messages.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19887
2018-12-13 16:47:54 -08:00
epriestley
080fb1985f Upgrade an old "weakDigest()" inside TOTP synchronization code
Summary:
Ref T13222. Ref T12509. When you add a new MFA TOTP authenticator, we generate a temporary token to make sure you're actually adding the key we generated and not picking your own key.

That is, if we just put inputs in the form like `key=123, response=456`, users could pick their own keys by changing the value of `key` and then generating the correct `response`. That's probably fine, but maybe attackers could somehow force users to pick known keys in combination with other unknown vulnerabilities that might exist in the future. Instead, we generate a random key and keep track of it to make sure nothing funny is afoot.

As an additional barrier, we do the standard "store the digest, not the real key" sort of thing so you can't force a known value even if you can read the database (although this is mostly pointless since you can just read TOTP secrets directly if you can read the database). But it's pretty standard and doesn't hurt anything.

Update this from SHA1 to SHA256. This will break any TOTP factors which someone was in the middle of adding during a Phabricator upgrade, but that seems reasonable. They'll get a sensible failure mode.

Test Plan: Added a new TOTP factor.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T12509

Differential Revision: https://secure.phabricator.com/D19884
2018-12-13 16:16:13 -08:00
epriestley
1d34238dc9 Upgrade sessions digests to HMAC256, retaining compatibility with old digests
Summary:
Ref T13222. Ref T13225. We store a digest of the session key in the session table (not the session key itself) so that users with access to this table can't easily steal sessions by just setting their cookies to values from the table.

Users with access to the database can //probably// do plenty of other bad stuff (e.g., T13134 mentions digesting Conduit tokens) but there's very little cost to storing digests instead of live tokens.

We currently digest session keys with HMAC-SHA1. This is fine, but HMAC-SHA256 is better. Upgrade:

  - Always write new digests.
  - We still match sessions with either digest.
  - When we read a session with an old digest, upgrade it to a new digest.

In a few months we can throw away the old code. When we do, installs that skip upgrades for a long time may suffer a one-time logout, but I'll note this in the changelog.

We could avoid this by storing `hmac256(hmac1(key))` instead and re-hashing in a migration, but I think the cost of a one-time logout for some tiny subset of users is very low, and worth keeping things simpler in the long run.

Test Plan:
  - Hit a page with an old session, got a session upgrade.
  - Reviewed sessions in Settings.
  - Reviewed user logs.
  - Logged out.
  - Logged in.
  - Terminated other sessions individually.
  - Terminated all other sessions.
  - Spot checked session table for general sanity.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13225, T13222

Differential Revision: https://secure.phabricator.com/D19883
2018-12-13 16:15:38 -08:00
epriestley
c58506aeaa Give sessions real PHIDs and slightly modernize session queries
Summary:
Ref T13222. See PHI873. I'm preparing to introduce a new MFA "Challenge" table which stores state about challenges we've issued (to bind challenges to sessions and prevent most challenge reuse).

This table will reference sessions (since each challenge will be bound to a particular session) but sessions currently don't have PHIDs. Give them PHIDs and slightly modernize some related code.

Test Plan:
  - Ran migrations.
  - Verified table got PHIDs.
  - Used `var_dump()` to dump an organic user session.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19881
2018-12-13 16:14:41 -08:00
epriestley
ecae936d97 Fix another qsprintf() straggler in "Has Open Subtasks"
Summary: See <https://discourse.phabricator-community.org/t/error-message-is-not-being-logged-when-unable-to-connect-to-the-database/2201/>.

Test Plan: Queried for "With Open Subtasks" and "With No Open Subtasks".

Reviewers: amckinley, joshuaspence

Reviewed By: joshuaspence

Subscribers: joshuaspence

Differential Revision: https://secure.phabricator.com/D19880
2018-12-13 05:17:02 -08:00
epriestley
02933acbd5 Remove all application callers to "putInSet()"
Summary: Ref T13218. This is the last public-facing API call for `loadRelatives/loadOneRelative`. This just "primed" objects to make the other calls work and had no direct effects.

Test Plan:
- Ran `bin/fact analyze`.
- Used `bin/storage upgrade -f --apply` to apply `20181031.board.01.queryreset.php`, which uses `LiskMigrationIterator`.
- Browsed user list.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim

Maniphest Tasks: T13218

Differential Revision: https://secure.phabricator.com/D19878
2018-12-12 16:41:12 -08:00
epriestley
793f185d29 Remove application callsites to "LiskDAO->loadOneRelative()"
Summary: Ref T13218. This is like `loadOneWhere(...)` but with more dark magic. Get rid of it.

Test Plan:
- Forced `20130219.commitsummarymig.php` to hit this code and ran it with `bin/storage upgrade --force --apply ...`.
- Ran `20130409.commitdrev.php` with `bin/storage upgrade --force --apply ...`.
- Called `user.search` to indirectly get primary email information.
- Did not test Releeph at all.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13218

Differential Revision: https://secure.phabricator.com/D19876
2018-12-12 16:39:44 -08:00
epriestley
5c99163b7c Remove application callers to "LiskDAO->loadRelatives()"
Summary: Ref T13218. See that task for some discussion. `loadRelatives()` is like `loadAllWhere(...)` except that it does enormous amounts of weird magic which we've moved away from.

Test Plan: Did not test whatsoever since these changes are in Releeph.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13218

Differential Revision: https://secure.phabricator.com/D19874
2018-12-12 16:33:39 -08:00
Austin McKinley
aba9945923 Move user approval to modular transactions
Summary: See https://discourse.phabricator-community.org/t/how-to-approve-user-via-conduit-api/2189. This particular use case doesn't seem very compelling, but moving this logic out of `PhabricatorUserEditor` is a win anyway.

Test Plan: Registered a new user, approved/unapproved them conduit, approved from the UI.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19877
2018-12-12 16:12:23 -08:00
Austin McKinley
5cb462d511 Show more of UTC offset when user's TZ is not an integer number of hours offset
Summary: See https://discourse.phabricator-community.org/t/personal-timezone-setting-mismatch-cleared-and-more-specific-cases/1680. The code has always worked correctly, but the resulting timezone mismatch warning messsage wasn't specific enough when the mismatch is by a non-integer number of hours.

Test Plan: Set timezone locally to Asia/Vladivostok and in Phabricator to Australia/Adelaide (which as of today's date are 30 minutes apart) and observed a more precise error message: F6061330

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19873
2018-12-12 14:02:30 -08:00
epriestley
2814d34036 Fix a stray qsprintf() in the Herald rules engine when recording rule application to objects
Summary: Ref T13217. See PHI1006.

Test Plan: Touched an object with associated Herald rules.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19872
2018-12-12 11:31:36 -08:00
epriestley
66ff6d4a2c Fix an issue with creating tasks directly into workboard columns
Summary:
See <https://discourse.phabricator-community.org/t/tasks-created-via-workboard-column-menu-are-moved-to-wrong-column/2200>. The recent `setIsConduitOnly()` / `setIsFormField()` change (in D19842) disrupted creating tasks directly into a column from the workboard UI.

This field //is// a form field, it just doesn't render a visible control.

Test Plan:
  - Created a task directly into a workboard column. Before: column selection ignored. After: appeared in correct column.
  - Used "move on workboard" comment action.
  - Edited tasks; edited forms for tasks. Didn't observe any collateral damage (weird "Column" fields being present).

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19870
2018-12-12 09:21:39 -08:00
epriestley
d8e2bb9f0f Fix some straggling qsprintf() warnings in repository import
Summary:
Ref T13217. See <https://discourse.phabricator-community.org/t/unsafe-raw-string-warnings-while-importing-git-commits/2191>.

Hunt down and fix two more `qsprintf()` things.

I just converted the "performance optimization" into a normal, safe call since we're dealing with far less SVN stuff nowadays and the actual issue has been lost in the mists of time. If it resurfaces, we can take another look.

Test Plan: Imported some commits, no longer saw these warnings in the daemon logs.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19869
2018-12-12 09:21:12 -08:00
epriestley
0e067213fb Make viewing a user's profile page clear notifications about that user
Summary: Ref T13222. See PHI996. This is a general correctness improvement, but also allows you to clear test notifications by clicking on them (since their default destination is the recipient's profile page).

Test Plan: Clicked a test notification, got taken to my profile page, saw notification marked as read.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19867
2018-12-10 16:26:25 -08:00
epriestley
05900a4cc9 Add a CLI workflow for testing that notifications are being delivered
Summary: Depends on D19865. Ref T13222. See PHI996. Provide a `bin/aphlict notify --user ... --message ...` workflow for sending test notifications from the CLI.

Test Plan: {F6058287}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19866
2018-12-10 16:05:53 -08:00
epriestley
e43f9124f8 Remove obsolete "NotifyTest" feed story
Summary: Depends on D19864. Ref T13222. See PHI996. This is no longer used by anything, so get rid of it.

Test Plan: Grepped; viewed a feed with these stories in it to make sure nothing crashed/exploded.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19865
2018-12-10 16:03:42 -08:00
epriestley
773b4eaa9e Separate "feed" and "notifications" better, allow stories to appear in notifications only
Summary:
Depends on D19861. Ref T13222. See PHI996. Fixes T10743. Currently, notifications only work if a story also has a feed rendering.

Separate "visible in feed" and "visible in notifications", and make notifications query only notifications and vice versa.

Then, set the test notification stories to be visible in notifications only, not feed.

This could be refined a bit (there's no way to have the two views render different values today, for example) but since the only actual use case we have right now is test notifications I don't want to go //too// crazy future-proofing it. I could imagine doing some more of this kind of stuff in Conpherence eventually, though, perhaps.

Test Plan: Sent myself test notifications, saw them appear on my profile timeline and in the JS popup, and in my notifications menu, but not in feed.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T10743

Differential Revision: https://secure.phabricator.com/D19864
2018-12-10 16:02:43 -08:00
epriestley
ba83380565 Update the "Notification Test" workflow to use more modern mechanisms
Summary:
Depends on D19860. Ref T13222. Ref T10743. See PHI996.

Long ago, there were different types of feed stories. Over time, there was less and less need for this, and nowadays basically everything is a "transaction" feed story. Each story renders differently, but they're fundamentally all about transactions.

The Notification test controller still uses a custom type of feed story to send notifications. Move away from this, and apply a transaction against the user instead. This has the same ultimate effect, but involves less weird custom code from ages long forgotten.

This doesn't fix the actual problem with these things showing up in feed. Currently, stories always use the same rendering for feed and notifications, and there need to be some additional changes to fix this. So no behavioral change yet, just slightly more reasonable code.

Test Plan: Clicked the button and got some test notifications, with Aphlict running.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T10743

Differential Revision: https://secure.phabricator.com/D19861
2018-12-10 16:02:11 -08:00
epriestley
508df60a62 When users mark their own inline comments as "Done", suppress the timeline/mail stories
Summary:
Depends on D19858. Ref T13222. See PHI995. In D19635 and related revisions, inline behavior changed to allow you to pre-mark your own inlines as done (as a reviewer) and to pre-mark your inlines for you (as an author).

These actions generate low-value stories in the timeline, like "alice marked 3 comments done." when an author adds some notes to their own revision. These aren't helpful and can be a little misleading.

Instead, just don't count it when someone marks their own inlines as "done". If we throw away all the marks after throwing away the self-marks, hide the whole story.

This happens in three cases:

  # You comment on your own revision, and don't uncheck the "Done" checkbox.
  # You comment on someone else's revision, and check the "Done" checkbox before submitting.
  # You leave a not-"Done" inline on your own revision, then "Done" it later.

Cases (1) and (2) seem unambiguously good/clear. Case (3) is a little more questionable, but I think this still isn't very useful for reviewers.

If there's still a clarity issue around case (3), we could change the story text to "alice marked 3 inline comments by other users as done.", but I think this is probably needlessly verbose and that no one will be confused by the behavior as written here.

(Also note that this story is never shown in feed.)

Test Plan: Created and marked a bunch of inlines as "Done" in Differential and Diffusion, as the author and reviewer/auditor. My own marks didn't generate timeline stories; marking others' comments still does.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19859
2018-12-10 15:37:18 -08:00
epriestley
46feccdfcf Share more inline "Done" code between Differential and Diffusion
Summary:
Ref T13222. See PHI995. Before making a change to inline rendering, consolidate this code for generating the "alice added inlines comments." and "alice marked X inlines as done." transactions.

Both Differential and Diffusion have four very similar chunks of code. Merge them into shared methods and reduce code duplication across the methods.

(In the next change, I plan to hide the "done" story when the mark affects your own inline, since users marking their own inlines as "done" is generally not very interesting or useful.)

Test Plan: As author and reviewer/auditor, added inlines in Differential and Diffusion. As author, marked own and others inlines as done and undone. Got sensible transaction rendering and persistence of "Done".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19858
2018-12-10 15:36:52 -08:00
epriestley
68b1dee139 Replace the "Choose Subtype" radio buttons dialog with a simpler "big stuff you click" sort of UI
Summary:
Ref T13222. Fixes T12588. See PHI683. In several cases, we present the user with a choice between multiple major options: Alamnac service types, Drydock blueprint types, Repository VCS types, Herald rule types, etc.

Today, we generally do this with radio buttons and a "Submit" button. This isn't terrible, but often it means users have to click twice (once on the radio; once on submit) when a single click would be sufficient. The radio click target can also be small.

In other cases, we have a container with a link and we'd like to link the entire container: notifications, the `/drydock/` console, etc. We'd like to just link the entire container, but this causes some problems:

  - It's not legal to link block eleements like `<a><div> ... </div></a>` and some browsers actually get upset about it.
  - We can `<a><span> ... </span></a>` instead, then turn the `<span>` into a block element with CSS -- and this sometimes works, but also has some drawbacks:
    - It's not great to do that for screenreaders, since the readable text in the link isn't necessarily very meaningful.
    - We can't have any other links inside the element (e.g., details or documentation).
  - We can `<form><button> ... </button></form>` instead, but this has its own set of problems:
    - You can't right-click to interact with a button in the same way you can with a link.
    - Also not great for screenreaders.

Instead, try adding a `linked-container` behavior which just means "when users click this element, pretend they clicked the first link inside it".

This gives us natural HTML (real, legal HTML with actual `<a>` tags) and good screenreader behavior, but allows the effective link target to be visually larger than just the link.

If no issues crop up with this, I'd plan to eventually use this technique in more places (Repositories, Herald, Almanac, Drydock, Notifications menu, etc).

Test Plan:
{F6053035}

  - Left-clicked and command-left-clicked the new JS fanciness, got sensible behaviors.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222, T12588

Differential Revision: https://secure.phabricator.com/D19855
2018-12-10 14:59:18 -08:00
epriestley
a6632f8c18 Allow "maniphest.subtypes" to configure which options are presented by "Create Subtask"
Summary:
Ref T13222. Ref T12588. See PHI683. After D19853, "Create Subtask" may pop a dialog to let you choose between multiple forms.

Allow users to configure which forms are available by using `maniphest.subtypes` to choose available children for each subtype. Users may either specify particular subtypes or specific forms.

Test Plan: Configured "Quest" tasks to have "Objective" children, got appropriate prompt behavior. Used "subtypes" and "forms" to select forms; used "forms" to reorder forms.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T12588

Differential Revision: https://secure.phabricator.com/D19854
2018-12-10 14:58:28 -08:00
epriestley
d1bcdaeda4 Allow the "Create Subtask" workflow to prompt for a subtype selection, and prepare for customizable options
Summary:
Ref T13222. Ref T12588. See PHI683. Currently, "Create Subtask" always uses the first edit form that the user has access to for the same task subtype. (For example, if you "Create Subtask" from a "Bug", you get the first edit form for "Bugs".)

I didn't want to go too crazy with the initial subtype implementation, but it seems like we're generally on firm ground and it's working fairly well: user requests are for more flexibility in using the system as implemented, not changes to the system or confusion/difficulty with any of the tradeoffs. Thus, I'm generally comfortable continuing to build it out in the same direction. To improve flexibility, I want to make the options from "Create Subtask" more flexible/configurable.

I plan to let you specify that a given subtype (say, "Quest") prompts you with creation options for a set of other subtypes (say, "Objective"), or prompts you with a particular set of forms.

If we end up with a single option, we just go into the current flow (directly to the edit form). If we end up with more than one option, we prompt the user to choose between them.

This change is a first step toward this:

  - When building "Create Subtask", query for multiple forms.
  - The default behavior is now "prompt user to choose among create forms of the same subtype". Previously, it was "use the first edit form of the same subtype". This is a behavioral change.
  - The next change will make the selected forms configurable.
  - (I also plan to make the dialog itself less rough.)

Test Plan: {F6051067}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T12588

Differential Revision: https://secure.phabricator.com/D19853
2018-12-10 14:44:26 -08:00
Austin McKinley
da4341cf8b Make it less confusing to create root-level Phriction doc
Summary: Without an existing root document, Phriction shows a nice little "fake" document as the landing page, which has its own nice "Edit this document" button. When showing that page, don't also render the standard "New Document" breadcrumb in the top right. That button always prompts first for a slug name, which is silly when the root document doesn't exist (because the slug name is required to be '').

Test Plan: Loaded Phriction with and without a root document.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19863
2018-12-10 14:10:18 -08:00
Austin McKinley
00a7071e2d Fix handling of Phriction conduit edits
Summary: See https://discourse.phabricator-community.org/t/conduit-method-phriction-edit-requires-title-while-the-docs-say-its-optional/2176. Make code consistent with documentation by not requiring either `content` or `title`.

Test Plan: Hit the method via the UI and no longer got an error on missing `content` or `title` fields.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19862
2018-12-10 13:38:13 -08:00
epriestley
bf6c534b56 Give "Track Only" repository detail proper getters/setters
Summary: Depends on D19856. Ref T13222. See D19829. Make access to "Track Only" slightly cleaner and more consistent..

Test Plan: Set, edited, and removed "Track Only" settings for a repository. Saw sensible persistence and display behaviors.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19857
2018-12-10 10:22:37 -08:00
epriestley
c3206476a3 Give "Autoclose Only" repository detail proper getters/setters
Summary:
Ref T13222. See D19829. We're inconsistent about using `getDetail()/setDetail()` to do some ad-hoc reads. Put this stuff in proper accessor methods.

Also a couple of text fixes from D19850.

Test Plan: Set, edited, and removed autoclose branches from a repository. Got sensible persistence and rendering behavior.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19856
2018-12-10 10:22:06 -08:00
epriestley
1a6a0181a8 Allow "bin/repository thaw --demote" to demote an entire service, not just a single device
Summary: Ref T13222. See PHI992. If you lose an entire cluster, you may want to aggressively demote it out of existence. You currently need to `xargs` your way through this. Allow `--demote <service>`, which demotes all devices in a service.

Test Plan: Demoted with `--demote <device>` and `--demote <service>`. Hit the `--promote service` error.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19850
2018-12-09 16:46:17 -08:00
epriestley
bba4186005 Allow "bin/repository thaw" to accept "--all-repositories" instead of a list of repositories
Summary:
Ref T13222. See PHI992. If you've lost an entire cluster (or have lost a device and are willing to make broad assumptions about the state the device was in) you currently have to `xargs` to thaw everything or do something else creative.

Since this workflow is broadly reasonable, provide an easier way to accomplish the goal.

Test Plan:
  - Ran with `--all-repositories`, a list of repositories, both (error) and neither (error).
  - Saw a helpful new list of affected repositories.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19849
2018-12-09 16:41:57 -08:00
epriestley
1e4bdc39a1 Add an "availaiblity" attachment for user.search
Summary: Ref T13222. See PHI990. The older `user.query` supports availability information, but it isn't currently available in a modern way. Make it available.

Test Plan: {F6048126}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19851
2018-12-09 16:41:12 -08:00
epriestley
1029081b28 Correct two straggling "%Q" + "implode(...)" callsites in Revision updates
Summary:
See <https://discourse.phabricator-community.org/t/error-seems-to-be-related-with-da40f8074-and-php-7-3/2102/12>. When creating or updating revisions, we do some manual query construction to update the affected path table.

Update these queries to modern `qsprintf()`.

Test Plan: Created and updated revisions affecting paths, no more logs in the webserver log.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19846
2018-12-09 16:39:59 -08:00
epriestley
b88a87c43a Address a transaction issue with some audit actions not applying correctly
Summary:
See <https://discourse.phabricator-community.org/t/cannot-accept-commits-in-audit/2166/>.

In D19842, I changed `PhabricatorEditField->shouldGenerateTransactionsFromComment()`.

  - Previously, it bailed on `getIsConduitOnly()`.
  - After the patch, it bails on a missing `getCommentActionLabel()`.

The old code was actually wrong, and it was previously possible to apply possibly-invalid actions in some cases (or, at least, sneak them through this layer: they would only actually apply if not validated properly).

In practice, it let a different bug through: we sometimes loaded commits without loading their audit authority, so testing whether the viewer could "Accept" the commit or not (or take some other actions like "Raise Concern") would always fail and throw an exception: "Trying to access data not attached to this object..."

Fixing the insufficiently-strict transaction generation code exposed the "authority not attached" bug, which caused some actions to fail to generate transactions.

This appeared in the UI as either an unhelpful error ("You can't post an empty comment") or an action with no effect. The unhelpful error was because we show that error if you aren't taking any //other// actions, and we wouldn't generate an "Accept" action because of the interaction of these bugs, so the code thought you were just posting an empty comment.

Test Plan: Without leaving comments, accepted and rejected commits. No more error messages, and actions took effect.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: stephan.senkbeil, hskiba

Differential Revision: https://secure.phabricator.com/D19845
2018-12-09 16:39:21 -08:00
epriestley
f0eefdd0b5 Replace the informal "array" subtype map with a more formal "SubtypeMap" object
Summary: Ref T13222. Ref T12588. See PHI683. To make "Create Subtask..." fancier, we need slightly more logic around subtype maps. Upgrade the plain old array into a proper object so it can have relevant methods, notably "get a list of valid child subtypes for some parent subtype".

Test Plan: Created and edited tasks, changed task subtypes. Grepped for affected symbols (`newEditEngineSubtypeMap`, `newSubtypeMap`).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222, T12588

Differential Revision: https://secure.phabricator.com/D19852
2018-12-09 16:37:35 -08:00
epriestley
5d54f26dac Support reading and querying Almanac service PHIDs via "diffusion.repository.search"
Summary:
Ref T13222. See PHI992. If you lose all hosts in a service cluster, you may need to get a list of affected repositories to figure out which backups to pull.

Support doing this via the API.

Test Plan: Queried by service PHID and saw service PHIDs in the call results.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19848
2018-12-06 07:46:36 -08:00
epriestley
70bf63bc3a Fix a transaction editor "continue;" inside "switch()" for PHP 7.3
Summary: See <https://discourse.phabricator-community.org/t/new-git-commit-processing-fails-on-php-7-3/>. This "continue" should be a "break".

Test Plan:
{F6045490}

  - Tried to assign a task to myself while I was already the owner, got an appropriate error.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19844
2018-12-05 11:25:53 -08:00
epriestley
9bfe558587 Add a "touched paths" limit to repositories, limiting the maximum number of paths any commit may touch
Summary:
Depends on D19831. Ref T13216. See PHI908. Allegedly, a user copied a large repository into itself and then pushed it. Great backup strategy, but it can create headaches for administrators.

Allow a "maximum paths you can touch with one commit" limit to be configured, to make it harder for users to make this push this kind of commit by accident.

If you actually intended to do this, you can work around this by breaking your commit into pieces (or temporarily removing the limit). This isn't a security/policy sort of option, it's just a guard against silly mistakes.

Test Plan: Set limit to 2, tried to push 3 files, got rejected. Raised limit, pushed changes successfully.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19839
2018-11-28 14:37:36 -08:00
epriestley
c86c5749ba Make the repository "Filesize Limit" and "Clone/Fetch Timeout" configurable in the UI
Summary: Depends on D19830. Ref T13216. See PHI908. See PHI750. See PHI885. Allow users to configure a filesize limit, and allow them to adjust the clone/fetch timeout.

Test Plan:
{F6021356}

  - Configured a filesize limit and pushed, hit it. Made the limit larger and pushed, change went through.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19831
2018-11-28 14:34:00 -08:00
epriestley
c6fc05ee09 Pull Git filesize logic into a separate LowLevel query and use more Iterators
Summary:
Depends on D19829. Ref T13216. See PHI908. The current implementation is kind of a lot to live in `CommitHookEngine` and will likely fail if `git diff-tree` produces more than 2GB of output.

Pull it out and make it slightly more robust against enormous commits. It's probably limited by this, now:

```
implode("\n", $every_path)
```

We could replace that with some `PhutilReverseRopeSource` primitive or something but since we don't have one of those and it seems unlikely that we'll hit this case in practice, I left it here for now with just the easy stuff converted to be stream-oriented.

Test Plan:
Used this script to test the query against various commits, got good results:

```
<?php

require_once 'scripts/init/init-script.php';

$viewer = PhabricatorUser::getOmnipotentUser();

$repository = id(new PhabricatorRepositoryQuery())
  ->setViewer($viewer)
  ->withCallsigns(array('P'))
  ->executeOne();

var_dump(
  id(new DiffusionLowLevelFilesizeQuery())
    ->setRepository($repository)
    ->withIdentifier($argv[1])
    ->execute());
```

Used this to find large commits in history and pull filesizes (worked great, although our largest commit only touches a couple thousand paths):

```
for hash in `git log --format=%H`; do echo -n $hash; echo -n ' '; git diff-tree -r --no-commit-id $hash | wc -l | awk '{print $1}'; done | awk '{print $2 " " $1}' | sort -n
```

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19830
2018-11-28 14:32:59 -08:00
epriestley
fd12b37d16 Modularize Repository transactions
Summary: Depends on D19828. Ref T13216. Before adding new transactions to repositories (filesize limit, copy time limit, etc) modularize the existing transactions.

Test Plan:

- Created repository.
- Edited callsign (invalid, valid, duplicate, add, remove).
- Edited short name (invaild, valid, duplicate, add, remove).
- Edited description (add, remove).
- Edited encoding (invalid, valid, remove).
- Allowed/denied dangerous changes.
- Allowed/denied enormous chagnes.
- Activated, deactivated, reactivated.
- Changed tags.
- Changed push policy.
- Changed default branch (add, remove).
- Changed track only: add, remove, invalid function, invalid regex.
- Changed autoclose only: add, remove, invalid function, invalid regex.
- Changed publish/notify.
- Changed autoclose.
- Changed staging area (add, remove, invalid).
- Changed blueprints (add, remove).
- Changed symbols (add, remove).
- Grepped for `PhabricatorRepositoryTransaction::TYPE_`.
- Reviewed transaction history:

{F6021036}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19829
2018-11-28 14:29:18 -08:00
epriestley
c25d2a399d Separate the repository management UI into sections
Summary: Depends on D19826. Ref T13216. We have a fair number of options here; add some groups so the "Build" stuff can go in a little subcategory and such.

Test Plan: {F6020896}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19827
2018-11-28 13:53:30 -08:00
epriestley
c457d23a1d Tailor the "no reviewers on this revision" warnings to handle the case where all reviewers have resigned
Summary:
Ref T13216. See PHI985. We currently use a banner to warn you when a revision has no reviewers or only disabled users, but since the changes to track "Resign" more explicilty we'll no longer warn you if everyone has resigned.

(Previously, they'd no longer be reviewers, so you'd end up with the "no reviewers are assigned" warning if everyone resigned.)

This can still interact slightly oddly with some states (e.g., only a package or project reviewer) but I'd like to wait for T731 to tighten those cases up, and they're more advanced/unusual.

Test Plan:
{F6026832}

{F6026833}

{F6026834}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19834
2018-11-28 13:50:29 -08:00
epriestley
01c7be059d Add support for "harbormaster.target.search"
Summary: Ref T13222. See PHI986. See PHI896. Harbormaster build targets don't currently have a modern "*.search" API, but there's no reason not to provide one (even if some of the use cases are a little bit questionable).

Test Plan: {F6032423}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19841
2018-11-28 13:49:27 -08:00
epriestley
2f11001f6e Allow "Change Subtype" to be selected from the comment action stack
Summary:
Ref T13222. See PHI683. Currently, you can "Change subtype..." via Conduit and the bulk editor, but not via the comment action stack or edit forms.

In PHI683 an install is doing this often enough that they'd like it to become a first-class action. I've generally been cautious about pushing this action to become a first-class action (there are some inevitable rough edges and I don't want to add too much complexity if there isn't a use case for it) but since we have evidence that users would find it useful and nothing has exploded yet, I'm comfortable taking another step forward.

Currently, `EditEngine` has this sort of weird `setIsConduitOnly()` method. This actually means more like "this doesn't show up on forms". Make it better align with that. In particular, a "conduit only" field can already show up in the bulk editor, which is goofy. Change this to `setIsFormField()` and convert/simplify existing callsites.

Test Plan:
There are a lot of ways to reach EditEngine so this probably isn't entirely exhaustive, but I think I got pretty much anything which is likely to break:

- Searched for `setIsConduitOnly()` and `getIsConduitOnly()`, converted all callsites to `setIsFormField()`.
- Searched for `setIsLockable()`, `setIsReorderable()` and `setIsDefaultable()` and aligned these calls to intent where applicable.
- Created an Almanac binding.
- Edited an Almanac binding.
- Created an Almanac service.
- Edited an Almanac service.
- Edited a binding property.
- Deleted a binding property.
- Created and edited a badge.
- Awarded and revoked a badge.
- Created and edited an event.
- Made an event recurring.
- Created and edited a Conpherence thread.
- Edited and updated the diff for a revision.
- Created and edited a repository.
- Created and disabled repository URIs.
- Created and edited a blueprint.
- Created and edited tasks.
- Created a paste, edited/archived a paste.
- Created/edited/archived a package.
- Created/edited a project.
- Made comments.
- Moved tasks on workboards via comment action stack.
- Changed task subtype via comment action stack.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19842
2018-11-28 13:40:40 -08:00
epriestley
1d0b99e1f8 Allow applications to require a High Security token without doing a session upgrade
Summary:
Ref T13222. See PHI873. Currently, when applications prompt users to enter MFA, their session upgrades as a side effect.

In some cases (like managing your email addresses) it makes sense to upgrade your session for a little while since it's common to make multiple edits in sequence (add a new address, make it primary, remove an old address). We generally want MFA to stay out of the way and not feel annoying.

In other cases, we don't expect multiple high-security actions in a row. Notably, PHI873 looks at more "one-shot" use cases where a prompt is answering a specific workflow. We already have at least one of these in the upstream: answering an MFA prompt when signing a Legalpad document.

Introduce a "token" workflow (in contrast to the existing "session") workflow that just does a one-shot prompt without upgrading your session statefully. Then, make Legalpad use this new workflow.

Note that this workflow has a significant problem: if the form submission is invalid for some other reason, we re-prompt you on resubmit. In Legalpad, this workflow looks like:

  - Forget to check the "I agree" checkbox.
  - Submit the form.
  - Get prompted for MFA.
  - Answer MFA prompt.
  - Get dumped back to the form with an error.
  - When you fix the error and submit again, you have to do another MFA check.

This isn't a fatal flaw in Legalpad, but would become a problem with wider adoption. I'll work on fixing this (so the MFA token sticks to the form) in the next set of changes.

Roughly, this is headed toward "MFA sticks to the form/workflow" instead of "MFA sticks to the user/session".

Test Plan:
  - Signed a legalpad document with MFA enabled.
  - Was prompted for MFA.
  - Session no longer upgraded (no purple "session in high security" badge).
  - Submitted form with error, answered MFA, fixed error, submitted form again.
    - Bad behavior: got re-prompted for MFA. In the future, MFA should stick to the form.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13222

Differential Revision: https://secure.phabricator.com/D19843
2018-11-28 13:39:59 -08:00
epriestley
bb369c7b71 Convert the "Repository Management" UI to a full-width, Phortune-style UI
Summary:
Ref T13216. I want to add some new management options to repositories (e.g., filesize limit, clone timeouts). Before adding new stuff here, update the UI to a full-width, Phortune-style UI.

This partially reverts D18523. About a year ago, several UIs got converted to fixed-width (repository management, config, settings, instance management in SAAS). I didn't think these were good changes and have never really gotten used to them. The rationale wasn't clear to me and these changes just felt like "be more like GitHub". I think usability is significantly worse, e.g. actions are now hidden inside button menus instead of immediately visible.

Phortune also got converted less dramatically to a full-width-with-menu UI, which I like much better. Adjust repository management to use that UI style instead of the fixed-width style.

Test Plan:
{F6020884}

Viewed every panel, including the Subversion panel.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19826
2018-11-27 05:06:07 -08:00
epriestley
88189f723f Make a Feed query construction less clever/sneaky for new qsprintf() semantics
Summary:
Ref T13216. Ref T13217. Currently, we build this query in a weird way so we end up with `(1, 2, 3)` on both 32-bit and 64-bit systems.

I can't reproduce the string-vs-int MySQL key issue on any system I have access to, so just simplify this and format as `('1', '2', '3')` instead.

The issue this is working around is that MySQL would (I think?) sometimes appear to do something goofy and miss the key if you formatted the query with strings. I never really nailed this down and could have either been mistaken about it or it could be fixed in all modern versions of MySQL. Until we have better evidence to the contrary, assume MySQL is smart enough to handle this sensibly now.

Test Plan: Ran daemons with Feed publish workers, no longer received query warnings.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217, T13216

Differential Revision: https://secure.phabricator.com/D19837
2018-11-26 10:47:19 -08:00
epriestley
5343b1f898 Remove defunct "metamta.herald.show-hints" Config option
Summary:
Ref T13216. See PHI985. This config option once controlled adding a Herald transcript link to email. However, this was never implemented in a generic way and was removed from revisions in D8459 and from commits in D10705. No one has noticed or asked for this option for several years, so this is probably a good opportunity to simplify the software and reduce the total amount of configuration.

If we did want to pursue this in the future, I'd generally prefer to make it part of the mail detail page (`/mail/detail/12345/`) anyway.

Test Plan: Grepped for `metamta.herald.show-hints` and `addHeraldSection()`, got no hits for either.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19833
2018-11-26 10:14:25 -08:00
epriestley
9473f60a36 Allow "Abandoned" revisions to be commandeered
Summary:
Ref T13216. See PHI985. You currently can't commandeer an abandoned revision, but this workflow is perfectly fine.

The caution here is just around weird use cases where, e.g., users want to reopen a revision to add a revert to it. These workflows tend to create problems so we try to guide users away from them.

Test Plan: {F6026841}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19835
2018-11-26 10:13:52 -08:00
epriestley
bcc90d8c6b Fix an off-by-one error affecting mail rendering of inlines on the final line of a file
Summary: Depends on D19837. Ref T13216. See PHI985. There's an off-by-one error here between how inline comments store "length" and how context rendering treats "length". We need to add 1 to the length, but currently do it a little too early. Do it slightly later so that inlines on the final line of a file render properly.

Test Plan: Left an inline on the final line of a new file, saw it render properly in HTML mail.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19838
2018-11-26 10:12:09 -08:00
epriestley
97e7ef0f01 When the last rejecting reviewer resigns from a revision, return it to "Needs Review"
Summary:
Ref T13216. Fixes T12920. See PHI911. If you reject a revision and then resign from it, it stays in "Needs Revision".

There's some arguable motivation for this, but it's inconsistent with how "Accept" works (if the last accepting reviewer resigns, we kick you out of "Accepted"). Make it consistent.

Test Plan:
  - As the only reviewer: requested changes to a revision, then resigned.
  - Before: revision stays in "Needs Revision".
  - After: revision moves back to "Needs Review".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216, T12920

Differential Revision: https://secure.phabricator.com/D19840
2018-11-26 10:11:41 -08:00
John Linahan
433a7321ff Add harbormaster.buildable.search API Method
Summary:
This revision adds a Conduit search method for buildables. It exposes:
  * `objectPHID`
  * `containerPHID`
  * `buildableStatus`
  * `isManual`

Test Plan:
Use the API Console to run searches. Example:
```
{
  "data": [
    {
      "id": 2,
      "type": "HMBB",
      "phid": "PHID-HMBB-m4k5lodx6naq22576a7d",
      "fields": {
        "objectPHID": "PHID-DIFF-vzvgqqcyscpd7ta4osy2",
        "containerPHID": "PHID-DREV-vsivs5276c7vtgpmssn2",
        "buildableStatus": {
          "value": "passed"
        },
        "isManual": true,
        "dateCreated": 1542407155,
        "dateModified": 1542407156,
        "policy": {
          "view": "users",
          "edit": "users"
        }
      },
      "attachments": {}
    },
    {
      "id": 1,
      "type": "HMBB",
      "phid": "PHID-HMBB-opxfl4auoz3ey5klplrx",
      "fields": {
        "objectPHID": "PHID-DIFF-vzvgqqcyscpd7ta4osy2",
        "containerPHID": null,
        "buildableStatus": {
          "value": "passed"
        },
        "isManual": false,
        "dateCreated": 1542406968,
        "dateModified": 1542406968,
        "policy": {
          "view": "users",
          "edit": "users"
        }
      },
      "attachments": {}
    }
  ],
  "maps": {},
  "query": {
    "queryKey": null
  },
  "cursor": {
    "limit": 100,
    "after": null,
    "before": null,
    "order": null
  }
}
```

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, O14 ATC Monitoring

Differential Revision: https://secure.phabricator.com/D19818
2018-11-26 14:16:57 +00:00
epriestley
03f249baf3 Remove rendering support for very old Repository transactions
Summary:
Depends on D19827. Ref T13221. Ref T13216. To prepare Repositories for a move to ModularTransactions, throw away some very old transaction rendering code.

This will cause these very old transactions (none of which have been written since at least April 2016) to render "epriestley edited this repository." instead of "epriestley changed the SSH login for this repository from X to Y."

These edits were generally obsoleted by repository URIs, Passphrase credentials, and general modernization.

Test Plan: Grepped for all constants, got no hits.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13221, T13216

Differential Revision: https://secure.phabricator.com/D19828
2018-11-24 07:21:20 -08:00
epriestley
45e93c8f1d Expose "identifiers" as a query constraint for Commit search
Summary: Ref T13216. See PHI984. The CommitSearchEngine (and, by extension, `diffusion.commit.search`) currently do not support identifier search, but this is a reasonable capability to provide.

Test Plan:
Testing that a commit exists on `master`:

{F6020742}

Same commit is not on `stable`:

{F6020743}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19825
2018-11-24 07:20:01 -08:00
epriestley
43cf4edfb1 When waiting for long-running Harbormaster futures to resolve, close idle database connections
Summary:
Ref T13216. See PHI916. Harbormaster builds may be long-running, particularly if they effectively wrap `ssh ... ./run-huge-build.sh`. If we spend more than a few seconds waiting for futures to resolve, close idle database connections.

The general goal here is to reduce the held connection load for installs with a very large number of test runners.

Test Plan: Added debugging code to `phlog()` closures, saw connections closed while running builds.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19824
2018-11-21 07:53:40 -08:00
epriestley
a0d4b6da4b Support (but do not actually enable) a maximum file size limit for Git repositories
Summary:
Depends on D19816. Ref T13216. See PHI908. See PHI750. In a few cases, users have pushed multi-gigabyte files full of various things that probably shouldn't be version controlled. This tends to create various headaches.

Add support for limiting the maximum size of any object. Specifically, we:

  - list all the objects each commit touches;
  - check their size after the commit applies;
  - if it's over the limit, reject the commit.

This change doesn't actually hook the limit up (the limit is always "0", i.e. unlimited), and doesn't have Mercurial or SVN support. The actual parser bit would probably be better in some other `Query/Parser` class eventually, too. But it at least roughly works.

Test Plan:
Changed the hard-coded limit to other values, tried to push stuff, got sensible results:

```
$ echo pew >> magic_missile.txt && git commit -am pew && git push
[master 98d07af] pew
 1 file changed, 1 insertion(+)
# Push received by "local.phacility.net", forwarding to cluster host.
# Acquiring write lock for repository "spellbook"...
# Acquired write lock immediately.
# Acquiring read lock for repository "spellbook" on device "local.phacility.net"...
# Acquired read lock immediately.
# Device "local.phacility.net" is already a cluster leader and does not need to be synchronized.
# Ready to receive on cluster host "local.phacility.net".
Counting objects: 49, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (48/48), done.
Writing objects: 100% (49/49), 3.44 KiB | 1.72 MiB/s, done.
Total 49 (delta 30), reused 0 (delta 0)
remote: +---------------------------------------------------------------+
remote: |      * * * PUSH REJECTED BY EVIL DRAGON BUREAUCRATS * * *     |
remote: +---------------------------------------------------------------+
remote:              \
remote:               \                    ^    /^
remote:                \                  / \  // \
remote:                 \   |\___/|      /   \//  .\
remote:                  \  /V  V  \__  /    //  | \ \           *----*
remote:                    /     /  \/_/    //   |  \  \          \   |
remote:                    @___@`    \/_   //    |   \   \         \/\ \
remote:                   0/0/|       \/_ //     |    \    \         \  \
remote:               0/0/0/0/|        \///      |     \     \       |  |
remote:            0/0/0/0/0/_|_ /   (  //       |      \     _\     |  /
remote:         0/0/0/0/0/0/`/,_ _ _/  ) ; -.    |    _ _\.-~       /   /
remote:                     ,-}        _      *-.|.-~-.           .~    ~
remote:   *     \__/         `/\      /                 ~-. _ .-~      /
remote:    \____(Oo)            *.   }            {                   /
remote:    (    (..)           .----~-.\        \-`                 .~
remote:    //___\\  \ DENIED!  ///.----..<        \             _ -~
remote:   //     \\                ///-._ _ _ _ _ _ _{^ - - - - ~
remote:
remote:
remote: OVERSIZED FILE
remote: This repository ("spellbook") is configured with a maximum individual file size limit, but you are pushing a change ("98d07af863e799509e7c3a639404d216f9fc79c7") which causes the size of a file ("magic_missile.txt") to exceed the limit. The commit makes the file 317 bytes long, but the limit for this repository is 1 bytes.
remote:
# Released cluster write lock.
To ssh://local.phacility.com/source/spellbook.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://epriestley@local.phacility.com/source/spellbook.git'
```

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: joshuaspence

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19817
2018-11-20 08:04:17 -08:00
epriestley
ab14f49ef8 On the Diffusion cluster status page, improve device sort order
Summary:
Ref T13216. See PHI943. When you have a large number of cluster bindings for a repository, the UI sorting can be a bit hard to manage.

One install that regularly cycles repository cluster devices had a couple dozen older disabled bindings, with the enabled bindings intermingled.

Sort the UI:

  - enabled devices come first;
  - in each group, sort by name.

Test Plan: Mixed disabled/enabled bindings, loaded {nav Diffusion > Repository > Storage} page with clustering configured. Before: relatively unhelpful sort order. After: more intuitive sort order.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19813
2018-11-20 08:03:31 -08:00
epriestley
4967cd6ab9 Fix some "%Q" behavior in PhortuneMerchantQuery
Summary: Ref T13217. This older query does some manual joins; update it for more modern joins.

Test Plan: Ran `instances/` unit tests and got a clean result, browsed Phortune merchants.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19820
2018-11-20 07:59:57 -08:00
epriestley
9481b9eff1 Allow "Can Configure Application" permissions to be configured
Summary:
Ref T13216. See PHI980. Currently, each application in {nav Applications > X > Configure} has a "Can Configure Application" permission which is hard-coded to "Administrators".

There's no technical reason for this, there just hasn't been a great use case for unlocking it. I think when I originally wrote it our protections against locking yourself out of things weren't that great (i.e., it was easier to set the policy to something that prevented you from editing it after the new policy took effect). Our protections are better now.

The major goal here is to let installs open up Custom Forms for given applications (mostly Maniphest) to more users, but the other options mostly go hand-in-hand with that.

Also, in developer mode, include stack traces for policy exceptions. This makes debugging weird stuff (like the indirect Config application errors here) easier.

Test Plan:
  - Granted "Can Configure Application" for Maniphest to all users.
  - Edited custom forms as a non-administrator.
  - Configured Maniphest as a non-administrator.
  - Installed/uninstalled Maniphest as a non-administrator.
  - Tried to lock myself out (got an error message).

{F6015721}

Reviewers: amckinley, joshuaspence

Reviewed By: joshuaspence

Subscribers: joshuaspence

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19822
2018-11-19 07:25:41 -08:00
epriestley
cb033673b6 Unify intracluster sync and Drydock working copy construction timeouts as a repository "copy time limit"
Summary:
Depends on D19814. Ref T13216. See PHI885. For various eldritch reasons, `git fetch` can hang. Although we'd probably like to fix this with `git fetch --require-sustained-network-transfer-rate=512KB/5s` or similar, that flag doesn't exist and we don't have a reasonable way to build it.

Short of that, move toward formalizing a repository "copy time limit": the longest amount of time anything may spend trying to make a copy of this repository.

This grows out of the existing intracluster sync limit, which is effectively the same thing. Here, apply it to `git clone` and `git fetch` in Drydock working copy construction, too. A future change may make it configurable.

Test Plan:
  - Set the limit to 0.001.
  - Tried to build and lease working copies, got sensible timeout errors (see D19815).

```
<Activation Failed> Lease activation failed: [CommandException] Command killed by timeout after running for more than 0.001 seconds.
COMMAND
ssh '-o' 'LogLevel=quiet' '-o' 'StrictHostKeyChecking=no' '-o' 'UserKnownHostsFile=/dev/null' '-o' 'BatchMode=yes' -l '********' -p '2222' -i '********' '127.0.0.1' -- '(cd '\''/var/drydock/workingcopy-163/repo/spellbook/'\'' && git clean -d --force && git fetch && git reset --hard)'
```

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19816
2018-11-16 13:08:12 -08:00
epriestley
933462b487 Continue cleaning up queries in the wake of changes to "%Q"
Summary: Depends on D19810. Ref T13217. Ref T13216. I mostly used `grep implode | grep OR` and `grep implode | grep AND` to find these -- not totally exhaustive but should be a big chunk of the callsites that are missing `%LO` / `%LA`.

Test Plan:
These are tricky to test exhaustively, but I made an attempt to hit most of them:

- Browsed Almanac interfaces.
- Created/browsed Calendar events.
- Enabled/disabled/showed the lock log.
- Browsed repositories.
- Loaded Facts UI.
- Poked at Multimeter.
- Used typeahead for users and projects.
- Browsed Phriction.
- Ran various fulltext searches.

Not sure these are reachable:

- All the lint stuff might be dead/unreachable/nonfunctional?

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim

Maniphest Tasks: T13217, T13216

Differential Revision: https://secure.phabricator.com/D19814
2018-11-16 12:49:44 -08:00
epriestley
49483bdb48 Use "%P" to protect session key hashes in SessionEngine queries from DarkConsole
Summary:
Ref T6960. Ref T13217. Ref T13216. Depends on D19811. Use the recently-introduced "%P" conversion ("Password/Secret") to load sessions in SessionEngine.

This secret isn't critical to protect (it's the //hash// of the actual secret and not useful to attackers on its own) but it shows up on every page in DarkConsole and is an obvious case where `%P` is a more appropriate conversion.

Test Plan:
Note "*********" in the middle of the output here, instead of a session key hash:

{F6012805}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217, T13216, T6960

Differential Revision: https://secure.phabricator.com/D19812
2018-11-16 12:36:35 -08:00
epriestley
b2e91d2205 Move the "container updated" message for Buildables that build Diffs outside of the transaction
Summary:
Ref T13216. See PHI970. Ref T13054. See some discussion in T13216.

When a Harbormaster Buildable object is first created for a Diff, it has no `containerPHID` since the revision has not yet been created.

We later (after creating a revision) send the Buildable a message telling it that we've added a container and it should re-link the container object.

Currently, we send this message in `applyExternalEffects()`, which runs inside the Differential transaction. If Harbormaster races quickly enough, it can read the `Diff` object before the transaction commits, and not see the container update.

Add a `didCommitTransaction()` callback after the transactions commit, then move the message code there instead.

Test Plan:
  - See T13216 for substantial evidence that this change is on the right track.
  - Before change: added `sleep(15)`, reproduced the issue reliably.
  - After change: unable to reproduce issue even with `sleep(15)` (the `containerPHID` always populates correctly).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216, T13054

Differential Revision: https://secure.phabricator.com/D19807
2018-11-16 12:34:06 -08:00
epriestley
44c32839a6 When you "Request Review" of a draft revision, change the button text from "Submit Quietly" to "Publish Revision"
Summary:
See PHI975. Ref T13216. Ref T2543. Previously, see D19204 and PHI433.

When you're acting on a draft revision, we change the button text to "Submit Quietly" as a hint that your actions don't generate notifications yet.

However, this isn't accurate when one of your actions is "Request Review", which causes the revision to publish.

Allow actions to override the submit button text, and make the "Request Review" action change the button text to "Publish Revision".

The alternative change I considered was to remove the word "Quietly" in all cases.

I'm not //thrilled// about how complex this change is to adjust one word, but the various pieces are all fairly clean individually. I'm not sure we'll ever be able to use it for anything else, but I do suspect that the word "Quietly" was the change in D19204 with the largest effect by far (see T10000).

Test Plan:
  - Created a draft revision. Saw "Submit Quietly" text.
  - Added a "Request Review" action, saw it change to "Publish Revision".
  - Reloaded page, saw stack saved and "Publish Revision".
  - Removed action, saw "Submit Quietly".
  - Repeated on a non-draft revision, button stayed put as "Submit".
  - Submitted the various actions, saw them have the desired effects.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216, T2543

Differential Revision: https://secure.phabricator.com/D19810
2018-11-15 20:50:21 -08:00
epriestley
533e4e13b3 Add a bin/herald test ... for doing test runs via the CLI
Summary: Ref T13216. See D19666. It's currently tricky to profile Herald test runs since you have to submit a form and repeating them is a bit of a mess. Provide a simple CLI wrapper so we can use `--xprofile`. This is also maybe nice-to-have if we're ever debugging anything here.

Test Plan: Ran `bin/herald test --object ... --type ...` and got a sensible looking transcript in the UI.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19806
2018-11-15 15:48:52 -08:00
epriestley
8a8123c9db Replace the primary "Disabled/Enabled" Herald Rule filter with "Active/Inactive", considering author status
Summary:
Ref T13216. See PHI947. In Herald, Personal rules do not run if their author's account is disabled.

This isn't communicated very clearly in the UI, and the way the SearchEngine/Query are set up isn't great.

Define "active" as "rule will actually run", which specifically means "rule is enabled, and has a valid (non-disabled) author if it needs one".

Change the meaning of the "Active" default filter from "rule is enabled" to "rule is enabled, and has a valid author if it needs one".

Refine the status badge on the view controller to show this "invalid author" state.

Tweak the language for "Disable/Enable" to be more consistent -- we currently call it "disabled" in some cases and "archived" in others.

Test Plan:
  - Disabled a user account and saw their personal rules behave properly with the new filters/options/view controller.
  - Disabled/enabled a rule, saw consistent text.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19805
2018-11-15 15:47:35 -08:00
epriestley
bbd292b9b3 Modernize the Herald rule search engine
Summary: Ref T13216. Update the Herald Rule SearchEngine and Query to use a more modern style.

Test Plan: Ran various rule queries in the UI, got sensible results

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19803
2018-11-15 15:37:00 -08:00
epriestley
e57bfbf421 Pull some debugging code back out of "master"
See D19778. This is a workaround for T13179 that landed by mistake.
2018-11-15 08:19:29 -08:00
epriestley
86fd204148 Fix all query warnings in "arc unit --everything"
Summary:
Ref T13216. Ref T13217. Depends on D19800. This fixes all of the remaining query warnings that pop up when you run "arc unit --everything".

There's likely still quite a bit of stuff lurking around, but hopefully this covers a big set of the most common queries.

Test Plan: Ran `arc unit --everything`. Before change: lots of query warnings. After change: no query warnings.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217, T13216

Differential Revision: https://secure.phabricator.com/D19801
2018-11-15 03:51:25 -08:00
epriestley
2f10d4adeb Continue making application fixes to Phabricator for changes to %Q semantics
Summary: Depends on D19789. Ref T13217. Continue updating things to use the new %Q-flavored conversions instead of smushing a bunch of strings together.

Test Plan: Browsed around, far fewer errors. These changes are largely mechanical in nature.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19790
2018-11-15 03:50:02 -08:00
epriestley
98690ee326 Update many Phabricator queries for new %Q query semantics
Summary: Depends on D19785. Ref T13217. This converts many of the most common clause construction pathways to the new %Q / %LQ / %LO / %LA / %LJ semantics.

Test Plan: Browsed around a bunch, saw fewer warnings and no obvious behavioral errors. The transformations here are generally mechanical (although I did them by hand).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: hach-que

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19789
2018-11-15 03:48:10 -08:00
epriestley
64b52b9952 Make SELECT construction in PolicyAwareQuery safer
Summary: Depends on D19784. Ref T13217. Reduce uses of unsafe `%Q` in SELECT construction.

Test Plan: This reduces the number of safety warnings when loading Phabricator home from ~900 to ~800.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19785
2018-11-14 15:32:09 -08:00
epriestley
e26c4bddab Replace magical "branch" behavior in "diffusion.branchquery" with an explicit "patterns"
Summary:
See PHI958. Ref T13210. Previously, see PHI720.

The use case for the magic in PHI720 involves multiple patterns, and no parameter can be passed to `branch` that will result in multiple patterns being passed to `git`.

Replace the implicit magic with an explicit `patterns` parameter.

This whole thing is a bit shaky but probably isn't hurting anything.

Test Plan:
  - Ran query with no `patterns`.
  - Ran query with invalid `patterns`, got readable error.
  - Ran query with various valid `patterns` (plain branch name, globs with "?" and "*"), got sensible results.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19771
2018-11-14 14:50:53 -08:00
epriestley
da40f80741 Update PhabricatorLiskDAO::chunkSQL() for new %Q semantics
Summary:
Ref T13217. This method is slightly tricky:

  - We can't safely return a string: return an array instead.
  - It no longer makes sense to accept glue. All callers use `', '` as glue anyway, so hard-code that.

Then convert all callsites.

Test Plan: Browsed around, saw fewer "unsafe" errors in error log.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13217

Differential Revision: https://secure.phabricator.com/D19784
2018-11-13 08:59:18 -08:00
epriestley
315d857a8a Add a basic web UI for intracluster sync logs
Summary: Depends on D19798. Ref T13216. This puts at least a basic UI on top of sync logs.

Test Plan:
Viewed logs from the web UI and exported data. Note that these syncs are somewhat simulated since I my local cluster is somewhat-faked (i.e., not actually multiple machines).

{F5995899}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19799
2018-11-10 04:58:36 -08:00
epriestley
1d7c960531 Put push log "hookWait" to data export and add all wait values to UI
Summary:
Depends on D19797. Ref T13216.

  - Put the new `hookWait` in the export and the UI.
  - Put the existing waits in the UI, not just the export.
  - Make order consistent: host, write, read, hook (this is the order the timers start in).

Test Plan: Pushed some stuff, viewed web UI and saw sensible numbers, exported data and got the same values.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19798
2018-11-10 04:47:38 -08:00
epriestley
2a7ac8e388 Make "bin/repository thaw" workflow more clear when devices are disabled
Summary:
Ref T13216. See PHI943. If autoscale lightning strikes all your servers at once and destroys them, the path to recovery can be unclear. You're "supposed" to:

  - demote all the devices;
  - disable the bindings;
  - bind the new servers;
  - put whatever working copies you can scrape up back on disk;
  - promote one of the new servers.

However, the documentation is a bit misleading (it was sort of written with "you lost one or two devices" in mind, not "you lost every device") and demote-before-disable is unnecessary and slightly risky if servers come back online. There's also a missing guardrail before the promote step which lets you accidentally skip the demotion step and end up in a confusing state. Instead:

  - Add a guard rail: when you try to promote a new server, warn if inactive devices still have versions and tell the user to demote them.
  - Allow demotion of inactive devices: the order "disable, demote" is safer and more intuitive than "demote, disable" and there's no reason to require the unintuitive order.
  - Make the "cluster already has leaders" message more clear.
  - Make the documentation more clear.

Test Plan:
  - Bound a repository to two devices.
  - Wrote to A to make it a leader, then disabled it (simulating a lightning strike).
  - Tried to promote B. Got a new, useful error ("demote A first").
  - Demoted A (before: error about demoting inactive devices; now: works fine).
  - Promoted B. This worked.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19793
2018-11-10 04:46:46 -08:00
epriestley
c32fa06266 Use phutil_microseconds_since(...) to simplify some timing arithmetic
Summary: Depends on D19796. Simplify some timing code by using phutil_microseconds_since() instead of duplicate casting and arithmetic.

Test Plan: Grepped for `1000000` to find these. Pulled, pushed, made a conduit call. This isn't exhaustive but it should be hard for these to break in a bad way since they're all just diagnostic.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19797
2018-11-08 16:46:32 -08:00
epriestley
b12e92e6e2 Add timing information for commit hooks to push logs
Summary:
Depends on D19779. Ref T13216. The push logs currently record the "hostWait", which is roughly "locking + subprocess cost". We also record locking separately, so we can figure out "subprocess cost" alone by subtracting the lock costs.

However, the subprocess (normally `git receive-pack`) runs hooks, and we don't have an easy way to figure out how much time was spent doing actual `git` stuff vs spent doing commit hook processing. This would have been useful in diagnosing at least one recent issue.

Track at least a rough hook cost and record it in the push logs.

Test Plan: Pushed to a repository, saw a reasonable hook cost appear in the database table.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19780
2018-11-08 06:00:26 -08:00
epriestley
966db4d38e Add an intracluster synchronization log for cluster repositories
Summary:
Depends on D19778. Ref T13216. See PHI943, PHI889, et al.

We currently have a push log and a pull log, but do not separately log intracluster synchronization events. We've encountered several specific cases where having this kind of log would be helpful:

  - In PHI943, an install was accidentally aborting locks early. Having timing information in the sync log would let us identify this more quickly.
  - In PHI889, an install hit an issue with `MaxStartups` configuration in `sshd`. A log would let us identify when this is an issue.
  - In PHI889, I floated a "push the linux kernel + fetch timeout" theory. A sync log would let us see sync/fetch timeouts to confirm this being a problem in practice.
  - A sync log will help us assess, develop, test, and monitor intracluster routing sync changes (likely those in T13211) in the future.

Some of these events are present in the pull log already, but only if they make it as far as running a `git upload-pack` subprocess (not the case with `MaxStartups` problems) -- and they can't record end-to-end timing.

No UI yet, I'll add that in a future change.

Test Plan:
  - Forced all operations to synchronize by adding `|| true` to the version check.
  - Pulled, got a sync log in the database.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19779
2018-11-07 18:24:20 -08:00
epriestley
e09d29fb1a Clean up the workflow for some post-push logging code
Summary:
Ref T13216. When a repository is clustered, we run this cleanup code (to tell the repository to update, and log some timing information) on both nodes. Currently, we do slightly too much work, which is unnecessary and can be a bit confusing to human readers.

The double update message doesn't hurt anything, but there's no reason to write it twice.

Likewise, the second timing information update query doesn't do anything: there's no PushEvent object with the right identifier, so it just updates nothing. We don't need to run it, and it's confusing that we do.

Instead, only do these writes if we're actually the final node with the repository on it.

Test Plan: Added some logging, saw double writes/updates before the change and no doubles afterwards, with no other behavioral changes.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19778
2018-11-07 17:46:50 -08:00
epriestley
8a4bf38655 Use 160-bit TOTP keys rather than 80-bit TOTP keys
Summary:
See <https://hackerone.com/reports/435648>. We currently use 80-bit TOTP keys. The RFC suggests 128 as a minimum and recommends 160.

The math suggests that doing the hashing for an 80-bit key is hard (slightly beyond the reach of a highly motivated state actor, today) but there's no reason not to use 160 bits instead to put this completely out of reach.

See some additional discussion on the HackerOne report about enormous key sizes, number of required observations, etc.

Test Plan: Added a new 160-bit TOTP factor to Google Authenticator without issue.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19792
2018-11-07 15:44:02 -08:00
epriestley
1f6a4cfffe Prevent users from selecting excessively bad passwords based on their username or email address
Summary:
Ref T13216. We occasionally receive HackerOne reports concerned that you can select your username as a password. I suspect very few users actually do this and that this is mostly a compliance/checklist sort of issue, not a real security issue.

On this install, we have about 41,000 user accounts. Of these, 100 have their username as a password (account or VCS). A substantial subset of these are either explicitly intentional ("demo", "bugmenot") or obvious test accounts ("test" in name, or name is a nonsensical string of gibberish, or looks like "tryphab" or similar) or just a bunch of numbers (?), or clearly a "researcher" doing this on purpose (e.g., name includes "pentest" or "xss" or similar).

So I'm not sure real users are actually very inclined to do this, and we can't really ever stop them from picking awful passwords anyway. But we //can// stop researchers from reporting that this is an issue.

Don't allow users to select passwords which contain words in a blocklist: their username, real name, email addresses, or the install's domain name. These words also aren't allowed to contain the password (that is, neither your password nor your username may be a substring of the other one). We also do a little normalization to try to split apart email addresses, domains, and real names, so I can't have "evan1234" as my password.

Test Plan:
  - Added unit tests and made them pass.
  - Tried to set my password to a bunch of variations of my username / email / domain name / real name / etc, got rejected.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19776
2018-11-06 12:44:07 -08:00
epriestley
c206b066df When {meme ...} embed has no text, just use the raw file data unmodified
Summary:
Ref T13216. See PHI948. When you use the remarkup hint button to embed a meme with no text, you get `{meme src=X}`.

If the source is a GIF, we currently split the source apart into frame-by-frame images, process them, and stitch them back together. The end result is the same image we started with, but this process can be slow/expensive, and may timeout for sufficiently large GIFs.

Instead: when there's no text, just return the original image data.

Test Plan:
  - Used `{meme src=X}` with no text, got an image faster.
  - Used `{meme src=X, above=...}` to add text, got an attempt to add text (which didn't get very far locally since I don't have GD configured).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19777
2018-11-06 09:40:22 -08:00
epriestley
d38e768ed8 Prevent users from voting for invalid Slowvote options
Summary:
Depends on D19773. See <https://hackerone.com/reports/434116>. You can currently vote for invalid options by submitting, e.g., `vote[]=12345`.

By doing this, you can see the responses, which is sort of theoretically a security problem? This is definitely a bug, regardless.

Instead, only allow users to vote for options which are actually part of the poll.

Test Plan:
  - Tried to vote for invalid options by editing the form to `vote[]=12345` (got error).
  - Tried to vote for invalid options by editing the radio buttons on a plurality poll into checkboxes, checking multiple boxes, and submitting (got error).
  - Voted in approval and plurality polls the right way, from the main web UI and from the embed (`{V...}`) UI.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19774
2018-11-06 09:21:18 -08:00
epriestley
5e1d94f336 Remove nonfunctional AJAX embed behavior for Slowvote
Summary:
See <https://hackerone.com/reports/434116>. Slowvote has a piece of Javascript that attempts to let you vote on `{V123}` polls inline.

It does not work: nothing ever triggers it (nothing renders a control with a `slowvote-option` sigil).

At least for now, just remove it. It has a completely separate pathway in the controller and both pathways are buggy, so this makes fixing them easier.

Test Plan: Voted in plurality and approval polls via Slowvote and the embedded widget.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19773
2018-11-06 09:20:07 -08:00
epriestley
d2316e8025 Fix an errant "switch ... continue"
Summary:
See <https://discourse.phabricator-community.org/t/unhandled-exception-on-create-task/2062>.

This construction has the same behavior as "switch ... break" but is unconventional. PHP 7.3 started warning about it because it's likely a mistake.

Test Plan: Created a task, edited a task owner. The new code is functionally identical to the old code.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19772
2018-11-05 10:26:27 -08:00
epriestley
24a061f844 Correct an ambiguous regexp in DiffusionRequest
Summary:
See <https://discourse.phabricator-community.org/t/diffusionrequest-regex-error/2057/>.

The intent of `[\d-,]` is "digits, hyphen, and comma" but `[x-y]` means "character range x-y".

Specify `[\d,-]` instead to disambiguate the hyphen as "literal hyphen", not a character range marker.

Test Plan: I can't reproduce the original error as reported, but browsed around Diffusion for a bit.

Reviewers: amckinley, avivey

Reviewed By: avivey

Differential Revision: https://secure.phabricator.com/D19770
2018-11-01 20:01:39 -07:00
Tim Hirsh
9bea00c159 Add harbormaster.buildplan.search api method
Summary: This revision adds a conduit search method for build plans.  Other api methods (eg: `harbormaster.build.search`) support build plan phid's as a constraint, but they weren't exposed anywhere, so this provides a way to fetch them.

Test Plan:
Used the api console to run some searches.  Output:
```
{
  "data": [
    {
      "id": 1,
      "type": "HMCP",
      "phid": "PHID-HMCP-q2c25wvegzdkxs7gzor6",
      "fields": {
        "name": "my build plan",
        "planStatus": "active",
        "dateCreated": 1538085249,
        "dateModified": 1538085249,
        "policy": {
          "view": "users",
          "edit": "admin"
        }
      },
    {
      "id": 1,
      "type": "HMCP",
      "phid": "PHID-HMCP-q2c25wvegzdkxs7gzor6",
      "fields": {
        "name": "my build plan",
        "status": {
          "value": "active"
        },
        "dateCreated": 1538085249,
        "dateModified": 1538085249,
        "policy": {
          "view": "users",
          "edit": "admin"
        }
      },
      "attachments": {}
    },
    ...
  ],
  "maps": {},
  "query": {
    "queryKey": null
  },
  "cursor": {
    "limit": 100,
    "after": null,
    "before": null,
    "order": null
  }
}
```

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, yelirekim

Differential Revision: https://secure.phabricator.com/D19769
2018-11-02 02:57:38 +00:00
epriestley
5d4970d6b2 Fix a bug where "View as Query" could replace a saved query row by ID, causing workboard 404s
Summary:
Fixes T13208. See that task for details.

The `clone $query` line is safe if `$query` is a builtin query (like "open").

However, if it's a saved query we clone not only the query parameters but the ID, too. Then when we `save()` the query later, we overwrite the original query.

So this would happen in the database. First, you run a query and save it as the workboard default (query key "abc123"):

| 123 | abc123 | {"...xxx..."} |

Then we `clone` it and change the parameters, and `save()` it. But that causes an `UPDATE ... WHERE id = 123` and the table now looks like this:

| 123 | def456 | {"...yyy..."} |

What we want is to create a new query instead, with an `INSERT ...`:

| 123 | abc123 | {"...xxx..."} |
| 124 | def456 | {"...yyy..."} |

Test Plan:
  - Followed reproduction steps from above.
    - With just the new `save()` guard, hit the guard error.
    - With the `newCopy()`, got a new copy of the query and "View as Query" remained functional without overwriting the original query row.
  - Ran migration, saw an affected board get fixed.

Reviewers: amckinley, joshuaspence

Reviewed By: joshuaspence

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13208

Differential Revision: https://secure.phabricator.com/D19768
2018-11-01 05:44:49 -07:00
epriestley
b950f877c5 Allow Drydock Blueprints to control "supplemental allocation" behavior so all hosts in an Almanac pool get used
Summary:
Fixes T12145. Ref T13210. See PHI570. See PHI536.

Currently, when you give Drydock an Almanac host pool with more than one host, it never voluntarily builds a second host resource: there is no way to say "maximum X working copies per host" (only "maximum X global working copies") to make the first host overflow, and the allocator tries to pack resources as tightly as possible.

If you can force it to allocate the 2nd..Nth host, things will work reasonably well from there (it will spread working copies across the hosts randomly), but tricking it is very hard, especially before D19761.

To deal with this, give blueprints a new behavior around "supplemental allocations". The idea here is that a blueprint may decide that it would prefer to allocate a fresh new resource instead of allowing an otherwise valid acquisition to occur.

These supplemental allocations follow all the normal allocation rules (they can't exceed limits or actually replace existing resources), so they can only happen if there's free space in the resource pool. But a blueprint can elect for a supplemental allocation to provide a "grow the pool" hint.

The only useful policies here are probably "true" (immediately use all resources, like Almanac) or "false" (pack resources as efficiently as possible) but some other policies //might// be useful (perhaps "start growing the pool when we're getting a bit full even if we aren't at the limit yet, since our workload is bursty").

Then, give Almanac host resources a "true" policy (always allocate supplemental resources) so they use all hosts once a similar number of concurrent jobs arrive.

One aspect of this approach is that we only do supplemental resources if the normal allocation algorithm already decided that the best resource to acquire was part of the same blueprint. I started with an approach like "look at all the blueprints and see if any of them want to be greedy", but then a not-very-desirable blueprint would end up filling up its whole pool before we skipped the supplemental allocation part and ended up picking a different resource. That felt a bit silly and this feels a little cleaner and more focused.

Test Plan:
  - Without changing the Almanac blueprint policy, allocated hosts. Got A, A, A, A, ... (second host never used).
  - Changed the Almanac policy.
  - Allocated hosts, got A, B, random mix of A and B.
  - Destroyed B. Destroyed all leases on A. Allocated. Got A. This tests the "don't build a supplemental resource if there are no leases on the natural resource".

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210, T12145

Differential Revision: https://secure.phabricator.com/D19762
2018-10-31 18:06:47 -07:00
epriestley
57b4b59819 When a Drydock host based on an Almanac blueprint has its binding disabled, stop handing out leases
Summary:
Ref T13210. Ref T12145. The "Almanac Host" blueprint currently hands out new leases on a given host even if the binding has been disabled.

Although there are some more complicated cases here (e.g., involving cleanup of the existing resource and existing leases), this one seems clear cut: if the binding has been disabled, we should stop handing out new leases on it.

Test Plan:
  - Created a service with two hosts.
  - Requested a lease, got host A.
  - Requested more leases, always got host A (we never build a new host when we don't have to, and we currently never have to).
  - Disabled the binding to host A.
  - Requested a lease.
    - Before patch: got host A.
    - After patch: got host B.
  - Also disabled the other binding to host B, requested a lease, got an indefinite wait for resources (which is expected and reasonable).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210, T12145

Differential Revision: https://secure.phabricator.com/D19761
2018-10-27 07:20:30 -07:00
epriestley
65e953658a Expose Audit actions for "transaction.search" in a basic way
Summary: Ref T13210. See PHI841. This mirrors D19509 for Differential.

Test Plan: Called `transaction.search` on a commit with a bunch of audit activity, got appropriate labels in the results.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19760
2018-10-27 07:19:50 -07:00
epriestley
61ec434208 Remove unicode marks for "Accept/Raise Concern" in Audit
Summary:
Ref T13210. The comment action dropdown for audits has a heavy checkmark next to "Accept" and a heavy "X" next to "Raise Concern".

We previously removed similar marks in Differential in D19405 and that seems to have gone fine. For consistency, remove these too.

Test Plan: Viewed the comment action dropdown, no longer saw checkmark and X-mark.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19759
2018-10-27 07:19:18 -07:00
epriestley
a7c008708d Correct a mangled translation string in "bin/phd log --id X"
Summary:
Ref T13210. See PHI930. This translation is wrong: the parameter is a comma-separated list as a string, but the USEnglish translation provides alternatives. We can't select among alternatives based on a random string (it isn't a plurality value to let us select "chair" vs "chairs", and isn't a gender value to let us select "his profile" vs "her profile") so we get an error.

But the string itself is also misleading, since "bin/phd log --id A --id B --id C" will say "none of these are valid" if //any// of them are invalid.

Instead, just tell the user explicitly about the first problem.

Test Plan:
  - Ran `bin/phd log --id` with good (got logs) and bad IDs (got sensible error).
  - Ran `bin/phd log` with any logs (got logs) and (simluated) without any logs (got error).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19755
2018-10-26 06:13:18 -07:00
Mike Riley
5f3a7cb41b Expose Drydock leases via Conduit
Summary:
See T13212 for some context and discussion on this being revived.
See T11694 for original context.

Add a query constraint for lease owners and implement the conduit search method for Drydock leases.

Ref T11694. Fixes T13212.

Test Plan:
- Called the API method from conduit and browsed lease queries from the UI.
- Used the new "ownerPHIDs" constraint via API console.

{F5963044}

Reviewers: yelirekim, amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam, epriestley

Maniphest Tasks: T11694, T13212

Differential Revision: https://secure.phabricator.com/D16594
2018-10-26 06:12:38 -07:00
epriestley
f6122547d7 When a lease triggers a resource allocation for a resource which must activate, awaken the lease task after the resource activates
Summary:
Depends on D19753. Ref T13210. This is a small optimization that saves us from waiting up to 15 seconds for a yield.

When there are no Working Copy resources and a new lease comes in, we'll allocate one and yield until it activates.

If activating it (SSH'ing and running `git clone`) takes less than 15 seconds, the resource will activate (say, at T+4) but the lease won't update again for a while (say, until T+15). This leaves us with a pointless wait (in this example, we're sitting around for 9 seconds when we could move forward).

To improve this a little bit, let resources wake up the lease update tasks that triggered allocation after they activate. In the best case, that task runs ~15 seconds sooner. In the worst case, the awaken is just a no-op.

With a more-full queue, this has a smaller effect (it's likely something else will run and be able to use the resource in those 9 seconds).

With already-activated resources, this has no effect (when resources are already activated, we can lease immediately).

Test Plan:
  - Cleaned up all working copy resources.
  - Requested a new "A" working copy.
  - Before patch: got a working copy after 17-18 seconds, most of which was spent yielded.
  - After patch: got a working copy after 3-4 seconds.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19754
2018-10-26 06:11:43 -07:00
epriestley
78ab675bd8 After a Drydock lease triggers a resource to be reclaimed, stop it from triggering another reclaim until the first one completes
Summary:
Depends on D19752. Ref T13210. If resources take a long time to reclaim/destroy (normally, more than 15 seconds) a single new lease may update several times during the reclaim/destroy process and end up reclaiming multiple resources.

Instead: after a lease triggers a reclaim, prevent it from triggering another reclaim as long as the resource it is reclaiming hasn't finished its reclaim/destroy cycle. Basically, each lease only gets to destroy one resource at a time.

Test Plan:
  - Added a `sleep(120)` to `destroyResource()` to simulate a long reclaim/destroy cycle.
  - Allocated A, A, A working copies. Leased a B working copy.
  - Before patch: saw "B" lease destroy all three "A" working copies after ~0, ~15, and ~30 seconds, then build a new "B" resource after ~120 seconds (when the first reclaim/destroy finished).
  - After patch: saw "B" lease destroy one "A" working copy after ~0 seconds, then wait patiently until it finished up, then build a new "B" resource.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19753
2018-10-26 06:11:05 -07:00
epriestley
e9309fdd6a When a Drydock lease schedules a resource to be reclaimed, awaken the lease update task when the reclaim completes
Summary:
Depends on D19751. Ref T13210. When Drydock needs to reclaim an existing unused resource in order to build a new resource to satisfy a lease, the lease which triggered the reclaim currently gets thrown back into the pool with a 15-second yield.

If the queue is pretty empty and the reclaim is quick, this can mean that we spend up to 15 extra seconds just waiting for the lease update task to get another shot at building a resource (the resource reclaim may complete in a second or two, but nothing else happens until the yield ends).

Instead, when a lease triggers a reclaim, have the reclaim reawaken the lease task when it completes. In the best case, this saves us 15 seconds of waiting. In other cases (the task already completed some other way, the resource gets claimed before the lease gets to it), it's harmless.

Test Plan:
  - Allocated A, A, A working copies with limit 3. Leased a B working copy.
  - Before patch: allocation took ~32 seconds.
  - After patch: allocation takes ~17 seconds (i.e., about 15 seconds less).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19752
2018-10-26 06:09:52 -07:00
epriestley
1f6869a765 In "bin/drydock lease", take a JSON "--attributes" so we can accept complex values
Summary:
Depends on D19750. See T13210. The `bin/drydock lease` command makes it easier to request ad-hoc leases, but currently takes lease attributes in the form `--attributes x=y,a=b`.

This was okay for all leases at the time, but doesn't really work for modern WorkingCopy resources since they take a `repositories.map` which has a dictionary as a value. You can't specify that with `repositories.map=...`.

Instead, point `--attributes` at a JSON file or use `--attributes -` to read from stdin.

Test Plan: Used `--attributes` with a file and stdin to allocate working copy leases with repositories.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19751
2018-10-26 06:09:20 -07:00
epriestley
6deb09efcd When leasing a Working Copy in Drydock, just "git reset --hard" so empty repositories work
Summary:
Ref T13210. We currently "git reset --hard HEAD" during working copy leasing, mostly by convention/familiarity.

However, this command does not work in an empty repository, because there is no HEAD yet.

The command "git reset --hard" appears to have the same meaning and effect in all cases, except that it also works correctly in an empty repository.

The manual suggests that omitting HEAD should be the same as specifying HEAD, too:

> The <tree-ish>/<commit> defaults to HEAD in all forms.

Test Plan: Successfully leased a working copy for an empty repository using Drydock.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19750
2018-10-26 06:08:47 -07:00
epriestley
e2cf1e4288 Skip copied code detection for changes that are too large for it to be useful
Summary:
Ref T13210. See PHI944. When parsing certain large diffs (the case in PHI944 is an 2.5-million line JSON diff), we spend ~66% of runtime and ~80% of memory doing copy detection (the little yellow bar which shows up to give you a hint that code was moved around within a diff).

This is pretty much pointless and copy hints are almost certainly never useful on large changes. Instead, just bail if the change is larger than some arbitrary "probably too big for copy hints to ever be useful" threshold (here, 65535 lines).

Test Plan:
Roughly, ran this against a 2.5 million line JSON diff:

```
$changes = id(new ArcanistDiffParser())->parseDiff($raw_diff);
$diff = DifferentialDiff::newFromRawChanges($viewer, $changes);
```

Before the changes, it took 20s + 2.5GB RAM to parse. After the changes, it took 7s + 500MB RAM to parse.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19748
2018-10-20 03:36:34 -07:00
epriestley
e0dea4c486 Fix packages(project) to work properly and add it to "MailableFunctionDatasource"
Summary:
Ref T13210. See PHI937. This function datasource isn't quite implemented correctly: it doesn't resolve `package(project)` properly, since the logic only handles users.

This blames back to D14013, where it looks like `packages(..)` was added mostly as a general nice-to-have as part of a larger modernization change.

Test Plan: Ran a `packages(project)` query in Differential, got accurate results (previously: no results).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19747
2018-10-19 13:53:27 -07:00
epriestley
bc6c8c0e93 Explicitly shuffle nodes before selecting one for cluster sync
Summary:
Depends on D19734. Ref T13202. Ref T13109. Ref T10884. See PHI905. See PHI889. We currently rank cluster nodes in three cases:

  # when performing a write, we can go to any node (D19734 should make our ranking good);
  # when performing a read, we can go to any node (currently random, but T10884 discusses ideas to improve our ranking);
  # when performing an internal synchronization before a read or a write, we must go to an up-to-date node.

Currently, case (3) is not-exactly-deterministic but not random, and we won't spread intracluster traffic acrosss the cluster evenly if, say, half of it is up to date and half of it is still synchronizing. For a given write, I believe all nodes will tend to synchronize from whichever node first received the write today.

Instead, shuffle the list and synchronize from any up-to-date node.

(I think we could improve upon this only by knowing which nodes actually have load and selecting the least-loaded -- doable, but not trivial.)

Test Plan: Poked at it locally, will deploy to `secure`. This is hard to measure/test terribly convincingly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202, T13109, T10884

Differential Revision: https://secure.phabricator.com/D19735
2018-10-17 08:11:23 -07:00
epriestley
51073b972e Try to route cluster writes to nodes which won't need to synchronize first
Summary:
Ref T13109. Ref T13202. See PHI905. See PHI889. When we receive a write to a repository cluster, we currently send it to a random writable node.

Instead, we can prefer:

  - the node currently holding the write lock; or
  - any node which is already up to date.

These should simply be better nodes to take writes in all cases. The write lock is global for the repository, so there's no scaling benefit to spreading writes across different nodes, and these particular nodes will be able to accept the write more quickly.

Test Plan:
  - This is observable by using `fprintf(STDERR, "%s\n", ...)` in the logic, then running `git push`. I'd like to pull this routing logic out of `PhabricatorRepository` at some point, probably into a dedicated `ClusterURIQuery` sort of class, but that is a larger change.
  - Added some `fprintf(...)` stuff around which nodes were being selected.
  - Added a `sleep(10)` after grabbing the write lock.
  - In one window, pushed. Then pushed in a second window.
    - Saw the second window select the lock holder as the write target based on it currently holding the lock.
    - Without a concurrent push, saw pushes select up-to-date nodes based on their up-to-date-ness.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: joshuaspence, timhirsh

Maniphest Tasks: T13202, T13109

Differential Revision: https://secure.phabricator.com/D19734
2018-10-17 08:08:25 -07:00
epriestley
0a51bc4f05 Add a space after "View Inline" in mail to prevent double-click on the filename from selecting "Inline"
Summary:
See PHI920. Ref T13210. Since the HTML is just:

```
<a>View Inline</a><span>filename.txt</span>
```

..double-clicking "filename.txt" in email selects "Inlinefilename.txt".

Add a space to stop this. At least in Safari, a space between the tags is not sufficient (perhaps because the parent is a `<div>`?). I couldn't find an authoritative-seeming source on what the rules for this actually are and adding a space here fixes the issue without changing the visual rendering, so just put it here.

Test Plan:
  - Made an inline.
  - Used `bin/mail show-outbound --id ... --dump-html` to dump the HTML.
  - Double-clicked the filename.

{F5929186}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19742
2018-10-11 13:44:20 -07:00
epriestley
8bffc9ea0e In "bin/bulk export", require "--output <path>" by default
Summary:
Depends on D19743. Ref T13210. Since this command can easily dump a bunch of binary data (or just a huge long blob of nonsense) to stdout, default to requiring "--output <file>".

Using `--output -` will print to stdout.

Test Plan: Ran with: no `--output`, `--output file`, `--output -`, `--output - --overwrite`. Got sensible results or errors in all cases.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19744
2018-10-11 13:35:16 -07:00
epriestley
4f54d483d5 Support export of revisions to Excel/CSV/JSON/etc
Summary: Ref T13210. See PHI806. This enables basic export of revisions into flat data formats. This isn't too fancy, but just covers the basics since the driving use case isn't especially concerned about getting all the fields and details.

Test Plan: Exported some revisions into JSON, got sensible output.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19743
2018-10-11 13:34:33 -07:00
epriestley
4f557ff075 When using "bin/bulk export --overwrite", actually overwrite the file
Summary: Depends on D19738. Ref T13210. Currently, when you use "--overwrite", we just //append// the new content. Instead, actually overwrite the file.

Test Plan: Used `--overwrite`, saw an actual clean overwrite instead of an append.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19739
2018-10-11 08:13:43 -07:00
epriestley
4928c34d00 Allow "bin/bulk export" to merge multiple queries and accept more flexible flags
Summary:
Ref T13210. Minor usability improvements to "bin/bulk export":

  - Allow `--class task` to work (previously, only `--class ManiphestTaskSearchEngine` worked).
  - If you run `--query jXIlzQyOYHPU`, don't require `--class`, since the query identifies the class on its own.
  - Allow users to call `--query A --query B --query C` and get a union of all results.

Test Plan:
  - Ran `--class task`, `--query A --query B`, `--query X` (with no `--class`), got good results.
  - Ran various flavors of bad combinations (queries from different engines, invalid engines, query and class differing, ambiguous/invalid `--class` name) and got sensible errors.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13210

Differential Revision: https://secure.phabricator.com/D19738
2018-10-10 09:14:14 -07:00
epriestley
99034efa8b Make Pholio mail render without a ton of over-escaped HTML
Summary:
Ref T13202. See PHI900. Fixes T12814. Pholio currently builds HTML comments in an older way that can dump a bunch of over-escaped HTML into mail bodies.

Update the logic to be more similar to the Differential rendering logic and stop over-escaping things.

The result isn't perfect, but is dramatically less broken.

Test Plan: {F5919860}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202, T12814

Differential Revision: https://secure.phabricator.com/D19733
2018-10-05 13:37:26 -07:00
epriestley
c6c1893dc0 Allow revisions to be filtered by created date
Summary:
Ref T13202. See PHI906. This is a reasonable capability which we support in some other applications already.

(The only real reason not to support this is that it creates some clutter in the UI, but I think we're generally in better shape now than we were in the past, and we could make this UI collapse/fold at some point.)

Test Plan: Ran queries with a minimum date, a maximum date, both, and neither. Saw appropriate results in all cases.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19732
2018-10-05 12:28:20 -07:00
epriestley
4858d43d16 Add 'autocomplete="off"' to MFA TOTP inputs
Summary:
Ref T13202. See <https://discourse.phabricator-community.org/t/2fa-input-box-isnt-hinted-as-a-password-so-browsers-suggest-auto-fills/1959>.

If browsers are autofilling this, I think browser behavior here is bad, but behavior is probably better on the balance if we hint this as `autocomplete="off"` and this is a minor concesssion.

Test Plan:
  - I couldn't immediately get any browser to try to autofill this field (perhaps I've disabled autofill, or just not enabled it aggressively?), but this change didn't break anything.
  - After the change, answered a TOTP prompt normally.
  - After the change, inspected page content and saw `autocomplete="off"` on the `<input />` node.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19722
2018-10-01 13:08:54 -07:00
epriestley
39b85c0be0 Use the changeset parse cache when rendering inline comments in mail
Summary:
Ref T13202. See PHI903 and PHI894. When a bot leaves 100 inline comments on the same file and the revision has a 30-member recipient list, we currently highlight the file 3000 times when building mail.

Instead, engage the parse cache so we highlight it once and reuse the cache 2,999 times.

Test Plan:
  - Added debugging code to stop after mail generation and show cache hits/misses.
    - Left a bunch of inlines in a file.
    - Ran the worker with `bin/worker execute --id ...`.
    - Before change: saw 4 comments x 2 recipients = 8 cache misses
    - After change: saw 8 cache hits (cache already filled from web UI rendering)
  - Removed debugging code, ran worker to completion.
  - Used `bin/mail show-outbound --id ... --dump-html` to inspect resulting mail, saw no functional differences.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19721
2018-10-01 13:08:27 -07:00
Austin McKinley
dbf2302b6c Fix self-cancelling typo
Summary: Ref D18268. This typo cancelled itself out, and I can't find any other callers.

Test Plan: arc unit

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19718
2018-09-28 17:44:33 -07:00
Austin McKinley
8065433ee8 Migrate DiffusionBlameController to use repo identities
Summary:
Now on the blame page, identities get `avatar.png` and there are little tooltips that show a few characters of the committer identity string.

Also add a default icon for repo identities.

Test Plan: Loaded some blame pages for files touched by users with and without repo identities attached.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19587
2018-09-26 14:45:58 -07:00
epriestley
021c612cb2 When we fail to acquire a repository lock, try to provide a hint about why
Summary:
Ref T13202. See PHI889. If the lock log is enabled, we can try to offer more details about lock holders.

When we fail to acquire a lock:

  - check for recent acquisitions and suggest that this is a bottleneck issue;
  - if there are no recent acquisitions, check for the last acquisition and print details about it (what process, how long ago, whether or not we believe it was released).

Test Plan:
  - Enabled the lock log.
  - Changed the lock wait time to 1 second.
  - Added a `sleep(10)` after grabbing the lock.
  - In one window, ran a Conduit call or a `git fetch`.
  - In another window, ran another operation.
  - Got useful/sensible errors for both ssh and web lock holders, for example:

> PhutilProxyException: Failed to acquire read lock after waiting 1 second(s). You may be able to retry later. (This lock was most recently acquired by a process (pid=12609, host=orbital-3.local, sapi=apache2handler, controller=PhabricatorConduitAPIController, method=diffusion.rawdiffquery) 3 second(s) ago. There is no record of this lock being released.)

> PhutilProxyException: Failed to acquire read lock after waiting 1 second(s). You may be able to retry later. (This lock was most recently acquired by a process (pid=65251, host=orbital-3.local, sapi=cli, argv=/Users/epriestley/dev/core/lib/phabricator/bin/ssh-exec --phabricator-ssh-device local.phacility.net --phabricator-ssh-key 2) 2 second(s) ago. There is no record of this lock being released.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19702
2018-09-24 15:20:07 -07:00
epriestley
7db265cd5d Parameterize the repository read and write locks
Summary:
Ref T13202. See PHI889. Update the read and write locks to the modern parameterized verison, which handles hashing/normalization and can store better logs.

This parameterized mode was added in D19173 and has been used successfully for some time, but not all locks have switched over to it yet.

Test Plan:
- Added an `fprintf(STDERR, $full_name)` to the lock code.
- Pulled a repository.
- Saw sensible lock name on stdout before "acquired read lock...".
- Additional changes in this patch series will vet this more completely.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19701
2018-09-24 15:14:28 -07:00
epriestley
3244324cb1 Fix comment box borders in timelines after Phriction commenting
Summary:
Ref T13202. In D19660, I added comments to Phriction and tweaked some CSS.

One of these tweaks was getting rid of an extra border which was rendering under the comment area. However, I took off too much and ended up removing borders from other applications.

I think we don't actually need this `setNoBorder()` stuff after all -- a later change was sufficient to stop the actual border I was trying to get rid of from rendering. So this mostly just reverts part of D19660.

This rendering still isn't perfect, but I'm fine leaving that for another day for now.

Test Plan:
  - Viewed comment areas in Phriction. Saw correct number of borders (1).
  - Viewed comment areas in Maniphest. Saw correct number of borders (1).
  - Grepped for extraneous removed classs, no hits.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19684
2018-09-19 13:56:58 -07:00
epriestley
5ba66e56fd Fix Phriction rendering for initial install and 404 pages
Summary:
Depends on D19682. Ref T13202. We currently fatal when trying to render a timeline if:

  - an install is fresh, so there are no pages yet, and you look at "/w/"; or
  - you're looking at a Phriction page which doesn't exist (yet) like "/w/aadsflknadsflnf/".

Rendering a timeline and comment area doesn't make sense in these cases, so don't render them.

Test Plan: Hit both cases described above, got "new/empty page" prompts instead of fatals.

Reviewers: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19683
2018-09-17 20:02:59 -07:00
epriestley
e5c6a5749a Fix fatal in rendering Phriction "Moved Away" stories
Summary:
Ref T13202. See PHI881. These stories have bad rendering methods, but they didn't previously render into the timeilne (since Phriction documents didn't have a timeline).

Update the rendering to work.

The rendered outcome isn't great (it isn't very clear or explicit about exactly what moved where), but I'll fix that in a followup. This is a net improvement since it doesn't fatal the page, at least.

Test Plan:
  - Moved page "X" to "Y".
  - Viewed the old page "X".
  - Before patch: bad timeline story would fatal rendering.
  - After patch: story renders, at least, just not great.

Reviewers: amckinley

Maniphest Tasks: T13202

Differential Revision: https://secure.phabricator.com/D19682
2018-09-17 20:02:06 -07:00
epriestley
0167f357b7 Provide a convenient way to log arbitrary text in Drydock without needing structured log classes
Summary: Depends on D19673. Ref T13197. See PHI873.

Test Plan:
Added some code like this:

```
$operation->logText('Nice convenient text logging.');
```

...then got:

{F5887712}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19674
2018-09-15 07:59:50 -07:00
epriestley
a7e060f062 Write a trivial log when starting a repository operation
Summary:
Depends on D19672. Ref T13197. See PHI873. This writes a trivial log when we begin acting on a working copy and makes it look reasonable in the UI.

This is mostly just to prove that logging works properly.

Test Plan: {F5887697}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19673
2018-09-15 07:57:11 -07:00
epriestley
92bcf85974 Add Drydock logs to the RepositoryOperation UI
Summary:
Depends on D19671. Ref T13197. See PHI873.

Expose logs in the RepositoryOperation UI. Nothing writes the logs yet, so these interfaces are currently always empty.

Test Plan:
{F5887102}

{F5887103}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19672
2018-09-15 07:56:35 -07:00
epriestley
10f219fb82 Allow Drydock logs to be associated with RepositoryOperation objects
Summary: Ref T13197. See PHI873. I want to give RepositoryOperation objects access to Drydock logging like leases, resources, and blueprints currently have. This just does the schema/query changes, no actual UI or new logging yet.

Test Plan: Ran storage upgrade, poked around the UI looking for anything broken.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19671
2018-09-15 07:55:14 -07:00
epriestley
40d5d5c984 Remove "mailKey" from "PhabricatorRepositoryCommit"
Summary: Ref T13197. Ref T13065. This continues the gradual purge of dedicated "mailKey" columns in favor of shared infrastructure.

Test Plan:
  - Ran migration.
  - Visually inspected database.
  - Grepped for `mailKey`.
  - Added some comments, saw the daemons generate corresponding mail.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13197, T13065

Differential Revision: https://secure.phabricator.com/D19670
2018-09-15 07:54:15 -07:00
epriestley
0abf29765e Make Phriction diff-of-changes view draft-aware and clarify some language
Summary:
Ref T13077. Ref T13197. See PHI840.

  - In the "History > Diff/Compare" view, the button language wasn't draft-aware.
  - Revise language to avoid the word "Revert", since this can be ambiguous.
    - "Edit this page, starting with an older version of the text" is now "Edit Older|Current|Draft Version X".
    - "Mark this older version of the page as the current published version" is now "Publish Older Version".
  - Let the user edit the current published version, too, since this is a reasonable operation if there are drafts.

Test Plan: Navigated the history diff view, saw better button and action text.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197, T13077

Differential Revision: https://secure.phabricator.com/D19668
2018-09-13 14:12:49 -07:00
epriestley
0089ef4b60 Don't show Phriction draft edit events in feed
Summary: Depends on D19663. Ref T13077. When you edit a Phriction draft, don't publish a feed story. (The eventual "Publish" event gets a story.)

Test Plan: Made draft / non-draft / publish edits, only saw feed stories for non-draft and publish edits.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19664
2018-09-12 13:56:10 -07:00
epriestley
b6ba75d991 Add a Phriction hint when a draft exists, and fix some draft editing bugs
Summary:
Depends on D19662. Ref T13077. See PHI840.

  - If you're looking at the published version of a document, but a draft version exists and you can edit it, add a hint/link.
  - Fix an issue where the "draft" transaction would complain when you created a document since the initial content is empty and no "draft" transaction is adding any content.

Test Plan: Created new documents, viewed documents with current published versions and unpublished drafts.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19663
2018-09-12 13:45:33 -07:00
epriestley
550028a882 Allow Phriction document edits to be saved as drafts
Summary:
Depends on D19661. Ref T13077. See PHI840.

When a user edits a page normally, add a "Save as Draft" button. Much of this change is around making that button render and behave properly: it needs to be an `<input type="submit" ...>` so browsers submit it and we can figure out which button the user clicked.

Then there are a few minor rules:

  - If you're editing a page which is already a draft, we only give you "Save as Draft". This makes edits to update/revise a draft more natural.
  - Highlight "Publish" if it's a likely action that you might want to take.

Internally, there are two types of edits. Both types create a new version with the new content. However:

  - A "content" edit sets the version shown on the live page to the newly-created version.
  - A "draft" edit does not update the version shown on the live page.

Test Plan: Edited a published document, edited the draft. Published documents. Reverted documents.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19662
2018-09-12 13:30:40 -07:00
epriestley
152e7713eb Remove obsolete, nonfunctional draft auto-saving in Phriction
Summary:
Depends on D19660. Ref T5811. Ref T13077.

Long ago, if you started editing a Phriction document but didn't save it, we'd save the draft in the background as part of the preview.

D11169 updated the preview to use shared infrastructure and broke this function, since we never save drafts.

Since this doesn't work right now, I want to add another thing called "draft", and the future of this feature should be more integrated with modern drafts and EditEngine (which fixed some bugs related to versioning), just get rid of this code for the moment.

Test Plan: Edited documents. This code doesn't do anything since D11169, so no behavior changed.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13077, T5811

Differential Revision: https://secure.phabricator.com/D19661
2018-09-12 13:22:16 -07:00
epriestley
e19c555913 Support (basic) commenting on Phriction documents
Summary:
Depends on D19659. Fixes T1894. Ref T13077. See PHI840.

  - Add an EditEngine, although it currently supports no fields.
  - Add (basic, top-level-only) commenting (we already had the table in the database).

This will probably create some issues. I'm most concerned about documents accumulating a ton of old, irrelevant comments over time which are hard to keep track of and no longer relevant. But I think this is probably a step forward in almost all cases, and a good thing on the balance.

This also moves us incrementally toward putting all editing on top of EditEngine.

Test Plan: {F5877347}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13077, T1894

Differential Revision: https://secure.phabricator.com/D19660
2018-09-12 13:20:52 -07:00
epriestley
f5e90a363e When a user takes actions while in a high security session, note it on the resulting transactions
Summary:
Ref T13197. See PHI873. Record when a user has MFA'd and add a little icon to the transaction, similar to the exiting "Silent" icon.

For now, this just makes this stuff more auditable. Future changes may add ways to require MFA for certain specific transactions, outside of the ones that already always require MFA (like revealing credentials).

Test Plan: {F5877960}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19665
2018-09-12 12:57:02 -07:00
epriestley
8268abcb78 Enrich "diffusion.commit.search" with identity, status, and message information
Summary:
Depends on D19657. Ref T13197. See PHI841.

This enriches the results from `diffusion.commit.search` with information similar to the information returned by the "commits" attachment from `differential.diff.search`.

Also include unreachable, imported, message, audit status, and repository PHID.

Test Plan: Called `diffusion.commit.search` and reviewed the results, which looked sensible.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19658
2018-09-12 12:45:44 -07:00
epriestley
c7e7b63f15 Rename "PhabricatorAuditCommitStatusConstants" to "DiffusionCommitAuditStatus"; remove "MODERN_"
Summary:
Depends on D19656. Ref T13197. See PHI851.

  - This class is now a real object, so get rid of the "Constants" part of the name.
  - Rename it for greater consistency with other modern objects.
  - Get rid of the `MODERN_` tag now that the old constants are gone.

Test Plan: Bunch of `grep`, browsed around.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19657
2018-09-12 12:44:43 -07:00
epriestley
aaf2269551 Remove legacy numeric Audit status constants
Summary: Depends on D19655. Ref T13197. These no longer have callers.

Test Plan: Grepped for these constants.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19656
2018-09-12 12:40:55 -07:00
epriestley
d63281cc54 Migrate remaining Audit database status constants
Summary: Depends on D19652. Ref T13197. See PHI851. This migrates the actual `auditStatus` on Commits, and older status transactions.

Test Plan:
  - Ran migrations.
  - Spot-checked the database for sanity.
  - Ran some different queries, got unchanged results from before migration.
  - Reviewed historic audit state transactions, and accepted/raised concern on new audits. All state transactions appeared to generate properly.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19655
2018-09-12 12:21:27 -07:00
epriestley
cc3b6d5790 Fix a "withHasTransactions()" typo in AuditEditor
Summary: See <https://discourse.phabricator-community.org/t/typo-in-phabricatorauditeditor-php/1910>. This is trivial and reproduces easily, I just missed it in testing.

Test Plan:
  - Left a comment on a commit which I was the author of.
  - Before change: fatal with obvious typo.
  - After change: smooth sailing.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19667
2018-09-12 12:17:58 -07:00
epriestley
2379c21fbb Give Phriction documents a normal timeline
Summary:
Ref T13077. See PHI840. Ref T1894. I'm planning to just let you comment on Phriction documents. I think this will create a few problems (e.g., around popular documents which collect long comment threads that are eventually obsolete) but nothing should be too terribly critical (e.g., we handle it gracefully when objects have very large number of comments/transactions) and for most documents this is likely just a net improvement.

"Just enable comments" is probably not the final iteration on this, but I think it's probably a step forward on the balance, not a step sideways or a slippery slope down into a dark hole or anything.

Test Plan: {F5877316}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13077, T1894

Differential Revision: https://secure.phabricator.com/D19659
2018-09-11 13:31:57 -07:00
epriestley
185e72c881 Add aural section headers for "Event Timeline", "Add Comment", and "Comment Preview"
Summary: See PHI871. Ref T13197. These sections are only divided visually and don't have textual headers. Add aural headers.

Test Plan: {F5875471}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19654
2018-09-11 13:30:10 -07:00
epriestley
8eb8e8e1d8 Make DiffusionCommitSearch accept modern (string) constants
Summary:
Depends on D19650. Ref T13197. Allow `SearchCheckboxesField` to have a "deprecated" map of older aliases, then convert them to modern values.

On the API method page, show all the values.

This technically resolves the issue in PHI841, although I still plan to migrate behind this.

Test Plan:
{F5875363}

- Queried audits, fiddled with `?status=1,audited`, etc.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19651
2018-09-10 16:25:42 -07:00
epriestley
853a816b3c Continue converting Audit constants, allowing the Query to handle either strings or integers
Summary: Ref T13197. We're almost ready to migrate: let the Query accept either older integer values or new string values. Then move some callsites to use strings.

Test Plan: Called `audit.query`, browsed audits, audited commits.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13197

Differential Revision: https://secure.phabricator.com/D19650
2018-09-10 14:46:47 -07:00
epriestley
bae8a95114 Continue replacing Commit/Audit status checks with object-based checks
Summary: Ref T13195. See PHI851. Continuing down the path toward replacing these legacy numeric constants with more modern string constants.

Test Plan:
- Raised concern, requested verification, verified.
- Looked at commit hovercard with audit status.
- Viewed header on a commit page.
- (Didn't test the Doorkeeper stuff since it requires linking to Asana and seems unlikely to break.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19647
2018-09-10 11:20:31 -07:00
epriestley
16a6fc8341 Allow reviewers to mark their own inlines as "Done" before they submit them
Summary:
Ref T13195. Ref T8573. This allows reviewers to mark their own inline comments as "Done" before they submit them.

If you're leaving a non-actionable comment like "this is good", you can pre-check "Done" to give the author a hint that you don't expect any response.

Test Plan: On revisions and commits, added inlines as the author and a reviewer/auditor. Marked them done/not-done before submitting. As author, marked the not-done ones done after submitting. Checked preivews, toggled done/not done states.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195, T8573

Differential Revision: https://secure.phabricator.com/D19634
2018-09-07 11:17:42 -07:00
epriestley
046c1b5b82 Expose "isDraft" and "holdAsDraft" revision properties over Conduit
Summary:
Ref T13195. See PHI861. The "+ Draft" flag is not currently exposed over the API, but seems stable enough to expose.

Also expose the "hold as draft" flag, normally `arc diff --draft`.

Today, you can get "+ Draft" with some other state by:

  - abandoning a draft revision ("Abandoned + Draft"); or
  - using `arc diff --plan-changes` with an older `arc` version ("Changes Planned + Draft").

Test Plan: Called `differential.revision.search` via Conduit and got sensible-looking results for revisions in various states.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19648
2018-09-07 11:16:42 -07:00
epriestley
a1ce23b9f5 Introduce an AuditStatus object for commits and move some callsites to it
Summary:
Ref T13195. See PHI851. Add an object, analogous to the `DifferentialRevisionStatus` object, to handle audit status management.

This will primarily make it easier to swap storage over to strings later, but also cleans things up a bit.

Test Plan: Viewed audit/commit lists, saw sensible state icons. Ran `bin/audit synchronize`, got sensible output.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19646
2018-09-07 10:20:04 -07:00
epriestley
ab4d33bede Allow underscores to appear in Harbormaster variable names
Summary:
See PHI859. Ref T13195. The regexp for replacing variables currently does not include underscores.

Include underscores.

I also made a note in T13088: we should (almost certainly?) throw immediately if you try to pass a bogus variable name as a custom parameter, but this is a slightly larger change.

Test Plan:
  - Made an "http request" build plan with `?x=${initiator.phid}&y={$some_variable}`.
  - Added `some_variable` as a parameter to the parameter collection.
  - Before patch: `initiator.phid` was replaced, but `some_variable` was not.
  - After patch: both variables are replaced.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19645
2018-09-07 09:53:59 -07:00
epriestley
e8e5dc0f56 Make a language improvement ("inlines" -> "inline comments")
Summary: See D19632. Agreed that this is more clear.

Test Plan: Read carefully.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19644
2018-09-06 11:43:52 -07:00
epriestley
ef26b06ca8 Begin transitioning audits to modern (string) status constants, from legacy (integer) status constants
Summary:
Ref T13195. See PHI851. Audits currently have older integer status constants. We've moved almost all object types away from this to string constants (which are better in basically every way, and particularly way better for exposing over the API).

Commits/audits are currently accessible over the API and expose these constants via a "statuses" constraint.

Prepare to move toward modern string constants by defining a new, more modern map of status details and defining the existing methods in terms of it.

Test Plan: Browsed audits checking for icons/names/open-ness, saw no changes. This change should have no user-visible effects, as it just reorganizes code.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19642
2018-09-06 10:56:47 -07:00
epriestley
5a38b75f16 In Conduit, let checkbox constraints self-document
Summary:
Ref T13195. Ref PHI851. Currently, checkbox constraints don't tell you what values are supported on the API page.

You can figure this out with "Inspect Element" or by viewing the source code, but just render a nice table instead.

Test Plan: {F5862969}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19641
2018-09-06 08:44:16 -07:00
epriestley
a20f0674a9 Improve some documentation for "diffusion.commit.search"
Summary:
Ref T13195. See PHI851. Start by making some minor improvements here:

  - Clarify that the example of what constraints look like is just an example.
  - Add descriptions for parameters missing descriptions.

Test Plan: Looked at API method page, saw more helpful/complete instructions.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19640
2018-09-06 08:39:21 -07:00
epriestley
e6ee5ee9a1 When applying repository operations via Drydock, provide more context on OperationType
Summary: Ref T13195. See PHI845. For custom OperationTypes, provide access to the Interface and Operation via getters. This is just for convenience, since passing these around everywhere can be a bit of a pain if you have a deep-ish stack of things or love using callbacks or whatever.

Test Plan: Landed a revision via upstream Drydock operations.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19636
2018-09-06 08:15:16 -07:00
epriestley
041392988e When a transaction adds more than 100 inline comments, include only the first 100 in email
Summary:
Ref T13195. An install had a user apply a transaction which added about 1,000 inline comments. Rendering the email for this transaction took a very long time because the context section for each comment must be highlighted separately.

We can make the highlighting faster (in this case, by porting the lexer to PHP) but it's also sort of silly to include more than 100 inlines in an email. These emails are likely to be truncated by outbound size rules at some point anyway. Instead, limit inlines rendered directly into email to the first 100 per transaction group.

Test Plan:
Set limit to 2, added 4 comments, viewed text and HTML emails:

{F5859967}

{F5859968}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195

Differential Revision: https://secure.phabricator.com/D19632
2018-09-06 07:58:05 -07:00
epriestley
650e74933a Update some inline comment logic to use more modern "Viewer"-oriented calls/variables
Summary:
Ref T13195. Ref T8573. The inline comment controllers currently use outdated `$user = $this->getRequest()->getUser()` calls.

Instead, use `$viewer = $this->getViewer()`.

This is just a small consistency update with no behavioral changes.

Test Plan: Viewed and added inlines in Differential and Diffusion.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13195, T8573

Differential Revision: https://secure.phabricator.com/D19633
2018-09-06 07:57:41 -07:00
epriestley
26d9ee456f Hide the new Phriction "Publish" operation behind the "Prototype" toggle
Summary: Ref T13077. This is currently a little too confusing to go out into the world, mostly because there's no way to edit documents without auto-publishing them. Keep it out of the spotlight for this release.

Test Plan: Viewed Phriction, saw publish operation marked as a prototype.

Reviewers: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19627
2018-08-31 10:30:20 -07:00
epriestley
18e8d9a452 Make the Phriction "History" view more aware of drafts
Summary: Ref T13077. Updates the "History" view to be slightly better organized and draft-aware.

Test Plan: Viewed page history in Phriction.

Reviewers: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19626
2018-08-30 10:36:55 -07:00
epriestley
3b1294cf45 Store Phriction max version on Document, improve editing rules for editing documents with drafts
Summary:
Ref T13077. We need to know the maximum version of a document in several cases, so denormalize it onto the Document object.

Then clean up some behaviors where we edit a document with, e.g., 7 versions but version 5 is currently published. For now, we: edit starting with version 7, save as version 8, and immediately publish the new version.

Test Plan:
  - Ran migration.
  - Edited a draft page without hitting any weird version errors.
  - Checked database for sensible `maxVersion` values.

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19625
2018-08-30 10:12:51 -07:00
epriestley
0a77b0e53e Work around an issue in MariaDB where dropping a column from a UNIQUE KEY fails
Summary:
See T13193. See T13077. If we drop a column which is part of a UNIQUE KEY, MariaDB raises an error.

This is probably a bad idea on our side anyway, but in this case it wasn't an obviously bad idea.

To get around this:

  - Drop the unique key, if it exists, before dropping the column.
  - Explicitly add the new unique key afterward.

Test Plan: Ran `bin/storage upgrade` locally without issue, but I'm on MySQL. Will follow up on T13193.

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19624
2018-08-30 06:25:39 -07:00
epriestley
75a0455152 Add "Revision test plan" as a Herald field; remove test plan from the "Revision summary" field
Summary:
See PHI844. Ref T13189.

Add "Revision test plan" as an available field for Herald. This is a little niche -- and a little odd because it sticks around even if you fully disable test plans -- but probably broadly reasonable.

The existing "Revision summary" field counterintuitively included the test plan. Separate this out since it's now a separate field and the behavior was weird historic nonsense. I'll note this in the changelog.

Test Plan: Wrote a rule using both fields, verified they generated the expected values.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19623
2018-08-29 14:17:38 -07:00
epriestley
876638e428 Add a UI element for navigating between versions of a Phriction document
Summary: Depends on D19621. Ref T13077. Fixes T4815. This adds previous/current/next/draft buttons and makes navigation between unpublished and published versions of a document more clear.

Test Plan: {F5841997}

Reviewers: amckinley

Maniphest Tasks: T13077, T4815

Differential Revision: https://secure.phabricator.com/D19622
2018-08-29 13:49:15 -07:00
epriestley
349686319e Allow the published version of a Phriction document to differ from the most recent version
Summary:
Depends on D19620. Ref T13077. This adds a "Publish" operation which points the current version at some historical version of the document -- not necessarily the most recent version. Newer versions become "drafts".

This is still quite rough and missing a lot of hinting in the UI, I'm just making it work so I can start making the UI understand it.

Test Plan: Used the "Publish" action to publish older versions of a document, saw the document revert. Many UI hints are missing and this operation is puzzling and not yet usable for normal users.

Reviewers: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19621
2018-08-29 13:47:36 -07:00
epriestley
50f4adef64 Remove on-object mailkeys from Phriction
Summary: Depends on D19619. Ref T13065. Ref T13077. Migrate Phriction mail keys to the new infrastructure and drop the column.

Test Plan: Ran migrations, spot-checked the database.

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13077, T13065

Differential Revision: https://secure.phabricator.com/D19620
2018-08-29 13:43:13 -07:00
epriestley
64cee4a902 Move Phriction internal document/content references from IDs to PHIDs
Summary:
Ref T13077. This is mostly just a small cleanup change, even though the actual change is large.

We currently reference content and document objects from one another with `contentID` and `documentID`, but this means that `contentID` must be nullable. Switching to PHIDs allows the column to be non-nullable.

This also supports reorienting some current and future transactions around PHIDs, which is preferable for the API. In particular, I'm adding a "publish version X" transaction soon, and would rather callers pass a PHID than an ID or version number, since this will make the API more consistent and powerful.

Today, `contentID` gets used as a cheaty way to order documents by (content) edit time. Since PHIDs aren't orderable and stuff is going to become actually-revertible soon, replace this with an epoch timestamp.

Test Plan:
  - Created, edited, moved, retitled, and deleted Phriction documents.
  - Grepped for `documentID` and `contentID`.
  - This probably breaks //something// but I'll be in this code for a bit and am likely to catch whatever breaks.

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19619
2018-08-29 13:41:24 -07:00
epriestley
04f8270a74 Remove some unusual UI policy hints in Phriction
Summary:
Ref T13077. We currently have these weird policy hints in Phriction that we don't use in other applications. Just remove them for consistency to make the eventual swap to EditEngine a little easier.

Also nuke some unreacahble code.

Test Plan: Loaded edit page, saw more standard UI.

Reviewers: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19618
2018-08-29 07:32:45 -07:00
epriestley
4afb6446d9 Allow DocumentView to render with a curtain, and make Phriction use a curtain
Summary:
Depends on D19616. Ref T13077. Fixes T8172. In the last round of design updates, a lot of actions got stuffed into "Actions" menus.

I never really got used to these and think they're a net usability loss, and broadly agree with the feedback in T8172. I'd generally like to move back toward a state where actions are available on the page, not hidden in a menu.

For now, just put a curtain view on these pages. This could be refined later (e.g., stick this menu to the right hand side of the screen) depending on where other Phriction changes go.

(Broadly, I'm also not satisfied with where we ended up on the fixed-width pages like Diffusion > Manage, Config, and Instances. In contrast, I //do// like where we ended up with Phortune in terms of overall design. I anticipate revisiting some of this stuff eventually.)

Test Plan:
  - Looked at Phriction pages on desktop/tablet/mobile/printable -- actions are now available on the page.
  - Looked at other DocumentView pages (like Phame blogs) -- no changes for now.

Reviewers: amckinley

Maniphest Tasks: T13077, T8172

Differential Revision: https://secure.phabricator.com/D19617
2018-08-28 14:58:05 -07:00
epriestley
fd0da4c41f Rename "PHUIDocumentViewPro" to "PHUIDocumentView"
Summary: Ref T13077. There is no "PHUIDocumentView" so toss the "Pro" suffix from this classname.

Test Plan: Grepped for `PHUIDocumentView` and `PHUIDocumentViewPro`.

Reviewers: amckinley

Maniphest Tasks: T13077

Differential Revision: https://secure.phabricator.com/D19616
2018-08-28 14:53:07 -07:00
epriestley
614f9ba1fb Allow unit test results to specify that their details are formatted with remarkup when reporting to "harbormaster.sendmessage"
Summary: Ref T13189. See PHI710. Ref T13088. Fixes T9951. Allow callers to `harbormaster.sendmessage` to specify that the test details are remarkup so they can use rich formatting and include links, files, etc.

Test Plan: {F5840098}

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13189, T13088, T9951

Differential Revision: https://secure.phabricator.com/D19615
2018-08-28 13:26:11 -07:00
epriestley
632cafec88 Pass commit authorship information to Buildkite
Summary:
Fixes T12251. Ref T13189. See PHI610. The difficulty here is that we don't want to disclose Phabricator account information to Buildkite. We're comfortable disclosing information from `git`, etc.

  - For commits, use the Identity to provide authorship information from Git.
  - For revisions, use the local commit information on the Diff to provide the Git/Mercurial/etc author of the HEAD commit.

Test Plan:
  - Built commits and revisions in Buildkite via Harbormaster.
  - I can't actually figure out how to see author information on the Buildkite side, but the values look sane when dumped locally.

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13189, T12251

Differential Revision: https://secure.phabricator.com/D19614
2018-08-27 12:52:11 -07:00
epriestley
2f5c6541fc Add an "Activated Epoch" and an "Acquired Epoch" to Drydock Leases
Summary: Ref T13189. See PHI690. When a lease is first acquired or activated, note the time. This supports better visibility into queue lengths. For now, this is only queryable via DB and visible in the UI, but can be more broadly exposed in the future.

Test Plan: Landed a revision, saw the leases get sensible timestamps for acquisition/activation.

Reviewers: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19613
2018-08-27 11:27:45 -07:00
epriestley
ee823982a4 Remove another old remarkup engine callsite in Config
Summary: Ref T13189. Summaries do not appear to be meaningfully rendered with Remarkup so just drop the engine. See D19610 for the previous change in this vein.

Test Plan: Viewed config list with option summaries.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19612
2018-08-27 11:10:14 -07:00
epriestley
b87a809b0b Make some remarkup handling in Config cleaner, fixing {{other.option} links
Summary:
Depends on D19609. Ref T13189. At some point, we switched from RemarkupEngine to RemarkupView and lost this piece of hack-magic.

Restore the hack-magic. It's still hack-magic instead of a real rule, but things are at least cleaner than they were before.

Test Plan: Viewed `auth.require-approval`, etc. Saw references to other config options linked properly.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19610
2018-08-27 09:54:19 -07:00
epriestley
0295a00229 When there are no setup issues, don't show a weird empty box
Summary: Ref T13189. When there are no setup issues, we currently double-render a weird setup issues box underneath the notice. Get rid of it.

Test Plan: Viewed page with and without setup issues, saw less awkward UI.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19609
2018-08-27 09:53:52 -07:00
epriestley
cd8b5b82c8 Stop requiring CAN_EDIT to reach the TransactionEditor via "*.edit" in EditEngine
Summary:
Depends on D19607. Ref T13189. See PHI642. Ref T13186.

Some transactions can sometimes be applied to objects you can not edit. Currently, using `*.edit` to edit an object always explicitly requires CAN_EDIT.

Now that individual transactions require CAN_EDIT by default and can reduce or replace this requirement, stop requiring CAN_EDIT to reach the editor.

The only expected effect of this change is that low-permission edits (like disabling a user, leaving a project, or leaving a thread) can now work via `*.edit`.

Test Plan:
  - Tried to perform a normal edit (changing a task title) against an object with no CAN_EDIT. Still got a permissions error.
  - As a non-admin, disabled other users while holding the "Can Disable Users" permission.
  - As a non-admin, got a permissions error while trying to disable other users while not holding the "Can Disable Users" permission.

Reviewers: amckinley

Maniphest Tasks: T13189, T13186

Differential Revision: https://secure.phabricator.com/D19608
2018-08-27 08:10:08 -07:00
epriestley
f9192d07f2 Align web UI "Disable" and "Approve/Disapprove" flows with new "Can Disable Users" permission
Summary:
Depends on D19606. Ref T13189. See PHI642.

  - Disabling/enabling users no longer requires admin. Now, you just need "Can Disable Users".
  - Update the UI to appropriately show the action in black or grey depending on what clicking the button will do.
  - For "Approve/Disapprove", fix a couple bugs, then let them go through without respect for "Can Disable Users". This is conceptually a different action, even though it ultimately sets the "Disabled" flag.

Test Plan:
  - Disabled/enabled users from the web UI as various users, including a non-administrator with "Can Disable Users".
  - Hit permissions errors from the web UI as various users, including an administrator without "Can Disable Users".
  - Saw the "Disable/Enable" action activate properly based on whether clicking the button would actually work.
  - Disapproved a user without "Can Disable Users" permission, tried to re-disapprove a user.
  - Approved a user, tried to reapprove a user.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19607
2018-08-27 08:09:42 -07:00
epriestley
058952e72e Add a "Can Disable Users" capability to the "People" application
Summary:
Depends on D19605. Ref T13189. See PHI642. This adds a separate "Can Disable Users" capability, and makes the underlying transaction use it.

This doesn't actually let you weaken the permission, since all pathways need more permissions:

  - `user.edit` needs CAN_EDIT.
  - `user.disable/enable` need admin.
  - Web UI workflow needs admin.

Upcoming changes will update these pathways.

Without additional changes, this does let you //strengthen// the permission.

This also fixes the inability to disable non-bot users via the web UI.

Test Plan:
  - Set permission to "No One", tried to disable users. Got a tailored policy error.
  - Set permission to "All Users", disabled/enabled a non-bot user.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19606
2018-08-27 08:01:27 -07:00
epriestley
8cf56913d8 Deprecate "user.enable" and "user.disable" API methods, redefine them in terms of "user.edit"
Summary:
Depends on D19604. Ref T13189. See PHI642. Deprecates these in favor of "user.edit", redefines them in terms of it, and removes the old `disableUser()` method.

I kept the "is admin" permissions check for consistency, since these methods have always said "(admin only)". This check may not be the most tailored check soon, but we can just keep executing it in addition to the real check.

For now, this change stops this method from actually disabling non-bot users (since it implicitly adds a CAN_EDIT requirement, and even administrators don't have that). An upcoming change will fix that.

Test Plan: Enabled and disabled a (bot) user via these methods. Checked API UI, saw them marked as "disabled".

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19605
2018-08-27 08:00:48 -07:00
epriestley
2f7b10c023 Replace "Disable User" web UI flow with transactions
Summary:
Ref T13189. See PHI642. Upgrades the "Disable" action in the web UI to be transaction-based.

This technically breaks things a little (you can't disable non-bot users, since they now require CAN_EDIT and you won't have it) but an upcoming change will fix the permissions issue.

Test Plan: Disabled and enabled a (bot) user from the web UI.

Reviewers: amckinley

Maniphest Tasks: T13189

Differential Revision: https://secure.phabricator.com/D19604
2018-08-27 08:00:21 -07:00
epriestley
4d89afcc61 Remove requireCapabilities() from ApplicationTransactionEditor and require CAN_EDIT by default
Summary:
Depends on D19585. Ref T13164.

Almost all transactions require CAN_EDIT on the object, but they generally do not enforce this directly today. Instead, this is effectively enforced by Controllers, API methods, and EditEngine doing a `CAN_EDIT` check when loading the object to be edited.

A small number of transactions do not require CAN_EDIT, and instead require only a weaker/lesser permission. These are:

  - Joining a project which you have CAN_JOIN on.
  - Leaving a project which isn't locked.
  - Joining a Conpherence thread you can see (today, no separate CAN_JOIN permission for Conpherence).
  - Leaving a Conpherence thread.
  - Unsubscribing.
  - Using the special `!history` command from email.

Additionally, these require CAN_INTERACT, which is weaker than CAN_EDIT:

  - Adding comments.
  - Subscribing.
  - Awarding tokens.

Soon, I want to add "disabling users" to this list, so that you can disable users if you have "Can Disable User" permission, even if you can not otherwise edit users.

It's possible this list isn't exhaustive, so this change might break something by adding a policy check to a place where we previously didn't have one. If so, we can go weaken that policy check to the appropriate level.

Enforcement of these special cases is currently weird:

  - We mostly don't actually enforce CAN_EDIT in the Editor; instead, it's enforced before you get to the editor (in EditEngine/Controllers).
  - To apply a weaker requirement (like leaving comments or leaving a project), we let you get through the Controller without CAN_EDIT, then apply the weaker policy check in the Editor.
  - Some transactions apply a confusing/redundant explicit CAN_EDIT policy check. These mostly got cleaned up in previous changes.

Instead, the new world order is:

  - Every transaction has capability/policy requirements.
  - The default is CAN_EDIT, but transactions can weaken this explicitly they want.
  - So now we'll get requirements right in the Editor, even if Controllers or API endpoints make a mistake.
  - And you don't have to copy/paste a bunch of code to say "yes, every transaction should require CAN_EDIT".

Test Plan:
- Tried to add members to a Conpherence thread I could not edit (permissions error).
- Left a Conpherence thread I could not edit (worked properly).
- Joined a thread I could see but could not edit (worked properly).
- Tried to join a thread I could not see (permissions error).
- Implemented `requireCapabilites()` on ManiphestTransactionEditor and tried to edit a task (upgrade guidance error).
- Mentioned an object I can not edit on another object (works).
- Mentioned another object on an object I can not edit (works).
- Added a `{F...}` reference to an object I can not edit (works).
- Awarded tokens to an object I can not edit (works).
- Subscribed/unsubscribed from an object I can not edit (works).
- Muted/unmuted an object I can not edit (works).
- Tried to do other types of edits to an object I can not edit (correctly results in a permissions error).
- Joined and left a project I can not edit (works).
- Tried to edit and add members to a project I can not edit (permissions error).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19586
2018-08-24 17:45:56 -07:00
epriestley
b584834b19 In Differential: when the file tree is enabled, default to the "History" tab instead of "Files"
Summary:
Ref T13187. See PHI811. If the file tree is enabled and visible, set the default tab to "History".

  - This is a bit magic.
  - It won't work entirely on mobile (we can't tell that you're on mobile on the server, so we'll pick the "History" tab even though the file tree isn't actually visible on your device).
  - There's no corresponding logic in Diffusion. Diffusion doesn't have the same tab layout, but this makes things somewhat inconsistent.

So I don't love this, but we can try it and see if it's confusing or helpful on the balance.

Test Plan: With filetree on and off, reloaded revisions. Saw appropriate tab selected by default.

Reviewers: amckinley

Maniphest Tasks: T13187

Differential Revision: https://secure.phabricator.com/D19601
2018-08-24 10:29:35 -07:00
epriestley
b6fa009cf0 Enrich "priority" transactions in Maniphest for "transaction.search"
Summary:
Ref T13187. See <https://discourse.phabricator-community.org/t/task-priority-change-info-missing-in-firehose-webhook/1832/2>. We can reasonably enrich these transactions.

Since priorities don't have unique authorative string identifiers, I've mostly mimicked the `maniphest.search` structure.

Test Plan: Called `transaction.search` on tasks which were: created normally, created with a priority change, saw a priority change after creation. All the output looked useful and sensible.

Reviewers: amckinley

Maniphest Tasks: T13187

Differential Revision: https://secure.phabricator.com/D19599
2018-08-24 10:05:05 -07:00
epriestley
5e4d9dfa92 Condition "Changes Since Last Action" Differential link on "first broadcast", not "new object"
Summary: Ref T13187. Ref T13176. With drafts, we actually want to suppress this link on "first broadcast" (the first time we send mail), not on "new object" (when the revision is created as a draft).

Test Plan: Poked at this locally, will keep an eye on it in production.

Reviewers: amckinley

Maniphest Tasks: T13187, T13176

Differential Revision: https://secure.phabricator.com/D19598
2018-08-24 10:03:55 -07:00
epriestley
ca618a8679 Document that phd.taskmasters is a local setting, per daemon
Summary: Ref T13187. See PHI807. The documentation currently does not make it very clear that this is a local setting, per `phd` process. Make it more clear.

Test Plan: {F5827757}

Reviewers: amckinley

Maniphest Tasks: T13187

Differential Revision: https://secure.phabricator.com/D19597
2018-08-24 08:08:19 -07:00
epriestley
7ef2bb1b56 Support Mercurial "protocaps" wire command
Summary:
Ref T13187. See PHI834. Mercurial has somewhat-recently (changeset is from Jan 2018) introduced a new "protocaps" command, that appears in Mercurial 4.7 and possibly before then.

We must explicitly enumerate all protocol commands because you can't decode the protocol without knowing how many arguments the command expects, so enumerate it.

(Also fix an issue where the related error message had an extra apostrophe.)

Test Plan:
  - Ran `hg clone ...` with client and server on Mercurial 4.7.
  - Before: fatal on unknown "protocaps" command.
  - Midway: better typography in error message.
  - After: clean clone.

Reviewers: amckinley

Maniphest Tasks: T13187

Differential Revision: https://secure.phabricator.com/D19596
2018-08-23 15:06:25 -07:00
epriestley
75a5dd8d8c Add more accessibility labels for screen readers
Summary:
Depends on D19594. See PHI823. Ref T13164.

  - Add a label for the "X" button in comment areas, like "Remove Action: Change Subscribers".
  - Add a label for the floating header display options menu in Differential.
  - Add `role="button"` to `PHUIButtonView` objects that we render with an `<a ...>` tag.

Test Plan:
Viewed a revision with `?__aural__=true`:

  - Saw "Remove Action: ..." label.
  - Saw "Display Options" label.
  - Used inspector to verify that some `<a class="button" ...>` now have `<a class="button" role="button" ...>`. This isn't exhaustive, but at least improves things. A specific example is the "edit", "reply", etc., actions on inline comments.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19595
2018-08-17 13:31:51 -07:00
Austin McKinley
5c4c593af3 Update DiffusionLastModifiedController to use identities
Summary: Ref T12164. Updates another controller to use identities.

Test Plan:
Pretty ad-hoc, but loaded the main pages of several different repos with and without repo identities. I'm not totally convinced the `author` from this data structure is actually being used:
```
$return = array(
  'commit'    => $modified,
  'date'      => $date,
  'author'    => $author,
  'details'   => $details,
);
```

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19580
2018-08-17 12:24:21 -07:00
epriestley
438edde031 Add some missing aural button labels for accessibility
Summary:
Ref T13164. See PHI823. (See that issue for some more details and discussion.)

Add aural labels to various buttons which were missing reasonable aural labels.

The "Search" button (magnifying glass in the global search input) had an entire menu thing inside it. I moved that one level up and it doesn't look like it broke anything (?). All the other changes are pretty straightforward.

Test Plan:
{F5806497}

{F5806498}

  - Will follow up on the issue to make sure things are in better shape for the reporting user.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19594
2018-08-17 11:00:29 -07:00
epriestley
a48e6897a4 Remove obsolete setup check call to Maniphest "Can Edit <X>" field checks
Summary: Ref T13164. Missed this in D19581.

Test Plan:
  - Forced setup checks to re-run by visiting {nav Config > Setup Issues} explicitly.
  - Before patch: fatal on call to nonexistent method.
  - After patch: setup issues.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19593
2018-08-16 13:56:20 -07:00
epriestley
0ccf1410e0 Give PhabricatorAuthPassword a formal CAN_EDIT policy
Summary:
Depends on D19585. Ref T13164. This is a precursor for D19586, which causes Editors to start doing more explicit CAN_EDIT checks.

Passwords have an Editor, but don't actually define a CAN_EDIT capability. Define one (you can edit a password if you can edit the object the password is associated with).

(Today, this object is always a User -- this table just unified VCS passwords and Account passwords so they can be handled more consistently.)

Test Plan:
  - With D19586, ran unit tests and got a pass.
  - Edited my own password.
  - Tried to edit another user's password and wasn't permitted to.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19592
2018-08-16 11:53:24 -07:00
epriestley
7e29ec2e2a Move the "Can Lock Projects" check from requireCapabilities() to transaction validation
Summary: Depends on D19584. Ref T13164. This check is an //extra// check: you need EDIT //and// this capability. Thus, we can do it in validation without issues.

Test Plan:
  - This code isn't reachable today: all methods of applying this transaction do a separate check for "Can Lock" upfront.
  - Commented out the "Can Lock" check in the LockController, tried to lock as a user without permission. Was rejected with a policy exception.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19585
2018-08-16 10:56:00 -07:00
epriestley
3b92da22f4 Move the hierarchical edit policy check in Phriction from requireCapabilities() to validateTransactions()
Summary:
Depends on D19583. Ref T13164. This continues the work of getting rid of `requireCapabilities()`.

This check is valid, but can be a `validateTransactions()` check instead. This is generally more consistent with how other applications work (e.g., creating subprojects).

The UI for this isn't terribly great: you get a policy error //after// you try to create the object. But that's how it worked before, so this isn't any worse than it was. The actual policy exception is (very) slightly more clear now (raised against the right object).

Test Plan:
  - Created a child as a user with permission to do so to make sure I didn't break that.
  - Set edit permission on `a/` to just me, tried to create `a/b/` as another user, got a policy exception since they can't edit the parent.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19584
2018-08-16 10:55:11 -07:00
epriestley
24d4445845 Remove pointless requireCapabilities() method from PhabricatorRepositoryEditor
Summary: Depends on D19582. Ref T13164. It's not possible to reach the editor without passing through a CAN_EDIT check, and it shouldn't be necessarily to manually specify that edits require CAN_EDIT by default.

Test Plan: Grepped for `RepositoryEditor`, verified that all callsites pass through a CAN_EDIT check.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19583
2018-08-16 10:53:42 -07:00
epriestley
a39852ae1b Remove pointless requireCapabilities() method from PhabricatorProjectColumnTransactionEditor
Summary:
Depends on D19581. Ref T13164. This method has no effect:

  - You must always have CAN_EDIT to reach an Editor in the first place.
  - Per previous change, I'm going to restructure this so transactions explicitly check CAN_EDIT by default anyway.

Test Plan: Tried to edit and hide a project column as a user without permission, hit global permission checks long before reaching this method.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19582
2018-08-16 10:51:57 -07:00
epriestley
296bf046a8 Remove deprecated Maniphest "Can Edit <Specific Property>" capabilities
Summary:
Depends on D19579. Fixes T10003. These have been deprecated with a setup warning about their impending removal for about two and a half years.

Ref T13164. See PHI642. My overall goal here is to simplify how we handle transactions which have special policy behaviors. In particular, I'm hoping to replace `ApplicationTransactionEditor->requireCapabilities()` with a new, more clear policy check.

A problem with `requireCapabilities()` is that it doesn't actually enforce any policies in almost all cases: the default is "nothing", not CAN_EDIT. So it ends up looking like it's the right place to specialize policy checks, but it usually isn't.

For "Disable", I need to be able to weaken the check selectively (you can disable users if you have the permission, even if you can't edit them otherwise). We have a handful of other edits which work like this (notably, leaving and joining projects) but they're very rare.

Test Plan: Grepped for all removed classes. Edited a Maniphest task.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164, T10003

Differential Revision: https://secure.phabricator.com/D19581
2018-08-16 10:51:06 -07:00
epriestley
f9673a72a8 Allow "user.edit" to enable or disable users
Summary:
Depends on D19577. Ref T13164. See PHI642. This adds modern transaction-oriented enable/disable support.

Currently, this also doesn't let you disable normal users even when you're an administrator. I'll refine the policy model later in this change series, since that's also the goal here (let users set "Can Disable Users" to some more broad set of users than "Administrators").

This also leaves us with two different edit pathways: the old UserEditor one and the new UserTransactionEditor one. The next couple diffs will redefine the other pathways in terms of this pathway.

Test Plan:
  - Enabled/disabled a bot.
  - Tried to disable another non-bot user. This isn't allowed yet, since even as an administrator you don't have CAN_EDIT on them and currently need it: right now, there's no way for a particular set of transactions to say they can move forward with reduced permissions.
  - Tried to enable/disable myself. This isn't allowed since you can't enable/disable yourself.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19579
2018-08-16 10:49:35 -07:00
epriestley
65904d7c51 Add a modern "user.edit" API method for users
Summary:
Depends on D19576. Ref T13164. See PHI642. This adds an EditEngine for users and a `user.edit` modern API method.

For now, all it supports is editing real name, blurb, title, and icon (same as "Edit Profile" from the UI).

Test Plan:
  - Edited my stuff via the new API method.
  - Tried to edit another user, got rejected by policies.
  - Tried to create a user, got rejected by policies.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19577
2018-08-16 10:48:38 -07:00
epriestley
39d415e90e Move users to modular transactions
Summary:
Ref T13164. See PHI642. I'd like to provide a third-generation `user.edit` API endpoint and make `user.enable` and `user.disable` obsolete before meddling with policy details, even if it isn't full-fledged yet.

Users do already have a transactions table and a Transaction-based editor, but it's only used for editing title, real name, etc. All of these are custom fields, so their support comes in automatically through CustomField extension code.

Realign it for modular transactions so new code will be fully modern. There are no actual standalone transaction types yet so this diff is pretty thin.

Test Plan:
  - Grepped for `UserProfileEditor`.
  - Edited a user's title/real name/icon.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19576
2018-08-16 10:47:47 -07:00
Austin McKinley
cc1def6cea Remove some array typehints for passing around
Summary: See discussion at https://secure.phabricator.com/D19492#241996

Test Plan: Refreshed a few Diffusion tabs; nothing broke.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19578
2018-08-13 16:07:56 -07:00
Austin McKinley
3b05e920e0 Start changing DiffusionCommitController to use identities
Summary: Depends on D19491.

Test Plan: Viewed some commits where the identity was mapped to a user and another that wasn't; saw the header render either a link to the user or the identity object.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19492
2018-08-13 15:23:31 -07:00
epriestley
92a29f72c1 Make the Drydock repository operation page slightly richer
Summary:
Ref T13164. See PHI788. The issue requests a "created" timestamp.

Also add filtering for repository, state, and author.

Test Plan:
Used all filters.

{F5795085}

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19574
2018-08-13 11:42:10 -07:00
epriestley
fb3ae72e36 When cancelling addition of an Almanac interface, return to the Device page
Summary:
Fixes T13184. In Almanac, interfaces are always added to devices. However, if you "Add New Interface" and then "Cancel", you go to the nonexistent `/interface/` page.

Instead, return to the device page.

Test Plan: From a device page, clicked "Add Interface" and then "Cancel". Ended up back where I was.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13184

Differential Revision: https://secure.phabricator.com/D19573
2018-08-13 11:39:37 -07:00
epriestley
b86dae6214 Fix an issue with error handling when no mailers are available
Summary:
Ref T13164. See PHI785. See D19546. I think I didn't test the updated error messaging here entirely properly, since I have some tasks in queue which error out here ("Missing argument 1 to newMailers(...)").

This is an error condition already, but we want to get through this call so we can raise a tailored message.

Test Plan: Tasks which errored out here now succeed. This condition is only reachable if you misconfigure things in the first place.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19572
2018-08-13 11:39:13 -07:00
epriestley
e5906f4e12 In Differential standalone views, disable some keyboard shortcuts which don't work
Summary:
Ref T13164. See PHI693. In Differential, you can {nav View Options > View Standalone} to get a standalone view of a single changeset. You can also arrive here via the big changeset list for revisions affecting a huge number of files.

We currently suggest that all the keyboard shortcuts work, but some do not. In particular, the "Next File" and "Previous File" keyboard shortcuts (and some similar shortcuts) do not work. In the main view, the next/previous files are on the same page. In the standalone view, we'd need to actually change the URI.

Ideally, we should do this (and, e.g., put prev/next links on the page). As a first step toward that, hide the nonfunctional shortcuts to stop users from being misled.

Test Plan:
  - Viewed a revision in normal and standalone views.
  - No changes in normal view, and all keys still work ("N", "P", etc).
  - In standalone view, "?" no longer shows nonfunctional key commands.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19571
2018-08-13 08:59:05 -07:00
Austin McKinley
a6951a0a5a Add migration to encourage rebuilding repository identities
Summary: Ref T12164. Defines a new manual activity that suggests rebuilding repository identities before Phabricator begins to rely on them.

Test Plan:
- Ran migration, observed expected setup issue: {F5788217}
- Ran `bin/config done identities` and observed setup issue get marked as done.
- Ran `/bin/storage upgrade --apply phabricator:20170912.ferret.01.activity.php` to make sure I didn't break the reindex migration; observed reindex setup issue appear as expected.
- Ran `./bin/config done reindex` and observed reindex issue cleared as expected.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19497
2018-08-10 13:47:03 -07:00
Austin McKinley
2951694c27 Correctly spell 'committer'
Summary: It's a funny word. h/t @joshuaspence

Test Plan: Inspection of correct spelling.

Reviewers: epriestley, joshuaspence

Reviewed By: joshuaspence

Subscribers: Korvin, joshuaspence

Differential Revision: https://secure.phabricator.com/D19570
2018-08-09 17:52:43 -07:00
epriestley
6df278bea8 In "bin/ssh-auth", cache a structure instead of a flat file because paths may change at runtime
Summary:
Fixes T12397. Ref T13164. See PHI801.

Several installs have hit various use cases where the path on disk where Phabricator lives changes at runtime. Currently, `bin/ssh-auth` caches a flat file which includes the path to `bin/ssh-exec`, so this may fall out of date if `phabricator/` moves.

These use cases have varying strengths of legitimacy, but "we're migrating to a new set of hosts and the pool is half old machines and half new machines" seems reasonably compelling and not a problem entirely of one's own making.

Test Plan:
  - Compared output on `master` to output after change, found them byte-for-byte identical.
  - Moved `phabricator/` to `phabricator2/`, ran `bin/ssh-auth`, got updated output.
  - Added a new SSH key, saw it appear in the output.
  - Grepped for `AUTHFILE_CACHEKEY` (no hits).
  - Dropped the cache, verified that the file regenerates cleanly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164, T12397

Differential Revision: https://secure.phabricator.com/D19568
2018-08-09 13:33:23 -07:00
epriestley
df31405d64 Improve compatibility of "Config > Cache Status" across APCu versions
Summary:
Ref T13164. See PHI790. Older versions of APCu reported cache keys as "key" from `apcu_cache_info()`. APC and newer APCu report it as "info".

Check both indexes for compatibility.

Test Plan:
  - Locally, with newer APCu, saw no behavioral change.
  - Will double check on `admin`, which has an older APCu with the "key" behavior.
  - (I hunted this down by dumping `apcu_cache_info()` on `admin` to see what was going on.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19569
2018-08-08 15:07:03 -07:00
epriestley
3ca3f09a1a Add an "--as" flag to "bin/conduit call ..." to improve flexibility and ease of profiling
Summary:
Ref T13164. In PHI801, an install reported a particular slow Conduit method call.

Conduit calls aren't easily profilable with normal tools (for example, `arc call-conduit --xprofile ...` gives you a profile of the //client//). They can be profiled most easily with `bin/conduit call ... --xprofile`.

However, `bin/conduit call` currently doesn't let you pick a user to execute the command on behalf of, so it's not terribly useful for profiling `*.edit`-style methods which do a write: these need a real acting user.

Test Plan:
Ran `bin/conduit call --method user.whoami --as epriestley ...` with valid, invalid, and no acting users.

```
$ echo '{}' | ./bin/conduit call --method user.whoami --as epriestley --input -
Reading input from stdin...
{
  "result": {
    "phid": "PHID-USER-icyixzkx3f4ttv67avbn",
    "userName": "epriestley",
    "realName": "Evan Priestley",
...
```

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19566
2018-08-08 09:51:21 -07:00
epriestley
91abc0f027 Stop indexing the chunk data objects for large Files stored in multiple chunks
Summary:
Ref T13164. See PHI766. Currently, when file data is stored in small chunks, we submit each chunk to the indexing engine.

However, chunks are never surfaced directly and can never be found via any search/query, so this work is pointless. Just skip it.

(It would be nice to do this a little more formally on `IndexableInterface` or similar as `isThisAnIndexableObject()`, but we'd have to add like a million empty "yes, index this always" methods to do that, and it seems unlikely that we'll end up with too many other objects like these.)

Test Plan:
  - Ran `bin/harbormaster rebuild-log --id ... --force` before and after change, saw about 200 fewer queries after the change.
  - Uploaded a uniquely named file and searched for it to make sure I didn't break that.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19563
2018-08-03 14:36:12 -07:00
epriestley
5839a54b60 Raise a tailored error when calling "transaction.search" with empty "phids" constraint
Summary:
Ref T13164. See PHI725. For real "*.search" methods, parameters get validated and you get an error if you use an empty list as a constraint.

Since "transaction.search" isn't really a normal "*.search" method, it doesn't benefit from this. Just do the check manually for now.

Test Plan: Made `transaction.search` calls with no constraints (got results); a valid costraint (got fewer results); and an invalid empty constraint (got an exception).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19562
2018-08-03 14:29:36 -07:00
epriestley
f3fa164882 Add a "Last Edited" property to Wiki pages
Summary:
Ref T13164. See PHI797. The last edit is available in the page header, but it's not precise (just says "180 days ago") and a little weird (it's unusual for us to put that kind of information in the header).

Add a precise timestamp to the footer for now. I'd imagine re-examining this the next time Phriction gets some UI work and maybe trying to integrate timeline/transactions more cleanly (see also T1894).

Test Plan: Looked at a wiki page, then edited it. Saw precise "Last Edit" timestamp adjacent to "Last Author".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19560
2018-08-03 14:29:17 -07:00
epriestley
3574a55a95 Deprecate Conduit method "diffusion.getrecentcommitsbypath"
Summary:
See D19558. This method has no callers and just wraps `diffusion.historyquery`, since D5960 (2013).

This was introduced in D315 (which didn't make it out of FB, I think) inside Facebook for unclear purposes in 2011.

Test Plan: Grepped for callers, found none.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: artms

Differential Revision: https://secure.phabricator.com/D19559
2018-08-03 09:48:58 -07:00
Arturas Moskvinas
356b2781bc Gracefully fail request if non existing callsign is passed to getrecentcommitsbypath instead of crashing
Summary:
`diffusion.getrecentcommitsbypath` fails with 500 error when non existing callsign is passed:
```
>>> UNRECOVERABLE FATAL ERROR <<<

Call to a member function getCommit() on null

```

Expected Behavior:
Return more graceful error notifying caller that such callsign/repository does not exist

Reproduction steps:
Open conduit: https://secure.phabricator.com/conduit/method/diffusion.getrecentcommitsbypath/
Enter:
callsign: "obviouslynotexisting"
path: "/random"
Click call method

Test Plan: after applying patch - call no longer fails with 500s

Reviewers: Pawka, epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19558
2018-08-02 19:49:10 +03:00
epriestley
e72296f927 Support querying Herald rules by monogram in typeahead datsources
Summary:
Depends on D19556. See PHI765. Ref T13164. Currently, if you type `H1` in this datasource, it isn't smart enough to pull up the right object.

Add support for querying by monogram. This is similar to existing support in Owners packages, etc.

Test Plan: Typed `H1` in the new push log filter, got the right object as a result in the typeahead.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19557
2018-08-01 17:52:27 -07:00
epriestley
06380e8079 Allow push events to be filtered by which Herald rule blocked the push
Summary: Depends on D19555. Ref T13164. See PHI765. An install is interested in getting a sense of the impact of a particular blocking rule, which seems reasonable. Support filtering for pushes blocked by a particular rule or set of rules.

Test Plan: {F5776385}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19556
2018-08-01 17:38:12 -07:00
epriestley
d8834377be When a Herald rule blocks a push, show which rule fired in the push log UI
Summary:
Ref T13164. See PHI765. We currently show "Rejected: Herald" in the push log UI, but don't show which rule rejected a push.

We store this data, and it's potentially useful: either for hunting down a particular issue, or for getting a general sense of how often a reject rule is triggering (maybe because you want to tune how aggressive it is).

Show this data in the web UI, and include it in the data export payload.

Test Plan:
  - Pushed to a hosted repository so that I got blocked by a Herald rule.
  - Viewed the push logs in the web UI, now saw which rule triggered things.
  - Exported logs to CSV, saw Herald rule PHIDs in the data.

{F5776211}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19555
2018-08-01 17:33:50 -07:00
epriestley
96e3c73159 Put "Subprojects" on top of "Milestones" in the Project UI
Summary:
Depends on D19550. Ref T13164. See T12144#226172, mostly. We get some requests to make milestones reorderable, but in most cases users probably wanted subprojects, not milestones.

One reason to end up here is that we put "Milestones" on top. Instead, put "Subprojects" on top, since they're the less specialized option and we aren't terribly consistent about it anyway.

Test Plan: Viewed project subprojects page, saw "Subprojects" above "Milestones".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19551
2018-08-01 13:49:42 -07:00
epriestley
45babe82f3 Add Spaces information to the project list UI
Summary: Depends on D19552. Ref T13164. We need this little `setObject(...)` hook to get the Space name into the search list UI.

Test Plan: Viewed project list, saw some Spaces listed.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19554
2018-07-31 10:24:51 -07:00
epriestley
8d8086fccf Add Spaces support to Phriction
Summary:
Ref T13164. See PHI774. Fixes T12435.

Since Phriction is hierarchical, there isn't a super strong motivation to support Spaces: you can generally set policies on a small number of documents to get the desired effective policy behavior.

However, it still improves consistency and there's no reason //not// to support Spaces. In the case where you have some moderately weird/complex policy on one or more Spaces, using Spaces to define the policy behavior can make things a bit simpler and easier to understand.

This probably doesn't actually fix whatever the root problem in T12435 was (complicated, non-hierarchical access policies?). See also a bunch of discussion in T12442. So we might end up going beyond this to address other use cases, but I think this is reasonable regardless.

Test Plan: Created and edited Phriction documents and shifted them between Spaces. Searched by Space, etc.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13164, T12435

Differential Revision: https://secure.phabricator.com/D19553
2018-07-31 10:24:28 -07:00
epriestley
d9b5b04950 Improve Space behavior for subprojects and milestones
Summary:
Depends on D19549. Ref T13164. See PHI774.

  - Make milestones inherit their parent project's space automatically, like they inherit their parent policies.
  - Make subprojects default to their parent project's space.

Test Plan: Created subprojects and milestones, got sensible default/effective Space behavior.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19550
2018-07-31 10:22:39 -07:00
epriestley
13cac5c362 Add Spaces to Projects
Summary:
See PHI774. Ref T13164. There is no reason projects //don't// support Spaces, just a vague concern that it's not hugely useful and might be a bit confusing.

However, it's at least somewhat useful (to improve consistency and reduce special casing) and doesn't necessarily seem more confusing than Projects are anyway. Support is trivial from a technical point of view, so just hook it up.

Test Plan: Created new projects, shifted projects between spaces. The support is all pretty much automatic.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19549
2018-07-31 10:15:41 -07:00
epriestley
9cf3b3bbf8 Count lines in build log slices more cheaply
Summary:
See PHI766. Ref T13164. Build log chunk processing does a `preg_split()` on slices, but this isn't terribly efficient.

We can get the same count more cheaply by just using `substr_count()` a few times.

(I also tried `preg_match_all()`, which was between the two in speed.)

Test Plan:
- Used `bin/harbormaster rebuild-log --id X --force` to rebuild logs. Verified that the linemap is identical before/after this change.
- Saw local time for the 18MB log in PHI766 drop from ~1.7s to ~900ms, and `preg_split()` drop out of the profiler (we're now spending the biggest chunk of time on `gzdeflate()`).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19545
2018-07-30 08:25:17 -07:00
epriestley
690a460c8e Allow mailers to be explicitly marked as inbound or outbound
Summary:
See PHI785. Ref T13164. In this case, an install wants to receive mail via Mailgun, but not configure it (DKIM + SPF) for outbound mail.

Allow individual mailers to be marked as not supporting inbound or outbound mail.

Test Plan:
  - Added and ran unit tests.
  - Went through some mail pathways locally, but I don't have every inbound/outbound configured so this isn't totally conclusive.
  - Hit `bin/mail send-test` with a no-outbound mailer.
  - I'll hold this until after the release cut so it can soak on `secure` for a bit.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19546
2018-07-30 08:25:06 -07:00
epriestley
9e451879d9 Add a "Changes Since Last Action" view to Differential revisions
Summary:
Ref T13151. See PHI616. Fixes T8163.

This adds `/D123/new/`, which shows the changes to the revision since the last timeline action you took.

It also adds a link to this view to diff update emails.

Test Plan:
  - Followed this link with a recent comment and no touches since update, ended up with sensible diff selections.
  - Updated revision, generated email, saw an appropriate link.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151, T8163

Differential Revision: https://secure.phabricator.com/D19541
2018-07-27 12:27:33 -07:00
epriestley
a5d3aea67c Carry the "silent" transaction flag through inverse edge edits
Summary:
See PHI751. Ref T13164. We added a "silent" flag for Editors somewhat recently (currently reachable only for bulk edits with `bin/bulk ...` command).

However, this flag doesn't carry through to the sub-editor when we make inverse edge edits. These are edits like "X is a parent of Y", which cause an implicit "Y is a child of X" edit to occur.

Pass the flag through.

Test Plan:
  - Rigged the relationships controller to make silent edits.
  - Changed the parents of a revision from the web UI. Saw no mail or feed stories.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19543
2018-07-27 12:27:16 -07:00
epriestley
cb99396c64 Make the "Is this JSON?" DocumentEngine heuristic a little tighter
Summary:
See PHI749. Ref T13164. We currently misdetect files starting with `[submodule ...` as JSON.

Make this a bit stricter:

  - If the file is short, just see if it's actually literally real JSON.
  - If the file is long, give up.

This should get the right result in pretty much all the cases people care about, I think. We could make the long-file guesser better some day.

Test Plan: Detected a `[submodule ...` file (no longer JSON) and a `{"duck": "quack"}` file (still JSON).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13164

Differential Revision: https://secure.phabricator.com/D19544
2018-07-27 12:27:02 -07:00
epriestley
727bc2234c Capitalize "OPcache" more consistently
Summary: Fixes T13174. PHP spells this "OPcache" (lowercase "c"); we're inconsistent. Be more consistent.

Test Plan:
  - `git grep OPCache`
  - `git grep -i opcache | grep -v opcache | grep -v OPcache`

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13174

Differential Revision: https://secure.phabricator.com/D19538
2018-07-26 12:15:57 -07:00
epriestley
9e1a1577c3 Make the meme cache case-sensitive
Summary:
Fixes T13172. At one point we always capitalized all the text, and the cache uses capitalized text.

However, we stopped capitalizing the text at some point. Modern memes are more more subtle than old memes, and when we eventualy add support for things like "explodey brain" we'll certainly want to support mixed case.

Practically, this stops you from changing the capitalization of a cached meme. Get rid of the cache transform.

Test Plan:
none lul

(I don't have `gd` installed locally and buiding it requires building libjpeg and libpng or giving up and using `brew`. I'l vet this in production.)

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13172

Differential Revision: https://secure.phabricator.com/D19537
2018-07-26 12:15:32 -07:00
epriestley
682c3bc9ee When migrating files between storage engines with "bin/files migrate ...", skip expired temporary files
Summary:
See T7148. This just cheats us out of a weird sort of race where we:

  - Dump an instance, including some `F123` which is a temporary file which expires in 3 minutes.
  - A few minutes later, the daemons delete the data for that file.
  - A few minutes after that, we try to `bin/files migrate --copy` to copy the data from S3 into the MySQL blob store.
  - This fails since the data is already gone.

Instead, just skip these files since they're already dead to us.

Test Plan: Faked this locally, will migrate the PHI769 instance on `aux001`.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19536
2018-07-26 06:22:23 -07:00
Kenneth Endfinger
6bdd74584e
Fix file encoding migration
Summary:
See [[ https://discourse.phabricator-community.org/t/file-encryption-corruption-when-trying-to-encode-existing-files/1605 | Discourse ]]

When migrating to aes-256-cbc, integrity hashes were not updated, so data was not properly

Test Plan:
I ran [[ https://gist.github.com/kaendfinger/3e0d78350af0ebe4e74b2c8a79707bae | this test script ]] to ensure it worked.
I created some files with lipsum, ensured that after encoding them with aes-256-cbc, they were not able to be cat'd.
After applying this patch and rerunning the script, it worked successfully.

Reviewers: epriestley, amckinley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Tags: #files, #storage

Differential Revision: https://secure.phabricator.com/D19533
2018-07-23 17:41:10 -05:00
Kenneth Endfinger
6b6e1f0ba8
Fix Lipsum generators for Differential Revisions and Pastes
Summary:
When generating test data to solve a bug I have encountered, I noticed Lipsum was not working correctly for Differential Revisions and Pastes.

It seemed like they weren't updated after some refactoring. This fixes that by updating them.

Test Plan: Run Lipsum for all objects, and note that it has much less failure.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D19534
2018-07-23 15:05:51 -05:00
epriestley
77d7bb7af0 Document the Ferret "=" operator and improve related documentation
Summary:
Depends on D19529. See PHI778.

  - Document the "name" constraint as deprecated. All callers are likely better served by the "query" constraint.
  - Guide users toward the "query" constraint a little better.
  - Document the `=` syntax.

Test Plan: Read various new documentation.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19531
2018-07-23 12:44:43 -07:00
epriestley
71d4fa41c9 Support the Ferret "=" (exact match) operator in the actual query engine
Summary:
Ref PHI778. In D18492, I added support for parsing this operator, but did not actually implement it in the query engine.

Implementation is fairly straightforward. This supports querying for objects by exact title with `title:="exact title"`. This is probably a bad idea, but sometimes maybe useful anyway.

Test Plan: Queried for `title:="xxx"`, found only exact matches.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: ahoffer2

Differential Revision: https://secure.phabricator.com/D19529
2018-07-23 12:44:00 -07:00
epriestley
dee453c94d Give Config the "" (SPARKLE LIKE NEW) emoji instead of "☺" (STUPID LOOKING FACE)
Summary: Fixes T13171. Open to suggestions but that face looks real, real dumb on High Sierra.

Test Plan: Visited Config, saw a serious professional emoji in the page title.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13171

Differential Revision: https://secure.phabricator.com/D19530
2018-07-23 12:43:25 -07:00
epriestley
dce6dd5d02 Add an explicit "null" to a missed diffusion.branchquery callsite to fix Diffusion "Branches" page
Summary:
See PHI775. See D19499. Originally, see PHI720.

D19499 broke the standalone "Branches" page for commits. Normally, you reach this by taking these steps:

  - View a commit which is contained by 11 or more branches.
  - Click the "More Branches..." link in the "Branches" field.
  - You should be taken to a list of all branches which contain the commit.

The change to the 'branch' parameter was adjusted in the query that builds the "x, y, z, More Branches..." list, but not on the actual "Branches" list with the full list. Adjust it.

Test Plan:
  - Set display limit to 1, viewed a commit on "master" and "stable", clicked "More Branches".
  - Before: saw only "master".
  - After: saw both "master" and "stable".

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19532
2018-07-23 11:21:11 -07:00
epriestley
eb80a5ede1 Make the Conduit auth error for an unrecognized public key a little more useful
Summary: Ref T13168. This is just a small quality-of-life fix: we can disclose which public key we're talking about because public keys are public.

Test Plan:
  - Hit public key error (through my own bumbling / not reading or following instructions). Specifically, I haven't associated the key with a device in Almanac.
  - Before: vague error.
  - After: more specific error with enough key material that I could grep for it.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim

Maniphest Tasks: T13168

Differential Revision: https://secure.phabricator.com/D19516
2018-07-20 09:43:54 -07:00
Austin McKinley
67283c7a45 Add test plan to differential.revision.search
Summary: Ref T13151. Ref PHI622.

Test Plan: Loaded a revision in the Conduit UI; observed presence of `testPlan` field.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19518
2018-07-20 08:40:27 -07:00
epriestley
185c28f307 Update parent/child revision timeline messages to use modern language ("parent revision")
Summary:
See PHI746. See also T11833, perhaps. Ref T13151.

Long ago, parent revisions were called "dependent revisions". This was changed to "parent revisions" in the action UI to improve clarity, but not changed in the timeline stories.

Update the timeline stories to use the same language the actions in the UI use.

Test Plan:
{F5732876}

{F5732877}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19514
2018-07-13 09:02:10 -07:00
epriestley
4214b56a4f Make the dashboard panel datasource work properly with hundreds of panels
Summary:
Ref T13151. See PHI727. Update the dashboard widget/panel datasource to actually query results using what the user typed.

The current approach is blind to what the user typed when pulling results from the database, and gets limited to an artificially small number of results somewhere in the pipeline.

Test Plan:
  - Queried for panels with text queries.
  - Queried for panels with `W123` queries.
  - This is substantially similar to the Owners datasource, which received a similar update in D17142 and has worked well since then.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19511
2018-06-28 08:54:29 -07:00
epriestley
a94528ee4a Expose Differential actions for "transaction.search" in a basic way
Summary:
See PHI725. Ref T13151. These actions are somewhat unusual and I considered different ways to represent them (make them look like "status" transactions; build multiple synthetic transactions) but ultimately landed on the simplest approach of just exposing them more or less as they exist internally.

I haven't included data for any of them. Most don't really have any data, but "accept" does. I'm holding off on providing more data until after T731, which may shake up the internal format.

Test Plan: Applied most of these transactions against a revision, queried for it with `transaction.search`, got distinguishable transactions out.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19509
2018-06-28 08:51:55 -07:00
epriestley
f94cee8628 Fix querying for transactions over "transaction.search" when the object does not support comments
Summary: See PHI725. Ref T13151. We currently try to load comments unconditionally, but not all objects (like projects) have comments. Only try to load comments if an object actually has comments.

Test Plan: Queried for an object with no comments, like project `#masonry`, via `transaction.search`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19507
2018-06-26 07:59:01 -07:00
epriestley
cac3dc4983 Give "create" transactions a readable type in "transaction.search"
Summary:
Ref T13151. See PHI725. By default, "transaction.search" doesn't provide details about transactions because many have bad/weird/policy-violating internal types or fields.

The "create" transaction is simple and straightforward, so label it to allow callers to distinguish it.

Test Plan:
  - Created a new task.
  - Called `transaction.search` on it.
  - Saw the labelled "create" transaction.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: swisspol

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19505
2018-06-22 17:42:35 -07:00
epriestley
d84f866ca0 When search indexers contend for a lock, just yield
Summary:
Depends on D19503. Ref T13151. See PHI719. If you have something like a script which updates an object in a loop, we can end up queueing many search reindex tasks.

These tasks may reasonably contend for the lock, especially if the object is larger (lots of text and/or lots of comments) and indexing takes a few seconds.

This isn't concerning, and the indexers should converge to good behavior quickly once the updates stop.

Today, they'll spew a bunch of serious-looking lock exceptions into the log. Instead, just yield so it's more clear that there's (normally) no cause for concern here.

Test Plan: Ran `bin/search index Txxx --force` on a large object in multiple windows with a 0 second lock, saw an explicit yield instead of a lock exception.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19504
2018-06-22 17:41:45 -07:00
epriestley
14e911a0d8 Index only the first 1,000 comments on any object
Summary:
Depends on D19502. Ref T13151. See PHI719. An install ended up with an object with 111,000+ comments on it because someone wrote a script to treat it like a logfile.

Although we seem to do mostly okay with this (locally, it only takes about 30s to index a similar object) we'll hit a wall somewhere (since we need to hold everything in memory), and it's hard to imagine a legitimate object with more than 1,000 comments. Just ignore comments past the first thousand.

(Conpherence threads may legitimately have more than 1,000 comments, but go through a different indexer.)

Test Plan:
  - Piped some comments into `maniphest.edit` in a loop to create a task with 100K comments.
  - Ran `bin/search index Txxx --force` to reindex it, with `--trace`.
    - Before: task indexed in about 30s.
    - After: script loaded comments with LIMIT 1000 and indexed in a couple seconds.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19503
2018-06-22 17:41:05 -07:00
epriestley
cbc71e75fa When queueing search index tasks, include the "objectPHID" in the task metadata
Summary:
Ref T13151. See PHI719. One minor hiccup in debugging the issue (which ended up being "revision has 100K comments") was that the `SearchWorker` did not show which object it was indexing.

Add `'objectPHID'` to the queue call so you can see which object is affected from the web UI.

Test Plan:
  - Stopped daemons.
  - Used `bin/search index D123 --background` to queue a search task.
  - Viewed task details in web UI from `/daemon/`.
    - Before change: no indication of which object was being indexed.
    - After change: page helpfully shows that the task is indexing D123.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19502
2018-06-22 17:40:32 -07:00
epriestley
b1f4a974fe Fix two minor breadcrumb issues in Config
Summary:
Fixes T13159. Two issues here:

  - When viewing a particular config setting, there's an extra "Config" crumb.
  - On the page for a config group, the link to the parent group has an extra "/config/" in it.

Test Plan:
  - Viewed a page for a particular setting, no longer saw an extra "Config" crumb.
  - Viewed a page for a setting group, clicked parent crumb, got taken to a real page.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13159

Differential Revision: https://secure.phabricator.com/D19501
2018-06-22 17:38:50 -07:00
epriestley
8ab8c390b7 If "branch" is provided to "diffusion.branchquery", use it as the "<pattern>" argument to "git branch --contains ..."
Summary:
Ref T13151. See PHI720. If you want to test if commit X appears on specific branch Y, `git branch --contains X -- Y` is faster than (effectively) `git branch --contains X | grep Y`.

Since this call has a "branch" parameter anyway, use it as the pattern argument if provided.

Test Plan:
  - Called the API method with no parameters, got all branches.
  - Called the API method with `master`, got just master.
  - Called the API method with `maste*`, got master. This behavior is not officially supported and may change in the future.
  - Viewed a commit, still saw all branches.
    - Grepped for `diffusion.branchquery` and verified that no remaining callsites pass a default "branch" parameter.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19499
2018-06-22 17:38:19 -07:00
epriestley
6136b83275 Fix changeset construction special case for empty commits in pre-commit hooks
Summary: Fixes T13155. Ref T13151. A recent change (D19455) changed the return format here, but I missed this special case for empty commits.

Test Plan:
  - T13155 has a good set of reproduction instructions.
  - Pushed an empty commit.
    - Before: bunch of warning log spew.
    - After: clean logs.

Reviewers: amckinley, avivey

Reviewed By: avivey

Maniphest Tasks: T13155, T13151

Differential Revision: https://secure.phabricator.com/D19500
2018-06-21 16:43:20 -07:00
Austin McKinley
9db5ad3476 Allow null identities to be attached to commit objects
Summary: I landed D19491 a little aggressively, so allow this field to be null until after the migration goes out.

Test Plan: Loaded commits without identity objects; did not get any errors.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19496
2018-06-20 08:35:36 -07:00
Austin McKinley
05f333dfba Attach identities to commits and users to identities
Summary: Ref T12164. Make it easier to work with identity objects by attaching them to commits and attaching users to identities.

Test Plan: Loaded some commits with `->needIdentities(true)` and checked the resulting objects.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19491
2018-06-18 15:31:41 -07:00
Austin McKinley
787c59744b Correctly attach users to identities
Summary: This never worked.

Test Plan: Ran `bin/repository rebuild-identities` and viewed identity objects with `currentEffectiveUserID`s and no longer got errors about attempting to attach `null` objects instead of `PhabricatorUser` objects.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19495
2018-06-18 15:21:11 -07:00
epriestley
a7c681b549 Don't set mail HTML bodies if there's no actual HTML body
Summary:
See <https://discourse.phabricator-community.org/t/commit-6011085b0fcd-breaks-sending-certain-email/1571>. Some mailers get upset if we `setHTMLBody(...)` with an empty string.

There's some possible argument they should be more graceful about this, but it's reasonably pretty ambiguous.

Only try to set the HTML body if we actually have a nonempty HTML body.

Test Plan:
  - Configured an "smtp" mailer.
  - Ran `echo hi | ./bin/mail send-test --to someone@somewhere.com --subject test`.
  - Before: error about empty message body.
  - After: no more message body error.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D19494
2018-06-15 14:01:40 -07:00
epriestley
1459fb3037 Make re-running rebuild-identities a bit faster and add a little progress information
Summary:
Ref T13151. Ref T12164. Two small tweaks:

  - If we aren't actually going to change anything, just skip the writes. This makes re-running/resuming a lot faster (~20x, locally).
  - Print when we touch a commit so there's some kind of visible status.

This is just a small quality-of-life tweak that I wrote anyway while investigating T13152, and will make finishing off db024, db025 and db010 manually a little easier.

Test Plan:
  - Set `authorIdentityPHID` + `committerIdentityPHID` to `NULL`.
  - Ran `rebuild-identities`, saw status information.
  - Ran `rebuild-identiites` again, saw it go faster with status information.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151, T12164

Differential Revision: https://secure.phabricator.com/D19484
2018-06-12 13:18:54 -07:00
epriestley
6011085b0f Respect "metamta.email-body-limit" when building mail HTML bodies
Summary:
Ref T13151. See T11767. See PHI686. Although we limit outbound mail text bodies, the limit doesn't currently apply to attachments, HTML bodies, or headers. T11767 discusses improving this in the general case.

In the wild, an install hit an issue (see PHI686) where edits to Phriction pages generate very large HTML bodies. Check and respect the limit when building HTML bodies.

If we don't have enough room for the HTML body, we just drop it. We have the text body to fall back to, and HTML is difficult to truncate safely.

Test Plan: Added unit tests and made them pass.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19489
2018-06-12 12:02:15 -07:00
epriestley
c5b13a6be3 Allow object subtypes to be changed via bulk editor
Summary:
Ref T13151. See PHI683. Ref T12314.

You can currently change object subtypes via Conduit (`maniphest.edit`) but not via the web UI.

Changing object subtypes is inherently a somewhat-perilous operation that likely has a lot of rough edges we'll need to smooth over eventually, mostly around changing an object from subtype X to subtype Y, where some field exists on one but not the other. This isn't a huge issue, just not entirely intuitive.

It should also, in theory, be fairly rare.

As a reasonable middle ground, provide web UI access via the bulk editor. This makes it possible, but doesn't clutter the UI up with a rarely-used option with rough edges.

Test Plan:
  - With subtypes not configured, saw a normal bulk editor with no new option.
  - With subtypes configured, swapped tasks subtypes via bulk editor.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151, T12314

Differential Revision: https://secure.phabricator.com/D19490
2018-06-12 11:58:44 -07:00
epriestley
62a402491a Allow encrypted mail to be more specific about which object is affected
Summary:
Depends on D19487. Ref T13151. See PHI647. For some objects, like revisions, we can build slightly more useful secure email without actually disclosing anything.

In the general case, the object monogram may disclose information (`#acquire-competitor`) but most do not, so applications can whitelist an acceptable nondisclosing subject and link.

Support doing this, and make Differential do it. When we don't have a whitelisted URI but do know the object the mail is about, include a generic PHID-based URI; these are always nondisclosing.

Test Plan:
  - Without the Differential changes, sent normal mail (no changes) and secure mail (new generic PHID-based link).
  - With the Differential changes, sent secure mail; got richer subject and body link.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19488
2018-06-12 11:55:18 -07:00
epriestley
94752278f4 Add a generic PHID-based object redirection controller
Summary:
Ref T13151. See PHI647. This allows us to link to any object by PHID, without disclosing information in the monogram (like `#fire-steve`).

This capability is relevant when building "secure mail", to provide a link to the object regardless of whether the monogram discloses information or not.

Test Plan: Visited `/object/D123/` (redirect), `/object/xyz/` (404), `/object/PHID-DREV-.../` (redirect).

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19487
2018-06-12 11:54:59 -07:00
epriestley
cbff913432 Add a "members of all projects" (vs "...any project") custom policy rule to the upstream
Summary:
Ref T13151. See PHI702. An install is interested in a "members of all projects" (vs "members of any project", which is currently implemented) rule.

Although this is fairly niche, I think it's reasonable and doesn't have much of a maintenance cost.

This could already be implemented as an extension, but it would have to copy/paste a bunch of code.

Test Plan:
  - Ran unit tests.
  - Used the UI to select this policy for a task, with various values. Joined/left projects to satisfy/fail the rule. Behavior seemed correct.
  - Used the UI to select the existing policy rule ("any project"), joined/left projects to satisfy/fail the rule. Doesn't look like I broke anything.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19486
2018-06-12 11:51:51 -07:00
Alex Vandiver
59b95f9397 Fix typo in "button"
Test Plan: Observation.

Reviewers: #blessed_reviewers, amckinley, epriestley

Reviewed By: #blessed_reviewers, amckinley, epriestley

Subscribers: Korvin, amckinley, epriestley

Differential Revision: https://secure.phabricator.com/D19483
2018-06-08 15:09:07 -07:00
epriestley
f375427177 Use more consistent diff coloration in unified diffs
Summary:
Ref T13151. See PHI701. Unified diffs are currently missing the logic to apply the "old-full" and "new-full" classes, which results in a too-light coloration for fully added or removed lines.

Make this logic consistent with the two-up renderer so we use the same colors in both.

Test Plan: Viewed diffs and swapped between 1-up and 2-up renderers, now saw the same coloration.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19482
2018-06-08 09:39:34 -07:00
epriestley
7acda7e94e Truncate package names in diff table of contents views
Summary:
Ref T13151. See PHI654. Depends on D19477. If you have long package names, the table of contents (e.g., in Differential) can end up expanding to be gigantic.

Getting tables to behave nicely is hard (or, at least, I can't figure it out after spending a decent amount of time on it; see also `AphrontTableView::renderSingleDisplayLine()`). I tried a bunch of things and Googled for a bit but didn't make any progress on finding a CSS solution. Just truncate the package names to get reasonable behavior without falling down any kind of CSS rabbit hole.

Test Plan:
  - Created a package named "Very long package name...".
  - Created a package named "MMMMMMMMMMMMMMMMMMMMMM...".
  - Had them own a file in a Differential revision, viewed that revision.
  - Before: table is pushed out to several times the browser window width and everything is kind of a mess.
  - After: package names get truncated to something reasonable.

{F5652953}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19478
2018-06-07 13:17:01 -07:00
epriestley
2951e0c86b Include owners packages in the MailableFunction datasource
Summary:
Ref T13151. See PHI684. Currently, the `MailableFunction` datasource does not include Owners packages, but they are valid subscribers and the `Mailable` datasource includes them.

Include them in the `MailableFunction` datasource, too.

Test Plan: Searched for revisions with particular package subscribers, got expected results in the UI (tokenizer knew about packages) and response.

Reviewers: amckinley, jmeador

Reviewed By: jmeador

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19476
2018-06-07 12:02:50 -07:00
Austin McKinley
b8b2d1672d Prevent creation of empty repository identities
Summary: Fixes issue reported in https://secure.phabricator.com/rPf191a66490b194785fae28c062b71be99bb14584#43240

Test Plan: Imported an SVN repo, observed clean import instead of daemon exception.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D19466
2018-06-05 16:13:59 -07:00
Aviv Eyal
dbe72df557 minor: fix translation error in exception
Test Plan: look hard at code.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D19463
2018-06-04 17:46:13 +00:00
epriestley
376ea1ddf5 Support logged-out access to more Harbormaster controllers
Summary:
Fixes T13145. The list controllers properly support public access already, but some of the view/detail controllers did not.

Allow logged-out users to browse builds, buildables, plans, etc., provided they can see the corresponding objects.

Test Plan: As a logged-out user, browsed around builds, build plans, logs, etc., without hitting any login pages.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13145

Differential Revision: https://secure.phabricator.com/D19459
2018-06-04 10:11:25 -07:00
epriestley
5bcca675e1 Add card expiration information to Phortune cart screen
Summary:
See PHI689. It can be difficult to distinguish between cards with the same number but different expiration dates (common when the bank sends you a new card).

For now, show the expiration date on the cart checkout screen.

Test Plan: Viewed a cart checkout screen with multiple cards, saw expiration dates.

Reviewers: amckinley

Differential Revision: https://secure.phabricator.com/D19462
2018-06-02 18:23:44 -07:00
Austin McKinley
2f6784ee1c Add workflow to create repository identities
Summary:
Depends on D19443. Creates a workflow for populating the new identity table by iterating over commits, either one repo at a time or all at once. Locally caches identities to avoid fetching them `inf` times. An actual migration that invokes this workflow will come in another revision that won't land until at least next week.

Performance is ~2k commits in 4.9s on my local machine.

Test Plan: Ran locally a few times with a few different states of the `repository_identity` table.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: jcox, Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D19446
2018-05-31 07:29:57 -07:00
Austin McKinley
fe5fde5910 Assign RepositoryIdentity objects to commits
Summary: Depends on D19429. Depends on D19423. Ref T12164. This creates new columns `authorIdentityPHID` and `committerIdentityPHID` on commit objects and starts populating them. Also adds the ability to explicitly set an Identity's assignee to "unassigned()" to null out an incorrect auto-assign. Adds more search functionality to identities. Also creates a daemon task for handling users adding new email address and attempts to associate unclaimed identities.

Test Plan: Imported some repos, watched new columns get populated. Added a new email address for a previous commit, saw daemon job run and assign the identity to the new user. Searched for identities in various and sundry ways.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19443
2018-05-31 07:28:23 -07:00
Austin McKinley
f191a66490 Add controllers/search/edit engine functionality to RepositoryIdentity
Summary: Depends on D19423. Ref T12164. Adds controllers capable of listing and editing `PhabricatorRepositoryIdentity` objects. Starts creating those objects when commits are parsed.

Test Plan: Reparsed some revisions, observed objects getting created in the database. Altered some `Identity` objects using the controllers and observed effects in the database. No attempts made to validate behavior under "challenging" author/committer strings.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19429
2018-05-31 07:03:25 -07:00
Austin McKinley
cd84e53c44 Begin building out RepositoryIdentity indirection layer
Summary: Ref T12164. Start building initial objects for managing `RepositoryIdentity` objects. This won't land until much more of the infrastructure is in place.

Test Plan: Ran `bin/storage upgrade` and observed expected table.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19423
2018-05-31 07:01:16 -07:00
epriestley
de999af614 Improve some behaviors around memory pressure when pushing many and/or large changes
Summary:
Ref T13142. When commits are pushed, we try to handle them on one of two pathways:

  - Normal changes: we load these into memory and potentially apply Herald content rules to them.
  - "Enormous" changes: we don't load these into memory and skip content rules for them.

The goal is to degrade gracefully when users push huge changes: they should work, just not support all the features.

However, some changes can slip through the cracks right now:

  - If you push a lot of commits at once, we'll try to cache all of the changes smaller than 1GB in memory. This can require an arbitrarily large amount of RAM.
  - We calculate sizes by just looking at the `strlen()` of the diff, but a changeset takes more RAM in PHP than the raw diff does. So even if a diff is "only" 500MB, it can take much more memory than that. On systems with relatively little memory available, this may result in OOM while processing changes that are close to the "enormous" limit.

This change makes two improvements:

  - Instead of caching everything, cache only 64MB of things.
    - For most pushes, this is the same, since they have less than 64MB of diffs.
    - For pushes of single very large changes, this is a bit slower (more CPU) since we have to do some work twice.
    - For pushes of many changes, this is slower (more CPU) since we have to do some work twice, but, critically, doesn't require unlimited memory.
  - Instead of flagging changes as "enormous" at 1GB, flag them as "enormous" at 256MB.
    - This reduces how much memory is required to process the largest "non-enormous" changes.
    - This also gets us under Git's hard-coded 512MB "always binary" cutoff; see T13143.
    - This is still completely gigantic and way larger than any normal change should be.

An additional improvement would be to try to reduce the amount of memory we need to use to hold a change in process memory. I think the other changes here alone will fix the immediate issue in PHI657, but it would be nice if the "largest non-enormous change" required only a couple gigs of RAM.

Test Plan:
- Used `ini_set('memory_limit', '1G')` to artificially limit memory to 1GB.
- Pushed a series of two commits which add two 550MB text files (Temporarily, I added a `--binary` flag to trick Git into showing real diffs for these, see T13143.)
- Got a memory limit error.
- Applied the "cache only 64MB of stuff" and "consider 256MB, not 1GB, to be enormous" changes.
- Pushed again, got properly rejected as enormous.
- Added `memory_get_usage()` calls to measure how actual memory size and reported "size" estimate compare. For these changes, saw a 639MB diff require 31,479MB of memory, i.e. a factor of about 50x. This is, uh, pretty not great.
- Allowed enormous changes, pushed again, push went through.

Reviewers: amckinley

Maniphest Tasks: T13142

Differential Revision: https://secure.phabricator.com/D19455
2018-05-18 17:15:34 -07:00
epriestley
8f9b948447 When showing a diff-of-diffs, hide files which didn't get any more changes and have no inlines
Summary:
Ref T13137. See that task for discussion.

When we show a diff-of-diffs, we often render stubs for files which didn't change between the diffs. These stubs usually aren't a big deal, but for certain types of changes (like refactors) they can create a lot of clutter.

Instead, hide these stubs and show a notice that we hid them.

Test Plan:
  - Created a revision affecting 4 files.
  - Updated it with a diff that changed only 1 of the 4 files.
  - Added an inline comment to a different file.
  - Viewed the diff of diffs.
    - Before: 4 changesets with two "nothing changed" stubs.
    - After: 2 changesets with the stubs hidden.

{F5621083}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13137

Differential Revision: https://secure.phabricator.com/D19453
2018-05-16 17:18:53 -07:00
epriestley
79fdf5c127 Separate changeset analysis code from DifferentialDiff and provide a standalone rebuild-changesets workflow
Summary:
Ref T13137. The "analyze/cache data about changesets" step is becoming more involved. We recently added detection for generated code to support "Ignore generated changes" in Owners, and I now plan to hash the new file content so we can hide changes which have no effect.

Before adding this new hashing step, pull the "detect copied code" and "detect generated code" stuff out and move them to a separate `ChangesetEngine`. Then support doing a changeset rebuild directly with `bin/differential rebuild-changesets`.

This simplifies things a bit and makes testing easier since you don't need to keep creating new revisions to re-run copy/generated/hash logic.

Test Plan: Ran `bin/differential rebuild-changesets --revision Dxxx`, saw changesets rebuild. See also next change.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13137

Differential Revision: https://secure.phabricator.com/D19452
2018-05-16 17:17:28 -07:00