1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-24 06:20:56 +01:00
Commit graph

15309 commits

Author SHA1 Message Date
epriestley
40d2346f29 Add a missing "null" check when rebuilding old/new diff content
Summary:
See <https://discourse.phabricator-community.org/t/exceptions-when-viewing-diffs/3787>. This list may include `null` values.

Until PHP 7.4, `$x = null; echo $x['y'];` does not emit a warning. Sneaky!

Test Plan:
  - Traced `null` values from `reparseHunksForSpecialAttributes()`, saw them no longer incorporated into corpus bodies.
  - This has some amount of test coverage.

Differential Revision: https://secure.phabricator.com/D21169
2020-04-25 09:23:05 -07:00
epriestley
454ecb56e3 When proxying HTTP repository responses from repository nodes, discard content description headers
Summary:
Ref T13517. See that task for details about the underlying issue here.

Currently, we may decode a compressed response, then retransmit it with leftover "Content-Encoding" and "Content-Length" headers. Instead, strip these headers.

Test Plan:
  - In a clustered repository setup, cloned a Git repository over HTTP.
  - Before: Error while processing content unencoding: invalid stored block lengths
  - After: Clean clone.

Maniphest Tasks: T13517

Differential Revision: https://secure.phabricator.com/D21167
2020-04-25 07:51:46 -07:00
epriestley
6f7147376f Parse "multipart/form-data" bodies even if "enable_post_data_reading" is on
Summary:
Ref T4369. During T13507, I set my "max_post_size" to a very small value, like 7 (i.e., 7 bytes). This essentially disables "enable_post_data_reading" even if the setting is technically on.

This breaks forms which use "multipart/form-data", which are rare but not nonexistent. Notably, forms in Config use this setting (because of `ui.header` stuff?) although perhaps they should not or no longer need to.

This can be fixed by parsing the raw input.

Since the only reason we don't parse the raw input is concern that we may not be able to read it (per documentation, but never actually observed), and we do a `strlen()` test anyway, just read it unconditionally.

This should fix cases where POST data wasn't read because of "max_post_size" without impacting anything else.

Test Plan: With very small "max_post_size", updated "ui.footer-items" in Config. Before: form acted as a no-op. After: form submitted.

Maniphest Tasks: T4369

Differential Revision: https://secure.phabricator.com/D21165
2020-04-24 11:25:57 -07:00
epriestley
5a460e4ea5 Stick the page footer in the right place on Formation View pages
Summary: Ref T13516. This isn't terribly clean, but get the page footer into the bottom of the content page on FormationView pages so it doesn't overlap into the side panel.

Test Plan: With and without a footer, viewed normal and FormationView pages. Saw footers in appropriate places at appropriate times.

Maniphest Tasks: T13516

Differential Revision: https://secure.phabricator.com/D21166
2020-04-24 11:25:31 -07:00
epriestley
d05d8f6558 Add a very forgiving GC for Differential viewstate information
Summary:
Ref T13455. Viewstates are fairly small and will probably grow less quickly than the changeset table, but the data is also not important to retain in the long term: if you revisit a change several months after hiding some files, it's fine if we've forgotten that you adjusted the view parameters.

Add a GC with a long default collection policy (180 days) so installs can manage the size of this table if it becomes necessary.

Test Plan: Ran via `bin/garbage` to adjust the GC policy and collect viewstates.

Maniphest Tasks: T13455

Differential Revision: https://secure.phabricator.com/D21164
2020-04-23 14:17:48 -07:00
epriestley
4793bfcb7c Don't show the "file tree" view on tablets/phones
Summary: Ref T13516. Hide this UI on devices without the screen width to reasonably support it.

Test Plan: Viewed a revision at various window widths, saw the elements vanish at device widths and reappear at desktop widths.

Maniphest Tasks: T13516

Differential Revision: https://secure.phabricator.com/D21162
2020-04-23 13:40:42 -07:00
epriestley
d2572f8b33 Refine more Differential review state behaviors
Summary:
Ref T13516.

- Add an "Add Comment" navigation anchor.
- Make selection state more clear.
- Make hidden state tidier and more clear.
- Hide "View Options" in the hidden state to dodge all the weird behaviors it implies.
- Click to select/deselect changesets.
- When you open the view dropdown menu, then press "h", close the dropdown menu.

Test Plan: Fiddled with all these behaviors.

Maniphest Tasks: T13516

Differential Revision: https://secure.phabricator.com/D21161
2020-04-23 10:14:52 -07:00
epriestley
0ede616f31 Update the "View Options" menu for recent filetree changes
Summary:
Ref T13516. Minor improvements here:

  - Show key commands in the "View Options" dropdown.
  - Organize it slightly better.
  - Improve disabled item behaviors a little bit.
  - Add a "Browse Directory" action.
  - Rename "...in Diffusion" to "...in Repository".
  - Make "d", "D", and "h" use the same targeting rules as "\".
  - When you hide a file with the "h" menu item, select it.

Test Plan: Poked at the menu a lot, ran into less questionable behavior.

Maniphest Tasks: T13516

Differential Revision: https://secure.phabricator.com/D21160
2020-04-23 08:23:12 -07:00
epriestley
befbea2f00 Don't pass "No newline at end of file." annotations to DocumentEngines as literal diff text
Summary: See PHI1707, which has a Jupyter notebook which fails to diff nicely when modified. The root cause seems to be that the document does not end in a newline.

Test Plan: Applied patch, diffed the file, got a Jupyter diff out of it.

Differential Revision: https://secure.phabricator.com/D21159
2020-04-22 20:22:33 -07:00
epriestley
60de1506fe Make "hidden" changesets sticky, and show hidden state in the filetree
Summary:
Ref T13455. Make "hidden" a changeset property similar to other changeset properties.

We don't need to render this on the server, so we make a request (to update the setting) and just discard the response.

Test Plan: {F7375468}

Maniphest Tasks: T13455

Differential Revision: https://secure.phabricator.com/D21158
2020-04-22 16:12:42 -07:00
epriestley
a72a66caa8 Mark "low importance" and "owned" changes in the filetree
Summary: Ref T13516. Mark low-importance changes (generated code, deleted files) and owned-with-authority changes in the filetree.

Test Plan: {F7375327}

Maniphest Tasks: T13516

Differential Revision: https://secure.phabricator.com/D21157
2020-04-22 11:22:34 -07:00
epriestley
ff88eb588e Show change information in file icons in the filetree
Summary: Ref T13516. Restores "deleted"/"added" information to the tree icons.

Test Plan: {F7375145}

Maniphest Tasks: T13516

Differential Revision: https://secure.phabricator.com/D21156
2020-04-22 08:38:29 -07:00
epriestley
12eddb18fb Entirely replace the old filetree UI with the "flank" UI
Summary:
Ref T13516. Deletes all old filetree / flex / active / collapse nav code in favor of the new code.

Restores the inline tips in the path tree.

Test Plan: {F7374175}

Maniphest Tasks: T13516

Differential Revision: https://secure.phabricator.com/D21154
2020-04-22 08:32:02 -07:00
epriestley
ba8071bbef Roughly style the new "flank" paths UI
Summary: Ref T13516. Apply basic UI styling to the new UI and make some more interaction work.

Test Plan: {F7374096}

Maniphest Tasks: T13516

Differential Revision: https://secure.phabricator.com/D21153
2020-04-22 08:31:40 -07:00
epriestley
8cd1f9a309 Generate file trees from changesets in the new flank UI
Summary: Ref T13516. Generate a tree structure based on the page changesets. Still missing styles and a whole lot of behavior.

Test Plan: {F7373967}

Maniphest Tasks: T13516

Differential Revision: https://secure.phabricator.com/D21152
2020-04-22 08:31:17 -07:00
epriestley
646280972b Glue the new FormationView on top of the older Filetree view in Differential
Summary: Ref T13516. This glues "FormationView" to "ChangesetList". The actual tree is not functional in any meaningful way yet.

Test Plan: {F7373838}

Maniphest Tasks: T13516

Differential Revision: https://secure.phabricator.com/D21151
2020-04-22 08:29:04 -07:00
epriestley
fef2cdabfe Add a "FormationView" to support dynamic flank panels
Summary:
Ref T13516. Currently, the "File Tree" element is a semi-dynamic side panel that's implemented as a special mode of a side nav panel.

This implementation is fairly clunky, and arose from organic growth out of the side nav. As such, it has some weird behaviors, doesn't have builtin support for show/hide, and can't generalize easily.

Introduce a "FormationView" which supports loading a page up with piles of side panels in various modes.

Test Plan: No callers and no user-visible impact.

Maniphest Tasks: T13516

Differential Revision: https://secure.phabricator.com/D21150
2020-04-22 08:23:21 -07:00
epriestley
ef69c7969f Restore editor behavior to Diffusion and support "\" shortcut
Summary:
Ref T13515. This restores the "Open in Editor" behavior to Diffusion, and makes "\" work there.

The URI pattern is now sent as a structured template to the client, so the code will work properly if a file path contains "%l".

Test Plan:
  - Clicked "Open in Editor" and pressed "\" in Diffusion when viewing a file.
  - Clicked a line, hit "\", got the file opened to that line.

Maniphest Tasks: T13515

Differential Revision: https://secure.phabricator.com/D21149
2020-04-19 09:41:37 -07:00
epriestley
f02024615a Add "short name", "id", and "phid" variables for external editor URIs
Summary: Ref T13515. External editor URIs currently depend on repositories having callsigns, but callsigns are no longer required. Add some variables to support configuring this feature for repositories that do not have callsigns.

Test Plan: Changed settings to use new variables, saw links generate appropriately.

Maniphest Tasks: T13515

Differential Revision: https://secure.phabricator.com/D21147
2020-04-19 09:37:53 -07:00
epriestley
c79094d7a8 Add static errors, supported protocols, and a dynamic function listing to external editor settings page
Summary:
Ref T13515.

  - Previously valid editor URIs may become invalid without being changed (if an administrator removes a protocol from the list, for example), but this isn't explained very well. Show an error on the settings page if the current value isn't usable.
  - Generate a list of functions from an authority in the parser.
  - Generate a list of protocols from configuration.

Test Plan: {F7370872}

Maniphest Tasks: T13515

Differential Revision: https://secure.phabricator.com/D21146
2020-04-19 09:37:31 -07:00
epriestley
7a79131bf2 Replace old hard-coded URI-based "changes saved" jank with new overgeneralized cookie-based "changes saved" jank
Summary:
Ref T13515. Settings currently has some highly specialized code for rendering "Changes saved." messages. The "saved" state is communicated across a redirect-after-POST by adding `/saved/` to the end of the URI.

This isn't great. It needs a lot of moving pieces, including special accommodations in routing rules. It's user-visible. It has the wrong behavior if you reload the page or navigate directly to the "saved" URI.

Try this scheme, which is also pretty sketchy but seems like an upgrade on the balance:

  - Set a cookie on the redirect which identifies the form we just saved.
  - On page startup: if this cookie exists, save the value and clear it.
  - If the current page started with a cookie identifying the form on the page, treat the page as a "saved" page.

This supports passing a small amount of state across the redirect-after-POST flow, and when you reload the page it doesn't keep the message around. Applications don't need to coordinate it, either. Seems somewhat cleaner?

Test Plan: In Firefox, Safari, and Chrome: saved settings, saw a "Saved changes" banner without any URI junk. Reloaded page, saw banner vanish properly.

Maniphest Tasks: T13515

Differential Revision: https://secure.phabricator.com/D21144
2020-04-19 09:04:31 -07:00
epriestley
3984c14260 Tokenize external editor links so they can be safely materialized on the client
Summary:
Ref T13515. Currently, opening a file to a particular line in an external editor relies on replacing "%l" with "%l" (which is escaped as "%25l") on the server, and then replacing "%25l" with the line number on the client. This will fail if the file path (or any other variable) contains "%l" in its unencoded form.

The parser also can't identify invalid variables.

Pull the parser out, formalize it, and make it generate an intermediate representation which can be sent to the client and reconstituted.

(This temporarily breaks Diffusion and permanently removes the weird, ancient integration in Dark Console.)

Test Plan:
  - Added a bunch of tests for the actual parser.
  - Used "Open in Editor" in Differential.

Maniphest Tasks: T13515

Differential Revision: https://secure.phabricator.com/D21143
2020-04-19 09:02:49 -07:00
epriestley
4168335619 Remove the "Edit Multiple Files" external editor setting
Summary: Ref T13515. No callsites actually use this, most editors don't support it, it doesn't seem terribly useful for the ones that do, it makes template-based APIs for line-number substitution complicated, and we can probably just loop on `window.open()` anyway.

Test Plan: Grepped for affected symbols, found no more references. Loaded settings page, saw no more setting.

Maniphest Tasks: T13515

Differential Revision: https://secure.phabricator.com/D21142
2020-04-19 09:02:03 -07:00
epriestley
8bdc713352 Make the "Keyboard Shortcuts" dialog in Differential less hideous
Summary:
Ref T13515. Adding "\" ("Open in External Editor") made this slighlty worse, but it was already pretty bad.

Long ago the keys had a special style on them, but this got changed and dropped somewhere around D16568 -- although at the time, I think they still had a grey background (see T11654).

Some later change removed this background.

Put the background back and separate the keystrokes into groups.

Test Plan: {F7370615}

Maniphest Tasks: T13515

Differential Revision: https://secure.phabricator.com/D21141
2020-04-19 09:01:07 -07:00
epriestley
84cd4a3854 Move "External Editor" settings to a separate settings group
Summary:
Ref T13515. It's not intuitive that these settings are "Display Preferences", even thought they're intenrally related to some of the other display preferences.

Give them a separate group.

Test Plan: {F7370500}

Maniphest Tasks: T13515

Differential Revision: https://secure.phabricator.com/D21140
2020-04-19 08:59:43 -07:00
epriestley
c3c55d82ae Make "renderer", "engine", and "encoding" sticky across reloads in Differential and Diffusion
Summary:
Ref T13455. Update the other "view state" properties to work like "highlight" now works.

Some complexity here arises from these concerns:

  - In "View Standalone", we render the changeset inline. This is useful for debugging/development, and desirable to retain.
  - In all other cases, we render the changeset with AJAX.

So the client needs to be able to learn about the "state" properties of the changeset on two different flows. Prior to this change, each pathway had a fair amount of unique code.

Then, some bookkeeping issues:

  - At inital rendering time, we may not know which renderer will be selected: it may be based on the client viewport dimensions.
  - Prior to this change, the client didn't separate "value of the property for the changeset as rendered" and "desired value of the property".

Test Plan:
  - Viewed changes in Differential, Diffusion, and in standalone mode.
  - Toggled renderer, character sets, and document engine (this one isn't terribly useful). Reloaded, saw them stick.
  - Started typing a comment, cancelled it, hit the undo UI.

Maniphest Tasks: T13455

Differential Revision: https://secure.phabricator.com/D21138
2020-04-19 08:59:09 -07:00
epriestley
8aac55cc57 Make "Highlight As..." sticky across reloads in Diffusion and Differential
Summary:
Ref T13455. Add container-level storage for persistent view state, and persist "Highlight As..." inside it.

The storage generates a "PhabricatorChangesetViewState" configuration object as an output.

When preferences are expressed on a diff and that diff is later attached to a revision, we attempt to copy the preferences.

The internal storage tracks per-changeset settings, but currently always uses "last update wins" to apply the settings in the UI.

Test Plan:
  - Viewed revisions, changed highlighting, reloaded. Saw highlighting stick in revision view and standalone view.
  - Viewed commits, changed highlighting, reloaded. Saw highlighting stick.
  - Created a diff, changed highlighting, turned it into a revision, saw highlighting persist.

Subscribers: jmeador, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13455

Differential Revision: https://secure.phabricator.com/D21137
2020-04-19 08:58:39 -07:00
epriestley
3adf082002 When inlines would disable a file shield in a diff, still apply the shield if all the comments are collapsed
Summary:
Ref T13515. We "shield" some changesets, including generated code and intradiffs with no intermediate changes.

These files don't get shielded if they have inline comments.

But, if the viewer has collapsed all the comments, we can shield the file again.

Test Plan:
  - Created a change affecting files A and B, with three diffs:
    - Touch A and B.
    - Touch B only.
    - Touch nothing.
  - Added an inline to A and collapsed it.
  - Viewed Diff 1 vs Diff 2:
    - Saw A collapse with a note about inlines.
    - Saw B changes, normally.
  - Viewed Diff 2 vs Diff 3:
    - Saw A collapse with a note about inlines.
    - Saw B collapse normally.
  - Uncollapsed the inline, viewed 1v2 and 2v3, saw A expand in both cases.

Maniphest Tasks: T13515

Differential Revision: https://secure.phabricator.com/D21136
2020-04-17 10:52:40 -07:00
epriestley
3d966d8a41 Add an "Open in External Editor" keystroke to Differential
Summary: Ref T13515. See PHI1661. If a file is selected, add a keystroke to click the "Open in External Editor" link.

Test Plan: In Safari, Chrome, and Firefox: used "J" to select a file, then "\" to open it in an external editor. (In Safari and Chrome, this prompts.)

Maniphest Tasks: T13515

Differential Revision: https://secure.phabricator.com/D21135
2020-04-17 10:06:46 -07:00
epriestley
83eb7447a1 When Owners packages are archived, annotate them in tokenizer results
Summary: Fixes T13512. Archived packages in Owners are missing hinting, but should have it.

Test Plan:
Before:

{F7369122}

After:

{F7369128}

Maniphest Tasks: T13512

Differential Revision: https://secure.phabricator.com/D21134
2020-04-17 06:18:04 -07:00
Austin McKinley
ef1340bd32 Add Ferret support to Paste
Summary:
Ref PHI1292. Enable fulltext searchs in paste. Maybe this should only index a snippet instead of the entire content?

Also updates table names in `PhabricatorPasteQuery`.

Test Plan: Created some pastes, indexed them, searched for them.

Reviewers: amckinley

Subscribers: codeblock, Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D20650
2020-04-16 14:10:23 -07:00
epriestley
2748f83e12 Modularize Ferret fulltext functions
Summary: Ref T13511. Currently, Ferret fulltext field functions (like "title:") are hard-coded. Modularize them so extensions may define new ones.

Test Plan: Added a new custom field which emits data for the indexer, searched for "animal-noises:moo", "animal-noises:-", etc., in global search and application search.

Maniphest Tasks: T13511

Differential Revision: https://secure.phabricator.com/D21131
2020-04-16 13:41:13 -07:00
epriestley
894d9b6587 Remove Ferret function aliases and overrides
Summary:
Ref T13511. Ferret functions currently define "aliases", and some applications override the default aliases.

This probably isn't really the right model, since it means the available function aliases in global search depend on the types of documents you're searching for. This isn't fundamentally unworkable but is kind of weird.

Regardless, these don't actually work. Searching for "description:x" is a syntax error.

Since they don't work, it's a good bet no one is relying on them. Just get rid of them until there's a clearer argument for the feature.

Test Plan: Grepped for "getFunctionMap", got no other hits. Ran some queries with the alias functions, got syntax errors.

Maniphest Tasks: T13511

Differential Revision: https://secure.phabricator.com/D21130
2020-04-16 13:40:17 -07:00
epriestley
9bdf477f2f Combine the two different ngram-splitting algorithms into a single engine
Summary:
Ref T13501. Depends on D21127. With the "prefix" behavior removed in D21127, we now have two virtually identical copies of the same code.

The newer one in Ferret is better: it slices utf8 correctly and is slightly more efficient on large inputs. Pull it out and make all callers call into it.

Test Plan:
  - Grepped for all affected symbols.
  - Ran `bin/search index --force ...` to reindex various objects (tasks, files).
  - Searched for things in the UI.

Maniphest Tasks: T13501

Differential Revision: https://secure.phabricator.com/D21128
2020-04-16 09:45:00 -07:00
epriestley
fb3f423279 Remove broken and unfixable "prefix" ngram behavior
Summary:
Ref T13501. The older ngram code has some "prefix" behavior that tries to handle cases where a user issues a very short (one or two character) query.

This code doesn't work, presumably never worked, and can not be made to work (or, at least, I don't see a way, and am fairly sure one does not exist).

If the user searches for "xy", we can find trigrams in the form "xy*" using the index, but not in the form "*xy". The code makes a misguided effort to look for " xy", but this will only find "xy" in words that begin with "xy", like "xylophone".

For example, searching Files for "om" does not currently find "random.txt".

Remove this behavior. Without engaging the trigram index, these queries fall back to an unidexed "LIKE" table scan, but that's about the best we can do.

Test Plan: Searched for "om", hit "random.txt".

Maniphest Tasks: T13501

Differential Revision: https://secure.phabricator.com/D21127
2020-04-16 09:44:37 -07:00
epriestley
b1b9c844ac Remove unused "getAllFunctionFields()" from Ferret
Summary: Ref T13511. This function does nothing interesting and has no callers.

Test Plan: Grepped for callers.

Maniphest Tasks: T13511

Differential Revision: https://secure.phabricator.com/D21126
2020-04-16 09:43:25 -07:00
epriestley
3573170dfa Compress file downloads if the client sends "Accept-Encoding: gzip" and we guess the file might compress alright
Summary:
Ref T13507. We currently compress normal responses, but do not compress file data responses because most files we serve are images and already compressed.

However, there are some cases where large files may be highly compressible (e.g., huge XML files stored in LFS) and we can benefit from compressing responses.

Make a reasonable guess about whether compression is beneficial and enable compression if we guess it is.

Test Plan:
  - Used `curl ...` to download an image with `Accept-Encoding: gzip`. Got raw image data in the response (as expected, because we don't expect images to be worthwhile to recompress).
  - Used `curl ...` to download a text file with `Accept-Encoding: gzip`. Got a compressed response. Decompressed the response into the original file.

Maniphest Tasks: T13507

Differential Revision: https://secure.phabricator.com/D21125
2020-04-15 11:53:35 -07:00
epriestley
45665dd3b4 Hide "notification.servers" configuration and don't follow redirects from Aphlict
Summary:
See <https://hackerone.com/reports/850114>.

An attacker with administrator privileges can configure "notification.servers" to connect to internal services, either directly or with chosen parameters by selecting an attacker-controlled service and having it issue a "Location" redirect.

Generally, we allow this attack to occur. The same administrator can use an authentication provider or a VCS repository to perform the same attack, and we can't reasonably harden these workflows without breaking things that users expect to be able to do.

There's no reason this particular variation of the attack needs to be allowable, though, and the current behavior isn't consistent with how other similar things work.

  - Hide the "notification.servers" configuration, which also locks it. This is similar to other modern service/server configuration.
  - Don't follow redirects on these requests. Aphlict should never issue a "Location" header, so if we encounter one something is misconfigured. Declining to follow this header likely makes the issue easier to debug.

Test Plan:
  - Viewed configuration in web UI.
  - Configured a server that "Location: ..." redirects, got a followed redirect before and a failure afterward.

{F7365973}

Differential Revision: https://secure.phabricator.com/D21123
2020-04-15 07:00:51 -07:00
epriestley
b52fa96238 Disable automatic decoding of "Content-Encoding" responses during "Accept-Encoding" setup test
Summary:
Ref T13507. Now that we handle processing of "Content-Encoding: gzip" headers by default, this setup check can get a decompressed body back. Since it specifically wants a raw body back, disable this behavior.

Also, "@" a couple things which can get in the way if they fail now that error handling is more aggressive about throwing on warnings.

Test Plan: Ran setup check after other changes in T13507, got clean result.

Maniphest Tasks: T13507

Differential Revision: https://secure.phabricator.com/D21122
2020-04-15 06:28:29 -07:00
epriestley
0ea6d131e0 In Conduit responses, assert that Phabricator supports a "gzip" capability
Summary: Ref T13507. If we believe the server can accept "Content-Encoding: gzip" requests, make the claim in an "X-Conduit-Capabilities" header in responses. Clients can use request compression on subsequent requests.

Test Plan: See D21119 for the client piece.

Maniphest Tasks: T13507

Differential Revision: https://secure.phabricator.com/D21120
2020-04-14 16:51:03 -07:00
epriestley
6b05d2be28 Add a setup warning to detect "SetInputFilter DEFLATE" and other "Content-Encoding" request mangling
Summary: Ref T13507. See that task for discussion.

Test Plan: Faked different response behaviors and hit both variations of this error.

Maniphest Tasks: T13507

Differential Revision: https://secure.phabricator.com/D21116
2020-04-14 14:48:43 -07:00
epriestley
99cbc20778 Reduce the verbosity of the "Aphlict" log
Summary:
See PHI1692. Currently, the Aphlict log is ridiculously verbose. As an initial pass at improving this:

  - When starting in "debug" mode, pass "--debug=1" to Node.
  - In Node, separate logging into "log" (lower-volume, more-important messages) and "trace" (higher-volume, less-important messages).
  - Only print "trace" messages in "debug" mode.

Test Plan: Ran Aphlict in debug and non-debug modes. Behavior unchanged in debug mode, but log has more sensible verbosity in non-debug mode.

Differential Revision: https://secure.phabricator.com/D21115
2020-04-14 13:24:44 -07:00
epriestley
59c855276b Provide a "--local" flag to "bin/conduit call" to force in-process execution
Summary:
See PHI1692. Currently, it's hard to get a local profile or "--trace" of some Diffusion API methods, since they always proxy via HTTP -- even if the local node can serve the request.

This always-proxy behavior is intentional (so we always go down the same code path, to limit surprises) but inconvenient when debugging. Allow an operator to connect to a node which can serve a request and issue a `--local` call to force in-process execution.

This makes it straightforward to "--trace" or "--xprofile" the call.

Test Plan: Ran `bin/conduit call ...` with and without `--local` using a Diffusion method on a clustered repository. Without `--local`, saw proxy via HTTP. With `--local`, saw in-process execution.

Differential Revision: https://secure.phabricator.com/D21114
2020-04-14 13:24:26 -07:00
epriestley
4655a5f059 Document the "field present" and "field absent" operators in Ferret
Summary: Ref T13509. Adds documentation for the new operators.

Test Plan: Read documentation, tried examples, got sensible-seeming results.

Maniphest Tasks: T13509

Differential Revision: https://secure.phabricator.com/D21112
2020-04-14 11:08:34 -07:00
epriestley
b3a8754013 Make the Ferret query compiler keep functions sticky across non-initial quoted tokens
Summary: Ref T13509. In `title:big "red" dog`, keep "title" sticky across all three terms, since this seems like it's probably the best match for intent.

Test Plan: Added unit tests; ran unit tests.

Maniphest Tasks: T13509

Differential Revision: https://secure.phabricator.com/D21111
2020-04-14 11:00:20 -07:00
epriestley
0511b2a012 Implement the "present" and "absent" operators in the Ferret execution engine
Summary:
Ref T13509. Now that the compiler can parse these queries, actually implement them.

These are fairly easy to implement:

  - For present, just "JOIN". If it works, the field is present.
  - For absent, we "LEFT JOIN" and then "WHERE any_column IS NULL".

Test Plan: Searched for various documents with and without fields present, got sensible results in Maniphest. For example, "body:-" finds tasks with no body, "body:- duck" finds tasks with no body and "duck" elsewhere in the content, and so on.

Maniphest Tasks: T13509

Differential Revision: https://secure.phabricator.com/D21110
2020-04-14 10:55:30 -07:00
epriestley
143f86d60b Tighten query compiler rules around spaces inside and after operators
Summary:
Ref T13509. Since `title:- cat` is now ambiguous, forbid spaces after operators.

Also, forbid spaces inside operators, although this has no effect today.

Test Plan: Added unit tests, ran unit tests.

Maniphest Tasks: T13509

Differential Revision: https://secure.phabricator.com/D21109
2020-04-14 10:51:55 -07:00
epriestley
8fa8d0e648 Make Ferret query functions sticky only if their values are not quoted
Summary:
Ref T13509. Currently, functions are "sticky", but this stickness is in the query execution layer.

Instead:

  - move stickiness to the query compiler; and
  - make it so that functions are not sticky if their arguments are quoted.

For example:

  - `title:x y` previously meant `title:x title:y` (and still does). The "title:" is sticky.
  - `title:"x" y` previously meant `title:x title:y`. It now means `title:x all:y`. The "title:" is not sticky because the argument is quoted.

Test Plan: Added unit tests, ran unit tests.

Maniphest Tasks: T13509

Differential Revision: https://secure.phabricator.com/D21108
2020-04-14 10:47:51 -07:00
epriestley
f31b9987ba Add "absent" and "present" field operators to the Ferret query compiler
Summary: Ref T13509. Parse "xyz:-" as "xyz is absent" and "xyz:~" as "xyz is present". These are new operators which the compiler emits separately from "not" and "substring".

Test Plan: Added unit tests, ran unit tests.

Maniphest Tasks: T13509

Differential Revision: https://secure.phabricator.com/D21107
2020-04-14 10:47:20 -07:00
epriestley
5c30a60e30 Tighten Ferret query parsing of empty tokens and empty functions
Summary:
Ref T13509. Certain query tokens like `title:=""` are currently accepted by the parser but discarded, and have no impact on the query. This isn't desirable.

Instead, require that tokens making an assertion about field content must be nonempty.

Test Plan: Added unit tests, made them pass.

Maniphest Tasks: T13509

Differential Revision: https://secure.phabricator.com/D21106
2020-04-14 10:32:46 -07:00
epriestley
471e89a8b7 Add "uri" to "paste.search" API output
Summary: Ref T13490. This simplifies some client behavior in the general case.

Test Plan: Called API method, saw URIs.

Maniphest Tasks: T13490

Differential Revision: https://secure.phabricator.com/D21105
2020-04-13 16:17:33 -07:00
epriestley
19e0abcb27 Fix an issue where raw diffs that are not attached to revisions could skip repository policy checks
Summary:
See PHI1697. If a diff is not attached to a revision (for example, if it was created with "arc diff --only"), but is attached to a repository, it is supposed to be visible only to users who can see that repository.

It currently skips this extended policy check and may incorrectly be visible to too many users.

(Once a diff is attached to a revision, this rule is enforced properly via the revision policy.)

Test Plan:
  - Set repository R to be visible only to Alice.
  - As Alice, created a diff from a working copy of repository R with "arc diff --only".
  - As Bailey, viewed the diff.
    - Before: visible diff.
    - After: policy exception (as expected).

Differential Revision: https://secure.phabricator.com/D21103
2020-04-13 12:08:35 -07:00
epriestley
5597f4e6f2 Add "uri" to the fields returned by "differential.revision.search"
Summary: Ref T13490. This simplifies mostly-theoretical cases where you're accessing Phabricator via arc-over-ssh and the Conduit protocol + domain may differ from the production protocol + domain.

Test Plan: Called API via web UI, saw sensible URI values in results.

Maniphest Tasks: T13490

Differential Revision: https://secure.phabricator.com/D21102
2020-04-13 12:06:39 -07:00
epriestley
c3be82fe6e Fix an out-of-date API call on the destruction pathway for Pholio mocks
Summary:
See <https://discourse.phabricator-community.org/t/destroying-a-mock-using-bin-remove-destroy-mx-gives-an-error/3728>.

Currently, Pholio calls an older API method on the mock destruction pathway. This call was introduced in D19911 but the callsite was only partially updated in D19914.

Test Plan: Ran "bin/remove destroy Mx" to destroy a mock. Before: fatal with a bad call; after: clean destruction.

Differential Revision: https://secure.phabricator.com/D21081
2020-04-10 08:01:34 -07:00
epriestley
58fbf64a27 Refine handling of "@task" attributes in Diviner
Summary: Ref T13505. See that task for details. When a class has exactly one "@task" block, this API returns a string. Some day, this should be made more consistent.

Test Plan: Viewed a class with exactly one "@task", no more fatal. Viewed classes with zero and more than one "@task" attributes, got clean renderings.

Maniphest Tasks: T13505

Differential Revision: https://secure.phabricator.com/D21062
2020-04-06 11:51:33 -07:00
epriestley
271e104c7e Update DivinerAtomController for a long-ago change to the docblock parser API
Summary: Ref T13505. See that task for discussion.

Test Plan: Ran `diviner generate` locally, found a page fataling on this `strlen()`, applied patch, got a sketchy but not-broken page.

Maniphest Tasks: T13505

Differential Revision: https://secure.phabricator.com/D21061
2020-04-06 11:31:31 -07:00
epriestley
f1d1ec3d77 Add an "isDone" flag to "transaction.search" for Differential inline comments
Summary: See PHI1684. Expose the published state of the "Done" checkbox to the API.

Test Plan: Made API calls on a comment in all four states, got correct published states via the API in all cases.

Differential Revision: https://secure.phabricator.com/D21059
2020-04-05 09:36:15 -07:00
epriestley
33b73d887a If daemon running-as-user setup check fails its query, don't bother with it
Summary:
See <https://discourse.phabricator-community.org/t/upgrade-from-sep-30-2016/3702/>. A user performing an upgrade from 2016 to 2020 ran into an issue where this setup query is overheating.

This is likely caused by too many rows changing state during query execution, but the particulars aren't important since this setup check isn't too critical and will catch the issue eventually. It's fine to just move on if this query fails for any reason.

Test Plan: Forced the query to overheat, loaded setup issues, got overheating fatal. Applied patch, no more fatal.

Differential Revision: https://secure.phabricator.com/D21057
2020-04-03 16:18:55 -07:00
epriestley
1e7cc72cd8 Improve performance when marking commits as unreachable after multiple ref deletions
Summary:
See PHI1688. If many refs with a large amount of shared ancestry are deleted from a repository, we can spend much longer than necessary marking their mutual ancestors as unreachable over and over again.

For example, if refs A, B and C all point near the head of an obsolete "develop" branch and have about 1K shared commits reachable from no other refs, deleting all three refs will lead to us performing 3,000 mark-as-unreachable operations (once for each "<ref, commit>" pair).

Instead, we can stop exploring history once we reach an already-unreachable commit.

Test Plan:
  - Destroyed 7 similar refs simultaneously.
  - Ran `bin/repository refs`, saw 7 entries appear in the `oldref` table.
  - Ran `bin/repository discover` with some debugging statements added, saw sensible-seeming behavior which didn't double-mark any newly-unreachable refs.

Differential Revision: https://secure.phabricator.com/D21056
2020-04-03 13:28:42 -07:00
epriestley
1a59cae743 Update some Phabricator behaviors for changes to Futures
Summary:
Depends on D21053. Ref T11968. Three things have changed:

  - Overseers can no longer use FutureIterator to continue execution of an arbitrary list of futures from any state. Use FuturePool instead.
  - Same with repository daemons.
  - Probably (?) fix an API change in the Harbormaster exec future.

Test Plan:
  - Ran "bin/phd debug task" and "bin/phd debug pull", no longer saw Future-management related errors.
  - The Harbormaster future is easiest to test by just seeing if production works once this change is deployed there.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11968

Differential Revision: https://secure.phabricator.com/D21054
2020-04-03 12:28:16 -07:00
epriestley
067b04aaf1 If HTTP response headers are already sent, don't fiddle with "zlib.output_compression"
Summary:
We write some synthetic HTTP responses inside unit tests. Some responses have an indirect side effect of adjusting "zlib.output_compression", but this adjustment fails if headers have already been output. From a CLI context, headers appear to count as already-output after we write anything to stdout:

```
<?php

echo headers_sent() ? "Y" : "N";
echo "\n";
echo headers_sent() ? "Y" : "N";
echo "\n";
```

This script prints "N", then "Y".

Recently, the default severity of warnings was increased in libphutil; this has been a long-standing warning but now causes test failures.

This behavior is sort of silly but the whole thing is kind of moot anyway. Just skip it if "headers_sent()" is true.

Test Plan: Ran "arc unit --everything", got clean results.

Differential Revision: https://secure.phabricator.com/D21055
2020-04-03 12:24:58 -07:00
Arturas Moskvinas arturas@uber.com
62f5bdbbd2 According to Jira Project keys must start with an uppercase letter, followed by one or more uppercase alphanumeric characters
Summary: Jira allows creating projects which contain number in names, phabricator will not allow such projects but it should

Test Plan: Pasted URL with Jira project which contain number in project name and it was parsed and resolved properly in phabricator

Reviewers: epriestley, Pawka, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D21040
2020-03-09 22:04:23 +02:00
epriestley
d0f4554dbe Read both email addresses and Google Account IDs from Google OAuth
Summary:
Ref T13493. Google returns a lower-quality account identifier ("email") and a higher-quality account identifier ("id"). We currently read only "email".

Change the logic to read both "email" and "id", so that if Google ever moves away from "email" the transition will be a bit easier.

Test Plan: Linked/unlinked a Google account, looked at the external account identifier table.

Maniphest Tasks: T13493

Differential Revision: https://secure.phabricator.com/D21028
2020-02-24 13:26:42 -08:00
epriestley
785f3c98da Extract raw commit messages from Git more faithfully across Git versions
Summary:
Fixes T5028. Older versions of Git (apparently, from before 2010) did not provide a way to extract the raw body of a commit message from "git log", so we approximate it with "subject" and "wrapped body".

In newer versions of Git, the raw body can be extracted exactly.

Adjust how we extract messages based on the version of Git, and try to be more faithful to edge cases: particularly, be more careful to extract the correct number of trailing newlines.

Test Plan:
  - Added "var_dump()" + "die(1)" later in this method, then pushed various commit messages. Used "&& false" to force execution down the old path (either path should work in modern Git).
  - Observed more faithful extraction of messages, including a more faithful extraction of the number of trailing newlines. Extraction is fully faithful if we can go down the "%B" path, which we should be able to in nearly all modern cases.
  - Not all messages extract faithfully or consistently across the old and new versions, but the old extraction is destructive so this is likely about as close as we can realistically ever get.

Maniphest Tasks: T5028

Differential Revision: https://secure.phabricator.com/D21027
2020-02-24 12:37:45 -08:00
epriestley
e58ef418c7 Read both older "key" and newer "accountId" identifiers from JIRA during authentication
Summary:
Depends on D21022. Ref T13493. The JIRA API has changed from using "key" to identify users to using "accountId".

By reading both identifiers, this linkage "just works" if you run against an old version of JIRA, a new version of JIRA, or an intermediate version of JIRA.

It also "just works" if you run old JIRA, upgrade to intermediate JIRA, everyone refreshes their link at least once, then you upgrade to new JIRA.

This is a subset of cases and does not include "sudden upgrade to new JIRA", but it's strictly better than the old behavior for all cases it covers.

Test Plan: Linked, unlinked, and logged in with JIRA. Looked at the "ExternalAccountIdentifier" table and saw a sensible value.

Maniphest Tasks: T13493

Differential Revision: https://secure.phabricator.com/D21023
2020-02-22 17:49:47 -08:00
epriestley
802b5aca05 Remove all readers and writers of "accountID" on "ExternalAccount"
Summary: Depends on D21019. Ref T13493. There are no more barriers to removing readers and writers of "accountID"; the new "ExternalAccountIdentity" table can replace it completely.

Test Plan: Linked and unlinked OAuth accounts, logged in with OAuth accounts, tried to double-link OAuth accounts, grepped for affected symbols.

Maniphest Tasks: T13493

Differential Revision: https://secure.phabricator.com/D21022
2020-02-22 17:49:22 -08:00
epriestley
84b5ad09e6 Remove all readers and all nontrivial writers for "accountType" and "accountDomain" on "ExternalAccount"
Summary:
Depends on D21018. Ref T13493. Ref T6703. The "ExternalAccount" table has a unique key on `<accountType, accountDomain, accountID>` but this no longer matches our model of reality and changes in this sequence end writes to `accountID`.

Remove this key.

Then, remove all readers of `accountType` and `accountDomain` (and all nontrivial writers) because none of these callsites are well-aligned with plans in T6703.

This change has no user-facing impact today: all the rules about linking/unlinking/etc remain unchanged, because other rules currently prevent creation of more than one provider with a given "accountType".

Test Plan:
- Linked an OAuth1 account (JIRA).
- Linked an OAuth2 account (Asana).
- Used `bin/auth refresh` to cycle OAuth tokens.
- Grepped for affected symbols.
- Published an Asana update.
- Published a JIRA link.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13493, T6703

Differential Revision: https://secure.phabricator.com/D21019
2020-02-22 17:48:46 -08:00
epriestley
b8f0613b30 Update Asana feed publishing integration for "ExternalAccountIdentifier"
Summary: Depends on D21017. Ref T13493. Update the Asana integration so it reads the "ExternalAccountIdentifier" table instead of the old "accountID" field.

Test Plan: Linked an Asana account, used `bin/feed republish` to publish activity to Asana.

Maniphest Tasks: T13493

Differential Revision: https://secure.phabricator.com/D21018
2020-02-22 17:48:16 -08:00
epriestley
bcaf60015a Write ExternalAccountIdentifiers when interacting with external authentication providers
Summary:
Depends on D21015. When we sync an external account and get a list of account identifiers, write them to the database.

Nothing reads them yet and we still write "accountId", this just prepares us for reads.

Test Plan: Linked, refreshed, unlinked, and re-linked an external account. Peeked at the database and saw a sensible-looking row.

Differential Revision: https://secure.phabricator.com/D21016
2020-02-22 17:46:51 -08:00
epriestley
0872051bfa Make AuthProvider, ExternalAccount, and ExternalAccountIdentifier all Destructible
Summary: Depends on D21014. Ref T13493. Make these objects all use destructible interfaces and destroy sub-objects appropriately.

Test Plan:
  - Used `bin/remove destroy --trace ...` to destroy a provider, a user, and an external account.
  - Observed destruction of sub-objects, including external account identifiers.

Maniphest Tasks: T13493

Differential Revision: https://secure.phabricator.com/D21015
2020-02-22 17:46:29 -08:00
epriestley
05eb16d6de Update unusual handling of external accounts in "Password" auth provider
Summary:
Depends on D21013. Ref T13493. When users log in with most providers, the provider returns an "ExternalAccount" identifier (like an Asana account GUID) and the workflow figures out where to go from there, usually a decision to try to send the user to registration (if the external account isn't linked to anything yet) or login (if it is).

In the case of password providers, the password is really a property of an existing account, so sending the user to registration never makes sense. We can bypass the "external identifier" indirection layer and just say "username -> internal account" instead of "external GUID -> internal mapping -> internal account".

Formalize this so that "AuthProvider" can generate either a "map this external account" value or a "use this internal account" value.

This stops populating "accountID" on "password" "ExternalAccount" objects, but this was only an artifact of convenience. (These records don't really need to exist at all, but there's little harm in going down the same workflow as everything else for consistency.)

Test Plan: Logged in with a username/password. Wiped the external account table and repeated the process.

Maniphest Tasks: T13493

Differential Revision: https://secure.phabricator.com/D21014
2020-02-22 17:46:04 -08:00
epriestley
e43ecad8af Make external account identifier APIs return multiple identifiers
Summary:
Depends on D21012. Ref T13493. Currently, auth adapters return a single identifier for each external account.

Allow them to return more than one identifier, to better handle cases where an API changes from providing a lower-quality identifier to a higher-quality identifier.

On its own, this change doesn't change any user-facing behavior.

Test Plan: Linked and unlinked external accounts.

Maniphest Tasks: T13493

Differential Revision: https://secure.phabricator.com/D21013
2020-02-22 17:45:45 -08:00
epriestley
4094624828 Remove an ancient no-op check for duplicated external accounts
Summary:
Ref T13493. This check was introduced in D4647, but the condition can never be reached in modern Phabricator because the table has a unique key on `<accountType, accountDomain, accountID>` -- so no row can ever exist with the same value for that tuple but a different ID.

(I'm not entirely sure if it was reachable in D4647 either.)

Test Plan: Used `SHOW CREATE TABLE` to look at keys on the table and reasoned that this block can never have any effect.

Maniphest Tasks: T13493

Differential Revision: https://secure.phabricator.com/D21012
2020-02-22 17:45:19 -08:00
epriestley
70845a2d13 Add an "ExternalAccountIdentifier" table
Summary:
Depends on D21010. Ref T13493. External accounts may have multiple different unique identifiers, most often when v1 of the API makes a questionable choice (and provies a mutable, non-unique, or PII identifier) and v2 of the API uses an immutable, unique, random identifier.

Allow Phabricator to store multiple identifiers per external account.

Test Plan: Storage only, see followup changes.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13493

Differential Revision: https://secure.phabricator.com/D21011
2020-02-22 17:44:13 -08:00
epriestley
fbf050167e Stop exposing raw "accountID" values directly in the web UI
Summary:
Ref T13493. The "AuthAccountView" UI element currently exposes raw account ID values, but I'm trying to make these many-to-one.

This isn't terribly useful as-is, so get rid of it. This element could use a design refresh in general.

Test Plan: Viewed the UI element in "External Accounts".

Maniphest Tasks: T13493

Differential Revision: https://secure.phabricator.com/D21010
2020-02-22 17:41:55 -08:00
epriestley
149155ee20 Whitelist "vscode://" as an allowed Editor protocol
Summary:
See PHI1647, which asks for "vscode://" to be a configurable protocol on hosted Phacility instances.

I made the configuration editable in D21008, but this can reasonably just come upstream too.

Test Plan: Viewed config in Config, set my editor URI to `vscode://blahblah`.

Differential Revision: https://secure.phabricator.com/D21009
2020-02-20 12:45:35 -08:00
epriestley
29923cc71a Remove old code for sending email to external users who create objects via inbound mail
Summary:
Ref T13493. I'm updating callers to `getAccountID()` to prepare to move it to a separate table.

This callsite once supported this flow:

  - External users with no accounts send mail to `bugs@`.
  - This creates tasks in Maniphest.
  - They're CC'd when the tasks are updated.

However, after T12237 we never actually send this mail (since their addresses are necessarily unverified).

I left this code in in case this needed to be revisited, but it hasn't been an issue. Just remove it and treat these users as undeliverable.

Test Plan: As a cursory test for nothing being horribly broken, sent some object mail.

Maniphest Tasks: T13493

Differential Revision: https://secure.phabricator.com/D21007
2020-02-20 12:41:51 -08:00
epriestley
64cc4fe915 Add a test to verify that all routing maps are plausibly valid, and remove some dead routes
Summary:
Previously, see D20999. See also <https://discourse.phabricator-community.org/t/the-phutil-library-phutil-has-not-been-loaded/3543/>.

There are a couple of dead "Config" routes after recent changes. Add test coverage to make sure routes all point somewhere valid, then remove all the dead routes that turned up.

Test Plan: Ran tests, saw failures. Removed dead routes, got clean tests.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D21000
2020-02-14 18:06:24 -08:00
epriestley
4790a3d94b Stop trying to version-check libphutil in "Config"
Summary:
Ref T13395. No library with this name loads any more, so we can't version check it.

(Ideally, the version check stuff would be more graceful when it fails now, since it's required to load "Config" after I moved it off a separate page.)

Test Plan: Loaded "Config".

Maniphest Tasks: T13395

Differential Revision: https://secure.phabricator.com/D20995
2020-02-14 08:44:50 -08:00
epriestley
dc35ce79e4 Unprototype "Draft" mode in Differential
Summary: Fixes T2543. This mode has been a stable prototype for a very long time now; promote it so "--draft" can promote out of "experimental" in Arcanist.

Test Plan: See T2543 for discussion.

Maniphest Tasks: T2543

Differential Revision: https://secure.phabricator.com/D20983
2020-02-12 16:20:39 -08:00
epriestley
35a18146a2 Merge a small amount of remaining "libphutil/" code with Phabricator, break libphutil dependency
Summary: Ref T13395. Moves a small amount of remaining "libphutil/" code into "phabricator/" and stops us from loading "libphutil/".

Test Plan: Browsed around; there are likely remaining issues.

Maniphest Tasks: T13395

Differential Revision: https://secure.phabricator.com/D20981
2020-02-12 15:17:36 -08:00
epriestley
f9b3e3360b Continue moving classes with no callers in libphutil or Arcanist to Phabricator
Summary: Ref T13395. Move cache classes, syntax highlighters, other markup classes, and sprite sheets to Phabricator.

Test Plan: Attempted to find any callers for any of this stuff in libphutil or Arcanist and couldn't.

Maniphest Tasks: T13395

Differential Revision: https://secure.phabricator.com/D20977
2020-02-12 13:14:04 -08:00
Arturas Moskvinas arturas@uber.com
8cc6fe465c Fix diffusion.branchquery returning dictionary instead of array when branches are filtered out
Summary:
`diffusion.branchquery` can return dictionary instead of array if some branches are filtered out.
Eg.:
```
{
  "result": [
    {
      "shortName": "master",
      "commitIdentifier": "2817b0d8f79748ddfad0220c46d9b20bea34f460",
      "refType": "branch",
      "rawFields": {
        "objectname": "2817b0d8f79748ddfad0220c46d9b20bea34f460",
        "objecttype": "commit",
```
might become:
```
{
  "result": {
    "1": {
      "shortName": "master",
      "commitIdentifier": "2817b0d8f79748ddfad0220c46d9b20bea34f460",
      "refType": "branch",
      "rawFields": {
        "objectname": "2817b0d8f79748ddfad0220c46d9b20bea34f460",
        "objecttype": "commit",

```
Reproduction - find repository which has couple of branches, setup to track only some of them, execute `diffusion.branchquery` API call - result is dictionary instead of array

Test Plan: Apply patch, execution `diffusion.branchquery` call - result is no longer dictionary if it was one before

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D20973
2020-02-12 11:50:22 -08:00
epriestley
af84f215f9 Move lingering "Aphront" classes to Phabricator
Summary: Ref T13395. Moves some Aphront classes from libphutil to Phabricator.

Test Plan: Grepped for symbols in libphutil and Arcanist.

Maniphest Tasks: T13395

Differential Revision: https://secure.phabricator.com/D20975
2020-02-12 11:50:14 -08:00
epriestley
2327578adc Respect linebreaks in full HTML tables in Remarkup
Summary:
Fixes T5427. See PHI1630. See also T13160 and D20568.

In the full HTML table syntax with "<table>", respect linebreaks as literals inside "<td>" cells.

Test Plan: Previewed some full-HTML tables with and without linebreaks, saw what seemed like sensible rendering behavior.

Maniphest Tasks: T5427

Differential Revision: https://secure.phabricator.com/D20971
2020-02-06 15:01:16 -08:00
epriestley
9d1af762d5 In summary interfaces, don't render very large inline remarkup details for unit test messages
Summary: Ref T10635. An install with large blocks of remarkup (4MB) in test details is reporting slow page rendering. This is expected, but I've mostly given up on fighting this unless I absolutely have to. Degrade the interface more aggressively.

Test Plan:
  - Submitted a large block of test details in remarkup format.
  - Before patch: they rendered inline.
  - After patch: degraded display.
  - Verified small blocks are not changed.

{F7180727}

{F7180728}

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T10635

Differential Revision: https://secure.phabricator.com/D20970
2020-02-05 14:26:38 -08:00
epriestley
fd46c597ae When sorting subscriber references for display in the curtain UI, sort without case sensitivity
Summary: Ref T13486. Currently, "Zarbo" sorts above "alice", but this isn't expected for a list of (mostly) human usernames.

Test Plan: Loaded a task with subscribers with mixed-case usernames.

Maniphest Tasks: T13486

Differential Revision: https://secure.phabricator.com/D20969
2020-02-04 15:26:05 -08:00
epriestley
fdbe9ba149 Improve Remarkup parsing performance for certain large input blocks
Summary: Fixes T13487. In PHI1628, an install has a 4MB remarkup corpus which takes a long time to render. This is broadly expected, but a few reasonable improvements fell out of running it through the profiler.

Test Plan:
  - Saw local cold-cache end-to-end rendering time drop from 12s to 4s for the highly secret input corpus.
  - Verified output has the same hashes before/after.
  - Ran all remarkup unit tests.

Maniphest Tasks: T13487

Differential Revision: https://secure.phabricator.com/D20968
2020-02-04 15:07:00 -08:00
epriestley
0e82bd024a Use the new "CurtainObjectRefList" UI element for subscribers
Summary:
Depends on D20966. Ref T13486. Curtains currently render subscribers in a plain text list, but the new ref list element is a good fit for this.

Also, improve the sorting and ordering behavior.

This makes the subscriber list take up a bit more space, but it should make it a lot easier to read at a glance.

Test Plan: Viewed object subscriber lists at varying limits and subscriber counts, saw sensible subscriber lists.

Maniphest Tasks: T13486

Differential Revision: https://secure.phabricator.com/D20967
2020-02-04 12:38:41 -08:00
epriestley
2a92fef879 Improve wrapping and overflow behavior for curtain panels containing long usernames
Summary:
Ref T13486. When a curtain element like "Author" in Maniphest has a very long username, the wrapping and overflow behavior is poor: the date is obscured.

Adjust curtain elements which contain lists of references to other objects to improve wrapping behavior (put the date on a separate line) and overflow behavior (so we get a "..." when a name overflows).

Test Plan: {F7179376}

Maniphest Tasks: T13486

Differential Revision: https://secure.phabricator.com/D20966
2020-02-04 12:31:18 -08:00
epriestley
84fd5cd5bb Fix an issue where intracontent empty lines were incorrectly trimmed in quoted blocks
Summary: Fixes T13335. When processing quoted blocks, we remove leading empty lines. This logic incorrectly continued after encountering a nonempty line.

Test Plan: Added a test, made it pass. Previewed blocks in web UI.

Maniphest Tasks: T13335

Differential Revision: https://secure.phabricator.com/D20965
2020-02-04 08:09:50 -08:00
epriestley
0f1acb6cef Update GitHub API calls to use "Authorization" header instead of "access_token" URI parameter
Summary: Fixes T13485. GitHub has deprecated the "access_token" URI parameter for API authentication. Update to "Authorization: token ...".

Test Plan: Linked and unlinked a GitHub account locally.

Maniphest Tasks: T13485

Differential Revision: https://secure.phabricator.com/D20964
2020-02-04 07:58:03 -08:00
epriestley
6d4c6924d6 Update Herald rule creation workflow to use more modern UI elements
Summary: Ref T13480. Creating a rule in Herald currently uses the older radio-button flow. Update it to the "clickable menu" flow to simplify it a little bit.

Test Plan: Created new personal, object, and global rules. Hit the object rule error conditions.

Maniphest Tasks: T13480

Differential Revision: https://secure.phabricator.com/D20956
2020-02-04 07:37:54 -08:00
epriestley
4904d7711e When publishing a commit, copy "Related Tasks" from the associated revision (if one exists)
Summary:
Fixes T13463. Currently, if you use the web UI to set "Related Tasks" for a revision, the resulting commit does not link to the tasks.

If you use "Ref ..." in the message instead, the resulting commit does link to the tasks.

Broadly, this should all be cleaner (see T3577) but we can step toward better behavior by just copying these edges when commits are published.

Test Plan:
  - Created a revision.
  - Used the web UI to edit "Related Tasks".
  - Landed the revision.
  - Saw the commit link to the tasks as though I'd used "Ref ..." in the message.

Maniphest Tasks: T13463

Differential Revision: https://secure.phabricator.com/D20961
2020-02-04 07:05:09 -08:00
epriestley
530145ba3b Give "Config" a full-width, hierarchical layout
Summary:
Depends on D20933. Ref T13362. This reorganizes Config a bit and attempts to simplify it.

Subsections are now in a landing page console and groupings have been removed. We "only" have 75 values you can edit from the web UI nowadays, which is still a lot, but less overwhelming than it was in the past. And the trend is generally downward, as config is removed/simplified or moved into application settings.

This also gets rid of the "gigantic blobs of JSON in the UI".

Test Plan: Browsed all Config sections.

Maniphest Tasks: T13362

Differential Revision: https://secure.phabricator.com/D20934
2020-02-04 06:59:51 -08:00
epriestley
26c2a1ba68 Move existing "Console" interfaces away from "setFixed(...)" on "TwoColumnView"
Summary: Depends on D20931. Ref T13362. Move all "Console"-style interfaces to use a consistent layout based on a new "LauncherView" which just centers the content.

Test Plan: Viewed all affected interfaces.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13362

Differential Revision: https://secure.phabricator.com/D20933
2020-02-04 06:52:23 -08:00
epriestley
cb481f36c5 Carve out a separate "Services" section of Config
Summary: Depends on D20930. Ref T13362. Put all the "Services" parts of Config in their own section.

Test Plan: Clicked through each section. This is just an organization / UI change with no significant behavioral impact.

Maniphest Tasks: T13362

Differential Revision: https://secure.phabricator.com/D20931
2020-02-04 06:47:31 -08:00
epriestley
a72ade9475 Carve out a separate "Modules/Extensions" section of Config
Summary:
Ref T13362. Config is currently doing a ton of stuff and fairly overwhelming. Separate out "Modules/Extensions" so it can live in its own section.

(This stuff is mostly useful for development and normal users rarely need to end up here.)

Test Plan: Visited seciton, clicked around. This is just a visual change.

Maniphest Tasks: T13362

Differential Revision: https://secure.phabricator.com/D20930
2020-02-04 06:41:55 -08:00
epriestley
c42c5983aa Fix an issue where loading a mangled project graph could fail too abruptly
Summary:
Ref T13484. If you load a subproject S which has a mangled/invalid `parentPath`, the query currently tries to execute an empty edge query and fatals.

Instead, we want to deny-by-default in the policy layer but not fail the query. The subproject should become restricted but not fatal anything related to it.

See T13484 for a future refinement where we could identify "broken / data integrity issue" objects explicilty.

Test Plan:
  - Modified the `projectPath` of some subproject in the database to `QQQQ...`.
  - Loaded that project page.
  - Before patch: fatal after issuing bad edge query.
  - After patch: "functionally correct" policy layer failure, although an explicit "data integrity issue" failure would be better.

Maniphest Tasks: T13484

Differential Revision: https://secure.phabricator.com/D20963
2020-02-03 08:54:04 -08:00
epriestley
42e46bbe5a Fix an issue where Herald rules could fail to evaluate at post-commit time
Summary: Ref T13480. Some Herald fields need audit information, which recent changes to Herald adapters discarded. For now, just load it unconditionally.

Test Plan: Triggered an Audit-related rule locally.

Maniphest Tasks: T13480

Differential Revision: https://secure.phabricator.com/D20962
2020-02-03 05:09:43 -08:00
epriestley
ccf28a8112 Fix an issue where the last line of block-based diffs could be incorrectly hidden
Summary:
Fixes T13468. See that task for discussion. The older source-rendering code mixes "line number" / "1-based" lists with "block number" / "0-based" lists and then has other bugs which cancel this out.

For block-based diffs, build an explicit block-based mask with only block numbers. This sort of sidesteps the whole issue.

Test Plan: Viewed the diff with the original reproduction case, plus various other block-based diffs, including one-block image diffs, in unified and side-by-side mode. Didn't spot any oddities.

Maniphest Tasks: T13468

Differential Revision: https://secure.phabricator.com/D20959
2020-01-30 08:19:09 -08:00
epriestley
12c3370988 When issuing a "no-op" MFA token because no MFA is configured, don't give the timeline story a badge
Summary:
Fixes T13475. Sometimes, we issue a "no op" / "default permit" / "unchallenged" MFA token, when a user with no MFA configured does something which is configured to attempt (but not strictly require) MFA.

An example of this kind of action is changing a username: usernames may be changed even if MFA is not set up.

(Some other operations, notably "Sign With MFA", strictly require that MFA actually be set up.)

When a user with no MFA configured takes a "try MFA" action, we see that they have no factors configured and issue a token so they can continue. This is correct. However, this token causes the assocaited timeline story to get an MFA badge.

This badge is incorrect or at least wildly misleading, since the technical assertion it currently makes ("the user answered any configured MFA challenge to do this, if one exists") isn't explained properly and isn't useful anyway.

Instead, only badge the story if the user actually has MFA and actually responded to some kind of MFA challege. The badge now asserts "this user responded to an MFA challenge", which is expected/desired.

Test Plan:
  - As a user with no MFA, renamed a user. Before patch: badged story. After patch: no badge.
  - As a user with MFA, renamed a user. Got badged stories in both cases.

Maniphest Tasks: T13475

Differential Revision: https://secure.phabricator.com/D20958
2020-01-30 07:35:40 -08:00
epriestley
c99485e8a0 Add "Author's Packages" and "Committer's Packages" Herald rules for Commits and Hooks
Summary: Fixes T13480. Adds the remaining missing Owners package rules for Herald commit adapters.

Test Plan: Created hooks which care about these fields, pushed commits, saw sensible transcript values.

Maniphest Tasks: T13480

Differential Revision: https://secure.phabricator.com/D20957
2020-01-29 15:52:07 -08:00
epriestley
6628cd2b4f In Herald "Commit" rules, use repository identities to identify authors and committers
Summary: Ref T13480. The Herald "Commit" rules still use raw commit data properties to identify authors and committers. Instead, use repository identities.

Test Plan: Wrote a Herald rule using all four fields, ran it against various commits with and without known authors. Checked transcript for sensible field values.

Maniphest Tasks: T13480

Differential Revision: https://secure.phabricator.com/D20955
2020-01-29 15:48:59 -08:00
epriestley
41f143f7fe Respect repository identities when figuring out authors/committers in Herald pre-commit hook rules
Summary:
Ref T13480. Currently, Herald commit hook rules use a raw address resolution query to identify the author and committer for a commit. This will get the wrong answer when the raw identity string has been explicitly bound to some non-default user (most often, it will fail to identify an author when one exists).

Instead, use the "IdentityEngine" to properly resolve identities.

Test Plan: Authored a commit as `X <y@example.com>`, a raw identity with no "natural" matches to users (e.g., no user with that email or username). Bound the identity to a particular user in Diffusion. Wrote a Herald pre-commit content rule, pushed the commit. Saw Herald recognize the correct user when evaluating rules.

Maniphest Tasks: T13480

Differential Revision: https://secure.phabricator.com/D20953
2020-01-29 15:15:11 -08:00
epriestley
a0a346be34 In Herald transcripts, render some field values in a more readable way
Summary:
Ref T13480. Currently, some Herald field types are rendered in an unfriendly way on transcripts. Particularly, PHID lists are rendered as raw PHIDs.

Improve this by delegating rendering to Value objects and letting "PHID List" value objects render more sensible handle lists. Also improve "bool" fields a bit and make more fields render an explicit "None" / empty value rather than just rendering nothing.

Test Plan: Viewed various transcripts, including transcripts covering boolean values, the "Always" condition, large blocks of text, and PHID lists.

Maniphest Tasks: T13480

Differential Revision: https://secure.phabricator.com/D20951
2020-01-29 15:14:06 -08:00
epriestley
19662e33bc In Herald transcript rendering, don't store display labels in keys
Summary:
Ref T13480. Currently, when Herald renders a transcript, it puts display labels into array keys. This is a bad pattern for several reasons, notably that the values must be scalar (so you can't add icons or other markup later) and the values must be unique (which is easily violated because many values are translated).

Instead, keep values as list items.

Test Plan: Viewed Herald transcripts, saw no (meaningful) change in rendering output.

Maniphest Tasks: T13480

Differential Revision: https://secure.phabricator.com/D20949
2020-01-29 15:11:41 -08:00
epriestley
a5a9a5e002 Remove legacy pre-loading of handles from Herald rendering
Summary: Ref T13480. When Herald renders rules, it partly uses a very old handle pre-loading mechanism where PHIDs are extracted and loaded upfront. This was obsoleted a long time ago and was pretty shaky even when it worked. Get rid of it to simplify the code a little.

Test Plan: Viewed Herald rules rendered into static text with PHID list actions, saw handles. Grepped for all affected methods.

Maniphest Tasks: T13480

Differential Revision: https://secure.phabricator.com/D20948
2020-01-29 15:07:23 -08:00
epriestley
b38449ce8f Implement an "Author's packages" Herald field for Differential
Summary: Ref T13480. Add an "Author's packages" field to Herald to support writing rules like "if affected packages include X, and author's packages do not include X, raise the alarm".

Test Plan: Wrote and executed rules with the field, saw a sensible field value in the transcript.

Maniphest Tasks: T13480

Differential Revision: https://secure.phabricator.com/D20947
2020-01-22 18:27:51 -08:00
epriestley
6c4500046f Add "Project tags added" and "Project tags removed" fields in Herald
Summary: Ref T13480. These fields don't serve a specific strong use case, but are broadly reasonable capabilities after "state" vs "change" actions were relaxed by T13283.

Test Plan: Wrote rules using the new fields, added and removed projects (and did neither) to fire them / not fire them. Inspected the transcripts to see the project PHIDs making it to the field values.

Maniphest Tasks: T13480

Differential Revision: https://secure.phabricator.com/D20946
2020-01-22 18:20:57 -08:00
epriestley
6ccb6a6463 Update "git rev-parse" invocation to work in Git 2.25.0
Summary:
Fixes T13479. The behavior of "git rev-parse --show-toplevel" has changed in Git 2.25.0, and it now fails in bare repositories.

Instead, use "git rev-parse --git-dir" to sanity-check the working copy. This appears to have more stable behavior across Git versions, although it's a little more complicated for our purposes.

Test Plan:
  - Ran `bin/repository update ...` on an observed, bare repository.
  - ...on an observed, non-bare ("legacy") repository.
  - ...on a hosted, bare repository.

Maniphest Tasks: T13479

Differential Revision: https://secure.phabricator.com/D20945
2020-01-16 11:39:23 -08:00
epriestley
f806528983 Allow the Herald Rule Editor to apply generic "Edge" transactions
Summary: Fixes T13469. Currently, "Mute" applies a generic edge transaction but the editor doesn't whitelist them.

Test Plan: Muted a Herald rule.

Maniphest Tasks: T13469

Differential Revision: https://secure.phabricator.com/D20943
2020-01-15 08:29:46 -08:00
epriestley
767528c0ed Move search query parser/compiler classes to Phabricator
Summary: Ref T13472. Ref T13395. These classes are only used by Phabricator and not likely to find much use in Arcanist.

Test Plan: Grepped libphutil and Arcanist for removed symbols.

Maniphest Tasks: T13472, T13395

Differential Revision: https://secure.phabricator.com/D20939
2020-01-14 11:49:49 -08:00
epriestley
54bcbdaba9 Fix an XSS issue with certain high-priority remarkup rules embedded inside lower-priority link rules
Summary:
See <https://hackerone.com/reports/758002>. The link rules don't test that their parameters are flat text before using them in unsafe contexts.

Since almost all rules are lower-priority than these link rules, this behavior isn't obvious. However, two rules have broadly higher priority (monospaced text, and one variation of link rules has higher priority than the other), and the latter can be used to perform an XSS attack with input in the general form `()[ [[ ... | ... ]] ]` so that the inner link rule is evaluated first, then the outer link rule uses non-flat text in an unsafe way.

Test Plan:
Tested examples in HackerOne report. A simple example of broken (but not unsafe) behavior is:

```
[[ `x` | `y` ]]
```

Differential Revision: https://secure.phabricator.com/D20937
2019-12-13 10:37:50 -08:00
Arturas Moskvinas
4cd333b33f Use same method to get object URI as used in DifferentialTransactionEditor and PhabricatorApplicationTransactionEditor
Summary: Maniphest object has `getURI` method, let's use it

Test Plan: Create event in task - URI generated as expected in email notification

Reviewers: epriestley, Pawka, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D20935
2019-12-10 17:37:30 +02:00
epriestley
33c534f9b7 Extend Config to full-width
Summary:
Ref T13362. Some applications moved to fixed-width a while ago but I was generally unsatisfied with where they ended up and have been pushing them back to full-width.

Push Config back to full-width. Some of the subpages end up a little weird, but this provides more space to work with to make some improvements, like makign `maniphest.statuses` more legible in the UI>

Test Plan: Grepped for `setFixed(`, updated each page in `/config/`. Browsed each controller, saw workable full-width UIs.

Maniphest Tasks: T13362

Differential Revision: https://secure.phabricator.com/D20925
2019-11-25 16:18:41 -08:00
epriestley
1667acfa5d Implement "PolicyInterface" on "UserEmail" so "EmailQuery" can load them properly
Summary:
See PHI1558. Ref T11860. Ref T13444. I partly implemented PHIDs for "UserEmail" objects, but they can't load on their own so you can't directly `bin/remove destroy` them yet.

Allow them to actually load by implementing "PolicyInterface".

Addresses are viewable and editable by the associated user, unless they are a bot/list address, in which case they are viewable and editable by administrators (in preparation for T11860). This has no real impact on anything today.

Test Plan:
  - Used `bin/remove destroy <phid>` to destroy an individual email address.
  - Before: error while loading the object by PHID in the query policy layer.
  - After: clean load and destroy.

Maniphest Tasks: T13444, T11860

Differential Revision: https://secure.phabricator.com/D20927
2019-11-25 15:08:13 -08:00
epriestley
eb6df7a209 Remove "phlog()" of exeptions during Conduit calls
Summary: Fixes T13465. This "phlog()" made some degree of sense at one time, but is no longer useful or consistent. Get rid of it. See T13465 for discussion.

Test Plan: Made a conduit call that hit a policy error, no longer saw error in log.

Maniphest Tasks: T13465

Differential Revision: https://secure.phabricator.com/D20924
2019-11-22 12:03:08 -08:00
epriestley
2abf292821 Fix an issue where editing paths in Owners packages could raise an error: undefined index "display"
Summary:
Fixes T13464. Fully-realized paths have a "path" (normalized, effective path) and a "display" path (user-facing, un-normalized path).

During transaction validation we build ref keys for paths before we normalize the "display" values. A few different approaches could be taken to resolve this, but just default the "display" path to the raw "path" if it isn't present since that seems simplest.

Test Plan: Edited paths in an Owners package, no longer saw a warning in the logs.

Maniphest Tasks: T13464

Differential Revision: https://secure.phabricator.com/D20923
2019-11-21 11:29:04 -08:00
epriestley
374f8b10b3 Add a "--dry-run" flag to "bin/repository rebuild-identities"
Summary: Ref T13444. Allow the effects of performing an identity rebuild to be previewed without committing to any changes.

Test Plan: Ran "bin/repository rebuild-identities --all-identities" with and without "--dry-run".

Maniphest Tasks: T13444

Differential Revision: https://secure.phabricator.com/D20922
2019-11-19 12:38:20 -08:00
epriestley
63d84e0b44 Improve use of keys when iterating over commits in "bin/audit delete" and "bin/repository rebuild-identities"
Summary:
Fixes T13457. Ref T13444. When we iterate over commits in a particular repository, the default iteration strategy can't effectively use the keys on the table.

Tweak the ordering so the "<repositoryID, epoch, [id]>" key can be used.

Test Plan:
  - Ran `bin/audit delete --repository X` and `bin/repository rebuild-identities --repository X` before and after changes.
    - With just the key changes, performance was slightly better. My local data isn't large enough to really emphasize the key changes.
    - With the page size changes, performance was a bit better (~30%, but on 1-3 second run durations).
  - Used `--trace` and ran `EXPLAIN ...` on the new queries, saw them select the "<repositoryID, epoch, [id]>" key and report a bare "Using index condition" in the "Extra" column.

Maniphest Tasks: T13457, T13444

Differential Revision: https://secure.phabricator.com/D20921
2019-11-19 10:18:55 -08:00
epriestley
a7aca500bc Update repository identities after all mutations to users and email addresses
Summary:
Ref T13444. Currently, many mutations to users and email addresses (particularly: user creation; and user and address destruction) do not propagate properly to repository identities.

Add hooks to all mutation workflows so repository identities get rebuilt properly when users are created, email addresses are removed, users or email addresses are destroyed, or email addresses are reassigned.

Test Plan:
- Added random email address to account, removed it.
- Added unassociated email address to account, saw identity update (and associate).
  - Removed it, saw identity update (and disassociate).
- Registered an account with an unassociated email address, saw identity update (and associate).
  - Destroyed the account, saw identity update (and disassociate).
- Added address X to account A, unverified.
  - Invited address X.
  - Clicked invite link as account B.
  - Confirmed desire to steal address.
  - Saw identity update and reassociate.

Maniphest Tasks: T13444

Differential Revision: https://secure.phabricator.com/D20914
2019-11-19 09:41:59 -08:00
epriestley
89dcf9792a Give "PhabricatorUserEmail" a PHID
Summary:
Ref T13444. To interact meaningfully with "DestructionEngine", objects need a PHID. The "UserEmail" object currently does not have one (or a real "Query").

Provide basic PHID support so "DestructionEngine" can interact with the object more powerfully.

Test Plan:
  - Ran migrations, checked data in database, saw sensible PHIDs assigned.
  - Added a new email address to my account, saw it get a PHID.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13444

Differential Revision: https://secure.phabricator.com/D20913
2019-11-19 09:41:29 -08:00
epriestley
d69a7360ea Use DestructionEngine to destroy UserEmail objects
Summary: Ref T13444. Prepare to hook identity updates when user email addreses are destroyed.

Test Plan:
  - Destroyed a user with `bin/remove destroy ... --trace`, saw email deleted.
  - Destroyed an email from the web UI, saw email deleted.

Maniphest Tasks: T13444

Differential Revision: https://secure.phabricator.com/D20912
2019-11-19 09:40:58 -08:00
epriestley
18da346972 Add additional flags to "bin/repository rebuild-identities" to improve flexibility
Summary:
Ref T13444. Repository identities have, at a minimum, some bugs where they do not update relationships properly after many types of email address changes.

It is currently very difficult to fix this once the damage is done since there's no good way to inspect or rebuild them.

Take some steps toward improving observability and providing repair tools: allow `bin/repository rebuild-identities` to effect more repairs and operate on identities more surgically.

Test Plan: Ran `bin/repository rebuild-identities` with all new flags, saw what looked like reasonable rebuilds occur.

Maniphest Tasks: T13444

Differential Revision: https://secure.phabricator.com/D20911
2019-11-19 09:39:48 -08:00
epriestley
0014d0404c Consolidate repository identity resolution and detection code
Summary: Ref T13444. Send all repository identity/detection through a new "DiffusionRepositoryIdentityEngine" which handles resolution and detection updates in one place.

Test Plan:
  - Ran `bin/repository reparse --message ...`, saw author/committer identity updates.
  - Added "goose@example.com" to my email addresses, ran daemons, saw the identity relationship get picked up.
  - Ran `bin/repository rebuild-identities ...`, saw sensible rebuilds.

Maniphest Tasks: T13444

Differential Revision: https://secure.phabricator.com/D20910
2019-11-19 09:39:11 -08:00
epriestley
6afbb6102d Remove "PhabricatorEventType::TYPE_DIFFUSION_LOOKUPUSER" event
Summary: Ref T13444. This is an ancient event and part of the old event system. It is not likely to be in use anymore, and repository identities should generally replace it nowadays anyway.

Test Plan: Grepped for constant and related methods, no longer found any hits.

Maniphest Tasks: T13444

Differential Revision: https://secure.phabricator.com/D20909
2019-11-19 09:38:03 -08:00
epriestley
a2b2c391a1 Distinguish between "Assigned" and "Effective" identity PHIDs more clearly and consistently
Summary:
Ref T13444. You can currently explicitly unassign an identity (useful if the matching algorithm is misfiring). However, this populates the "currentEffectiveUserPHID" with the "unassigned()" token, which mostly makes things more difficult.

When an identity is explicitly unassigned, convert that into an explicit `null` in the effective user PHID.

Then, realign "assigned" / "effective" language a bit. Previously, `withAssigneePHIDs(...)` actualy queried effective users, which was misleading. Finally, bulk up the list view a little bit to make testing slightly easier.

Test Plan:
  - Unassigned an identity, ran migration, saw `currentEffectiveUserPHID` become `NULL` for the identity.
  - Unassigned a fresh identity, saw NULL.
  - Queried for various identities under the modified constraints.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13444

Differential Revision: https://secure.phabricator.com/D20908
2019-11-19 09:37:44 -08:00
epriestley
df0f5c6cee Make repository identity email address association case-insensitive
Summary:
Ref T13444. Currently, identities for a particular email address are queried with "LIKE" against a binary column, which makes the query case-sensitive.

  - Extract the email address into a separate "sort255" column.
  - Add a key for it.
  - Make the query a standard "IN (%Ls)" query.
  - Deal with weird cases where an email address is 10000 bytes long or full of binary junk.

Test Plan:
  - Ran migration, inspected database for general sanity.
  - Ran query script in T13444, saw it return the same hits for "git@" and "GIT@".

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13444

Differential Revision: https://secure.phabricator.com/D20907
2019-11-19 09:37:26 -08:00
epriestley
d58eddcf0a When predicting project membership during edits, predict milestones will have parent membership
Summary:
Depends on D20919. Ref T13462. When editing milestones, we currently predict they will have no members for policy evaluation purposes. This isn't the right rule.

Instead, predict that their membership will be the same as the parent project's membership, and pass this hint to the policy layer.

See T13462 for additional context and discussion.

Test Plan:
  - Set project A's edit policy to "Project Members".
  - Joined project A.
  - Tried to create a milestone of project A.
    - Before: policy exception that the edit policy excludes me.
    - After: clean milestone creation.
  - As a non-member, tried to create a milestone. Received appropriate policy error.

Maniphest Tasks: T13462

Differential Revision: https://secure.phabricator.com/D20920
2019-11-18 23:07:05 -08:00
epriestley
959504a488 When predicting object policies for project milestones, adjust objects so they behave like milestones
Summary:
Ref T13462. Currently, when testing milestone edit policies during creation, the project object does not behave like a milestone:

  - it doesn't have a milestone number yet, so it doesn't try to access the parent project; and
  - the parent project isn't attached yet.

Instead: attach the parent project sooner (which "should" be harmless, although it's possible this has weird side effects); and give the adjusted policy object a dummy milestone number if it doesn't have one yet. This forces it to act like a milestone when emitting policies.

Test Plan:
  - Set "Projects" application default edit policy to "No One".
  - Created a milestone I had permission to create.
    - Before: failed with a policy error, because the project behaved like a non-milestone and returned "No One" as the effective edit policy.
    - After: worked properly, correctly evaluting the parent project edit policy as the effective edit policy.
  - Tried to create a milestone I did not have permission to create (no edit permission on parent project).
    - Got an appropriate edit policy error.

Maniphest Tasks: T13462

Differential Revision: https://secure.phabricator.com/D20919
2019-11-18 23:06:03 -08:00
epriestley
de66a8ece1 Remove "stronger/weaker" policy color hints from object headers
Summary:
Fixes T13461. Some applications provide hints about policy strength in the header, but these hints are inconsistent and somewhat confusing. They don't make much sense for modern objects with Custom Forms, which don't have a single "default" policy.

Remove this feature since it seems to be confusing things more than illuminating them.

Test Plan:
  - Viewed various objects, no longer saw colored policy hints.
  - Grepped for all removed symbols.

Maniphest Tasks: T13461

Differential Revision: https://secure.phabricator.com/D20918
2019-11-18 22:05:26 -08:00
epriestley
1996b0cd55 Update the "owner can always view/edit" policy exception rule
Summary: Fixes T13460. This rule vanished from the UI in D20165; update things so it returns to the UI.

Test Plan: {F7035134}

Maniphest Tasks: T13460

Differential Revision: https://secure.phabricator.com/D20917
2019-11-18 21:54:45 -08:00
epriestley
e86aae99de Surface edits to "Text" panels on dashboards as remarkup edits
Summary: Fixes T13456. These edits are remarkup edits and should attach files, trigger mentions, and so on.

Test Plan: Created a text panel, dropped a file in. After changes, saw the file attach properly.

Maniphest Tasks: T13456

Differential Revision: https://secure.phabricator.com/D20906
2019-11-13 10:35:09 -08:00
epriestley
2adc36ba0b Correctly identify more SSH private key problems as "formatting" or "passphrase" related
Summary:
Ref T13454. Fixes T13006. When a user provide us with an SSH private key and (possibly) a passphrase:

  # Try to verify that they're correct by extracting the public key.
  # If that fails, try to figure out why it didn't work.

Our success in step (2) will vary depending on what the problem is, and we may end up falling through to a very generic error, but the outcome should generally be better than the old approach.

Previously, we had a very unsophisticated test for the text "ENCRYPTED" in the key body and questionable handling of the results: for example, providing a passphrase when a key did not require one did not raise an error.

Test Plan:
Created and edited credentials with:

  - Valid, passphrase-free keys.
  - Valid, passphrased keys with the right passphrase.
  - Valid, passphrase-free keys with a passphrase ("surplus passphrase" error).
  - Valid, passphrased keys with no passphrase ("missing passphrase" error).
  - Valid, passphrased keys with an invalid passphrase ("invalid passphrase" error).
  - Invalid keys ("format" error).

The precision of these errors will vary depending on how helpful "ssh-keygen" is.

Maniphest Tasks: T13454, T13006

Differential Revision: https://secure.phabricator.com/D20905
2019-11-13 10:22:00 -08:00
epriestley
72f82abe07 Improve recovery from panel action rendering exceptions, and mark "Changeset" queries as not suitable for panel generation
Summary:
Fixes T13443. When a panel raises an exception during edit action generation, it currently escapes to top level. Instead, catch it more narrowly.

Additionally, mark "ChangesetSearchEngine" as not being a suitable search engine for use in query panels. There's no list view or search URI so it can't generate a sensible panel.

Test Plan:
  - Added a changeset panel to a dashboard.
  - Before: entire dashboard fataled.
  - After: panel fataled narrowly, menu fatals narrowly, dashboard no longer permits creation of another Changeset query panel.

Maniphest Tasks: T13443

Differential Revision: https://secure.phabricator.com/D20902
2019-11-08 17:15:21 -08:00
epriestley
a3f4cbd748 Correct rendering of workboard column move stories when a single transaction performs moves on multiple boards
Summary:
See <https://discourse.phabricator-community.org/t/unhandled-exception-rendering-maniphest-task/3234>.

If a single transaction performs column moves on multiple different boards (which is permitted in the API), the rendering logic currently fails. Make it render properly.

Test Plan: {F7011464}

Differential Revision: https://secure.phabricator.com/D20901
2019-11-08 16:57:35 -08:00
epriestley
b83b3224bb Add an "Advanced/Developer..." action item for viewing object handle details and hovercards
Summary:
Ref T13442. Ref T13157. There's a secret URI to look at an object's hovercard in a standalone view, but it's hard to remember and impossible to discover.

In developer mode, add an action to "View Hovercard". Also add "View Handle", which primarily shows the object PHID.

Test Plan: Viewed some objects, saw "Advanced/Developer...". Used "View Hovercard" to view hovercards and "View Handle" to view handles.

Maniphest Tasks: T13442, T13157

Differential Revision: https://secure.phabricator.com/D20887
2019-11-08 16:47:05 -08:00
epriestley
cd60a8aa56 Update various Asana odds-and-ends for "gid" API changes
Summary:
Ref T13453. Some of the Asana integrations also need API updates.

Depends on D20899.

Test Plan:
  - Viewed "asana.workspace-id" in Config, got a sensible GID list.
  - Created a revision, saw the associated Asana task get assigned.
  - Pasted an Asana link I could view into a revision description, saw it Doorkeeper in the metadata.

Maniphest Tasks: T13453

Differential Revision: https://secure.phabricator.com/D20900
2019-11-08 09:07:59 -08:00
epriestley
2223d6b914 Update Asana Auth adapter for "gid" API changes
Summary: Ref T13453. The Asana API has changed, replacing all "id" fields with "gid", including the "users/me" API call result.

Test Plan: Linked an Asana account. Before: error about missing 'id'. After: clean link.

Maniphest Tasks: T13453

Differential Revision: https://secure.phabricator.com/D20899
2019-11-08 09:02:15 -08:00
epriestley
338b4cb2e7 Prevent workboard cards from being grabbed by the "Txxx" object name text
Summary:
Fixes T13452. We currently give users mixed signals about the interaction mode of this text: the cursor says "text" but the behavior is "grab".

Make the behavior "text" to align with the cursor. An alternate variation of this change is to remove the cursor, but this is preferable if it doesn't cause problems, since copying the task ID is at least somewhat useful.

Test Plan: In Safari, Firefox, and Chrome: selected and copied object names from workboard cards; and dragged workboard cards by other parts of their UI.

Maniphest Tasks: T13452

Differential Revision: https://secure.phabricator.com/D20898
2019-11-08 08:29:53 -08:00
epriestley
d4491ddc22 Fix an issue with 1up diff block rendering for added or removed blocks
Summary: Ref T13425. When a change adds or removes a block (vs adding or removing a document, or changing a block), we currently try to render `null` as cell content in the unified view. Make this check broader to catch both "no opposite document" and "no opposite cell".

Test Plan: {F7008772}

Subscribers: artms

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20897
2019-11-08 07:37:06 -08:00
epriestley
502ca4767e When "Fetch Refs" are configured for a repository, highlight the "Branches" menu item in the Diffusion Management UI
Summary: Ref T13448. Minor UI issue: setting a "Fetch Refs" value does not highlight the associated menu item, but should.

Test Plan: Set only "Fetch Refs", now saw menu item highlighted.

Maniphest Tasks: T13448

Differential Revision: https://secure.phabricator.com/D20895
2019-11-07 16:28:33 -08:00
epriestley
1b40f7e540 Always initialize Git repositories with "git init", never with "git clone"
Summary:
Fixes T13448. We currently "git clone" to initialize repositories, but this will fetch too many refs if "Fetch Refs" is configured.

In modern Phabricator, there's no apparent reason to "git clone"; we can just "git init" instead. This workflow naturally falls through to an update, where we'll do a "git fetch" and pull in exactly the refs we want.

Test Plan:
  - Configured an observed repository with "Fetch Refs".
  - Destroyed the working copy.
  - Ran "bin/repository pull X --trace --verbose".
    - Before: saw "git clone" pull in the world.
    - After: saw "git init" create a bare empty working copy, then "git fetch" fill it surgically.

Both flows end up in the same place, this one is just simpler and does less work.

Maniphest Tasks: T13448

Differential Revision: https://secure.phabricator.com/D20894
2019-11-07 16:17:35 -08:00
epriestley
8dd57a1ed2 When fetching Git repositories, pass "--no-tags" to make explicit "Fetch Refs" operate more narrowly
Summary:
Ref T13448. The default behavior of "git fetch '+refs/heads/master:refs/heads/master'" is to follow and fetch associated tags.

We don't want this when we pass a narrow refspec because of "Fetch Refs" configuration. Tell Git that we only want the refs we explicitly passed.

Note that this doesn't prevent us from fetching tags (if the refspec specifies them), it just stops us from fetching extra tags that aren't part of the refspec.

Test Plan:
  - Ran "bin/repository pull X --trace --verbose" in a repository with a "Fetch Refs" configuration, saw only "master" get fetched (previously: "master" and reachable tags).
  - Ran "git fetch --no-tags '+refs/*:refs/*'", saw tags fetched normally ("--no-tags" does not disable fetching tags).

Maniphest Tasks: T13448

Differential Revision: https://secure.phabricator.com/D20893
2019-11-07 16:17:08 -08:00
epriestley
9dbde24dda Manually prune Git working copy refs instead of using "--prune", to improve "Fetch Refs" behavior
Summary:
Ref T13448. When "Fetch Refs" is configured:

  - We switch to a narrow mode when running "ls-remote" against the local working copy. This excludes surplus refs, so we'll incorrectly detect that the local and remote working copies are identical in cases where the local working copy really has surplus refs.
  - We rely on "--prune" to remove surplus local refs, but it only prunes refs matching the refspecs we pass "git fetch". Since these refspecs are very narrow under "Fetch Only", the pruning behavior is also very narrow.

Instead:

  - When listing local refs, always list everything. If we have too much stuff locally, we want to get rid of it.
  - When we identify surplus local refs, explicitly delete them instead of relying on "--prune". We can just do this in all cases so we don't have separate "--prune" and "manual" cases.

Test Plan:
  - Created a new repository, observed from a GitHub repository, with many tags/refs/branches. Pulled it.
  - Observed lots of refs in `git for-each-ref`.
  - Changed "Fetch Refs" to "refs/heads/master".
  - Ran `bin/repository pull X --trace --verbose`.

On deciding to do something:

  - Before: since "master" did not change, the pull declined to act.
  - After: the pull detected surplus refs and deleted them. Since the states then matched, it declined further action.

On pruning:

  - Before: if the pull was forced to act, it ran "fetch --prune" with a narrow refspec, which did not prune the working copy.
  - After: saw working copy pruned explicitly with "update-ref -d" commands.

Also, set "Fetch Refs" back to the default (empty) and pulled, saw everything pull.

Maniphest Tasks: T13448

Differential Revision: https://secure.phabricator.com/D20892
2019-11-07 16:15:46 -08:00
epriestley
f5f2a0bc56 Allow username changes which modify letter case to go through as valid
Summary:
Fixes T13446. Currently, the validation logic here rejects a rename like "alice" to "ALICE" (which changes only letter case) but this is a permissible rename.

Allow collisions that collide with the same user to permit this rename.

Also, fix an issue where an empty rename was treated improperly.

Test Plan:
  - Renamed "alice" to "ALICE".
    - Before: username collision error.
    - After: clean rename.
  - Renamed "alice" to "orange" (an existing user). Got an error.
  - Renamed "alice" to "", "!@#$", etc (invalid usernames). Got sensible errors.

Maniphest Tasks: T13446

Differential Revision: https://secure.phabricator.com/D20890
2019-11-06 08:10:36 -08:00
epriestley
6bada7db4c Change the Herald "do not include [any of]" condition label to "include none of"
Summary: Fixes T13445. Make the meaning of this condition more clear, since the current wording is ambiguous between "any of" and "all of".

Test Plan: Edited a Herald rule with a PHID list field ("Project tags"), saw text label say "include none of".

Maniphest Tasks: T13445

Differential Revision: https://secure.phabricator.com/D20889
2019-11-06 07:55:53 -08:00
epriestley
be2b8f4bcb Support querying projects by "Root Projects" in the UI, and "min/max depth" in the API
Summary:
Fixes T13441. Internally, projects can be queried by depth, but this is not exposed in the UI.

Add a "Is root project?" contraint in the UI, and "minDepth" / "maxDepth" constraints to the API.

Test Plan:
  - Used the UI to query root projects, got only root projects back.
  - Used "project.search" in the API to query combinations of root projects and projects at particular depths, got matching results.

Maniphest Tasks: T13441

Differential Revision: https://secure.phabricator.com/D20886
2019-10-31 12:56:33 -07:00
epriestley
e46e383bf2 Clean up "Revisions/Commits" table in Maniphest slightly
Summary: Ref T13440. Give the table more obvious visual structure and get rid of the largely useless header columns.

Test Plan: Viewed table, saw a slightly cleaner result.

Maniphest Tasks: T13440

Differential Revision: https://secure.phabricator.com/D20885
2019-10-31 12:29:53 -07:00
epriestley
c48f300eb1 Add support for rendering section dividers in tables; use section dividers for changes on tasks
Summary: Depends on D20883. Ref T13440. In most cases, all changes belong to the same repository, which makes the "Repository" column redundant and visually noisy. Show repository information in a section header.

Test Plan: {F6989932}

Maniphest Tasks: T13440

Differential Revision: https://secure.phabricator.com/D20884
2019-10-31 12:13:25 -07:00
epriestley
7bdfe5b46a Show commits and revisions on tasks in a tabular view instead of handle lists
Summary: Depends on D20882. Ref T13440. Instead of lists of "Differential Revisions" and "Commits", show all changes related to the task in a tabular view.

Test Plan: {F6989816}

Maniphest Tasks: T13440

Differential Revision: https://secure.phabricator.com/D20883
2019-10-31 12:10:09 -07:00
epriestley
b0d9f89c95 Remove "State Icons" from handles
Summary: Ref T13440. This feature is used in only one interface which I'm about to rewrite, so throw it away.

Test Plan: Grepped for all affected symbols, didn't find any hits anywhere.

Maniphest Tasks: T13440

Differential Revision: https://secure.phabricator.com/D20882
2019-10-31 12:04:43 -07:00
epriestley
97bed35085 Show repository information (and use repository identities) in commit hovercards
Summary:
Ref T12164. Ref T13439. Commit hovercards don't currently show the repository. Although this is sometimes obvious from context, it isn't at other times and it's clearly useful/important.

Also, use identities to render author/committer information and show committer if the committer differs from the author.

Test Plan: {F6989595}

Maniphest Tasks: T13439, T12164

Differential Revision: https://secure.phabricator.com/D20881
2019-10-31 09:58:20 -07:00
Arturas Moskvinas
bcf15abcd3 Return empty data if fact dimension is missing, not yet available
Summary:
On fresh installation which doesn't have yet any task closed you will not be able to open charts because of error below:
Fixes:
```
[Mon Oct 21 15:42:41 2019] [2019-10-21 15:42:41] EXCEPTION: (TypeError) Argument 1 passed to head_key() must be of the type array, null given, called in ..phabricator/src/applications/fact/chart/PhabricatorFactChartFunction.php on line 86 at [<phutil>/src/utils/utils.php:832]
[Mon Oct 21 15:42:41 2019]   #0 phlog(TypeError) called at [<phabricator>/src/aphront/handler/PhabricatorAjaxRequestExceptionHandler.php:27]
[Mon Oct 21 15:42:41 2019]   #1 PhabricatorAjaxRequestExceptionHandler::handleRequestThrowable(AphrontRequest, TypeError) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:797]
[Mon Oct 21 15:42:41 2019]   #2 AphrontApplicationConfiguration::handleThrowable(TypeError) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:345]
[Mon Oct 21 15:42:41 2019]   #3 AphrontApplicationConfiguration::processRequest(AphrontRequest, PhutilDeferredLog, AphrontPHPHTTPSink, MultimeterControl) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:214]
[Mon Oct 21 15:42:41 2019]   #4 AphrontApplicationConfiguration::runHTTPRequest(AphrontPHPHTTPSink) called at [<phabricator>/webroot/index.php:35
```

To fix issue - lets return empty data set instead

Test Plan:
1) Create fresh phabricator installation
2) Create fresh project
3) Try viewing charts

Reviewers: epriestley, Pawka, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, yelirekim

Differential Revision: https://secure.phabricator.com/D20861
2019-10-30 16:34:22 +02:00
epriestley
9d8cdce8e1 Make the top-level burndown chart in "Maniphest > Reports" show open tasks, not total tasks
Summary:
Ref T13279. See PHI1491. Currently, the top-level "Burnup Rate" chart in Maniphest shows total created tasks above the X-axis, without adjusting for closures.

This is unintended and not very useful. The filtered-by-project charts show the right value (cumulative open tasks, i.e. open minus close). Change the value to aggregate creation events and status change events.

Test Plan: Viewed top-level chart, saw the value no longer monotonically increasing.

Maniphest Tasks: T13279

Differential Revision: https://secure.phabricator.com/D20879
2019-10-29 13:48:43 -07:00
epriestley
114166dd32 Roughly implement "harbormaster.artifact.search"
Summary: Ref T13438. This is a sort of minimal plausible implementation.

Test Plan: Used "harbormaster.artifact.search" to query information about artifacts.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13438

Differential Revision: https://secure.phabricator.com/D20878
2019-10-29 13:37:35 -07:00
epriestley
5d8457a07e In the repository URI index, store Phabricator's own URIs as tokens
Summary:
Fixes T13435. If you move Phabricator or copy data from one environment to another, the repository URI index currently still references the old URI, since it writes the URI as a plain string. This may make "arc which" and similar workflows have difficulty identifying repositories.

Instead, store the "phabricator.base-uri" domain and the "diffusion.ssh-host" domain as tokens, so lookups continue to work correctly even after these values change.

Test Plan:
  - Added unit tests to cover the normalization.
  - Ran migration, ran daemons, inspected `repository_uriindex` table, saw a mixture of sensible tokens (for local domains) and static domains (like "github.com").
  - Ran this thing:

```
$ echo '{"remoteURIs": ["ssh://git@local.phacility.com/diffusion/P"]}' | ./bin/conduit call --method repository.query --trace --input -
Reading input from stdin...
>>> [2] (+0) <conduit> repository.query()
>>> [3] (+3) <connect> local_repository
<<< [3] (+3) <connect> 555 us
>>> [4] (+5) <query> SELECT `r`.* FROM `repository` `r` LEFT JOIN `local_repository`.`repository_uriindex` uri ON r.phid = uri.repositoryPHID WHERE (uri.repositoryURI IN ('<base-uri>/diffusion/P')) GROUP BY `r`.phid ORDER BY `r`.`id` DESC LIMIT 101
<<< [4] (+5) <query> 596 us
<<< [2] (+6) <conduit> 6,108 us
{
  "result": [
    {
      "id": "1",
      "name": "Phabricator",
      "phid": "PHID-REPO-2psrynlauicce7d3q7g2",
      "callsign": "P",
      "monogram": "rP",
      "vcs": "git",
      "uri": "http://local.phacility.com/source/phabricator/",
      "remoteURI": "https://github.com/phacility/phabricator.git",
      "description": "asdf",
      "isActive": true,
      "isHosted": false,
      "isImporting": false,
      "encoding": "UTF-8",
      "staging": {
        "supported": true,
        "prefix": "phabricator",
        "uri": null
      }
    }
  ]
}
```

Note the `WHERE` clause in the query normalizes the URI into "<base-uri>", and the lookup succeeds.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13435

Differential Revision: https://secure.phabricator.com/D20872
2019-10-28 14:44:06 -07:00
epriestley
292f8fc612 Fix an issue where added or removed source files could incorrectly select a DocumentEngine
Summary:
Ref T13425. The changes in D20865 could incorrectly lead to selection of a DocumentEngine that can not generate document diffs if a file was added or removed (for example, when a source file is added).

Move the engine pruning code to be shared -- we should always discard engines which can't generate a diff, even if we don't have both documents.

Test Plan: Viewed an added source file, no more document ref error arising from document engine selection.

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20866
2019-10-26 12:16:49 -07:00
epriestley
8ff0e3ab35 Support rich diff rendering with DocumentEngine for added/removed files
Summary: Ref T13425. When a file (like a Jupyter notebook) is added or removed, we can still render a useful line-by-line diff.

Test Plan:
  - Viewed add/modify/remove of Jupyter, source code, and images in 2up/1up mode, everything looked okay.

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20865
2019-10-26 08:21:46 -07:00
epriestley
38694578e1 Improve project member list behaviors related to disabled users
Summary:
Fixes T13431.

Increase the "panel" version of project member lists to 10 users, hide disabled users, swap the buttons to "tail buttons".

Sort disabled users to the bottom of "full list" versions of member lists.

For UI consistency, render the remove "X" as disabled but visible if users don't have permission to remove members.

Test Plan:
  - Viewed a project with disabled members.
    - Saw only enabled members on the main project page.
    - Saw disabled members sorted to the bottom on the members page.
  - Clicked "View All" to jump from the panel to the members page.
  - As a user who could not edit a project, viewed the members page and saw a disabled "X" with a policy error when clicked.
    - Removed a member as before, as a normal user with permission to remove members.

Maniphest Tasks: T13431

Differential Revision: https://secure.phabricator.com/D20864
2019-10-24 18:41:33 -07:00
epriestley
633aa5288c Persist login instructions onto flow-specific login pages (username/password and LDAP)
Summary:
Fixes T13433. Currently, "Login Screen Instructions" in "Auth" are shown only on the main login screen. If you enter a bad password or bad LDAP credential set and move to the flow-specific login failure screen (for example, "invalid password"), the instructions vanish.

Instead, persist them. There are reasonable cases where this is highly useful and the cases which spring to mind where this is possibly misleading are fairly easy to fix by making the instructions more specific.

Test Plan:
  - Configured login instructions in "Auth".
  - Viewed main login screen, saw instructions.
  - Entered a bad username/password and a bad LDAP credential set, got kicked to workflow sub-pages and still saw instructions (previously: no instructions).
  - Grepped for other callers to `buildProviderPageResponse()` to look for anything weird, came up empty.

Maniphest Tasks: T13433

Differential Revision: https://secure.phabricator.com/D20863
2019-10-24 18:38:15 -07:00
epriestley
5b36f0c97a Add default branch, description, and metrics (commit count, recent commit) to "diffusion.repository.search"
Summary: Fixes T13430. Provide more information about repositories in "diffusion.repository.search".

Test Plan: Used API console to call method (with new "metrics" attachment), reviewed output. Saw new fields returned.

Maniphest Tasks: T13430

Differential Revision: https://secure.phabricator.com/D20862
2019-10-24 17:16:58 -07:00
epriestley
f497b93e43 Fix a fatal in the "Projects" curtain extension when a project edge connects an object to a non-project
Summary:
Ref T13429. It's currently possible to write "TYPE_EDGE" relationships for the "object has project" edge to PHIDs which may not actually be projects. Today, this fatals.

As a first step, unfatal it. T13429 discusses general improvements and greater context.

Test Plan:
Used "maniphest.edit" to write a "project" edge to a user PHID, viewed the task in the UI. Previously it fataled; now it renders unusually (the object is "tagged" with a user) but faithfully reflects database state.

{F6957606}

Maniphest Tasks: T13429

Differential Revision: https://secure.phabricator.com/D20860
2019-10-17 09:49:01 -07:00
epriestley
d34dfa3746 Fix an error message when calling "transaction.search" with a non-transactional object PHID as an "objectIdentifier"
Summary: See PHI1499. This error message doesn't provide parameters, and can be a little bit more helpful.

Test Plan: {F6957550}

Differential Revision: https://secure.phabricator.com/D20859
2019-10-17 09:19:54 -07:00
epriestley
5dafabd5b4 Fix deprecated argument order for "implode()"
Summary: Fixes T13428. In modern PHP, "implode()" should take the glue parameter first.

Test Plan:
Used the linter introduced in D20857 to identify affected callsites.

```
$ git grep -i implode | cut -d: -f1 | sort | uniq | xargs arc lint --output summary --never-apply-patches | grep -i glue
```

Maniphest Tasks: T13428

Differential Revision: https://secure.phabricator.com/D20858
2019-10-17 09:11:27 -07:00
epriestley
7badec23a7 Use "git log ... --stdin" instead of "git log ... --not ..." to avoid oversized command lines
Summary:
Depends on D20853. See PHI1474. If the list of "--not" refs is sufficiently long, we may exceed the maximum size of a command.

Use "--stdin" instead, and swap "--not" for the slightly less readable but functionally equivalent "^hash", which has the advantage of actually working with "--stdin".

Test Plan: Ran `bin/repository refs ...` with nothing to be done, and with something to be done.

Differential Revision: https://secure.phabricator.com/D20854
2019-10-02 14:08:15 -07:00
epriestley
6d74736a7e When a large number of commits need to be marked as published, issue the lookup query in chunks
Summary: See PHI1474. This query can become large enough to exceed reasonable packet limits. Chunk the query so it is split up if we have too many identifiers.

Test Plan: Ran `bin/repository refs ...` on a repository with no new commits and a repository with some new commits.

Differential Revision: https://secure.phabricator.com/D20853
2019-10-02 14:06:07 -07:00
Arturas Moskvinas
960c447aab Support more than 9 portals
Summary: If portal is created with id > 9 - then those portals are not reachable and always return 404.

Test Plan: Create at least 10 portals, 10th and rest of them will no longer return 404.

Reviewers: epriestley, Pawka, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D20852
2019-10-02 22:35:59 +03:00
epriestley
344a2e39be In Jupyter notebooks, apply intraline diffing to source code lines
Summary: Ref T13425. When we render a diff between two source lines, highlight intraline changes.

Test Plan: Viewed a Jupyter notebook with a code diff in it, saw the changed subsequences in the line highlighted.

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20851
2019-10-02 12:34:59 -07:00
epriestley
9f7aaa8ee4 Fix an issue where any diff which could possibly be rendered as Jupyter decided to render as Jupyter
Summary:
See PHI1468. Engine selection for diffs is currently too aggressive in trying to find a shared engine and will fall back a shared engine with a very low score, causing all ".json" files to render as Jupyter files.

Only pick an engine as a difference engine by default if it's the highest-scoring engine for the new file.

Test Plan: Viewed ".json" files and ".ipynb" files in a revision. Before, both rendered as Jupyter. Now, the former rendered as JSON and the latter rendered as Jupyter.

Differential Revision: https://secure.phabricator.com/D20850
2019-10-01 18:52:36 -07:00
epriestley
76d9912932 Force unified abstract block diffs into roughly usable shape
Summary: Depends on D20845. Ref T13425. In unified mode, render blocks diffs less-unreasonably.

Test Plan: {F6910436}

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20848
2019-09-30 10:44:07 -07:00
epriestley
d9515e82a3 Perform basic block interdiffs when diffing abstract blocks, and interdiff markdown in Jupyter notebooks
Summary:
Depends on D20844. Ref T13425. When we line up two blocks and they can be interdiffed (generally: they both have the same type of content), let the Engine interdiff them.

Then, make the Jupyter engine interdiff markdown.

Test Plan: {F6898583}

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20845
2019-09-30 10:43:15 -07:00
epriestley
5afdc620db Make basic Juypter notebook rendering improvements and roughly support folding unchanged context
Summary:
Depends on D20843. Ref T13425. Add very basic support for "Show Hidden Context", in the form of folding it behind an unclickable shield. This isn't ideal, but should be better than nothing.

Prepare for "intraline" diffs on content blocks.

Fix newline handling in Markdown sections in Jupyter notebooks.

Remove the word "visibile" from the codebase.

Test Plan: {F6898192}

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20844
2019-09-30 10:41:21 -07:00
epriestley
a7f3316aa3 Improve sequencing of various content/header checks in abstract block diffs
Summary:
Ref T13425. Some diff checks currently sequence incorrectly:

  - When we're rendering block lists, syntax highlighting isn't relevant.
  - The "large change" guard can prevent rendering of otherwise-renderable changes.
  - Actual errors in the document engine (like bad JSON in a ".ipynb" file) aren't surfaced properly.

Improve sequencing somewhat to resolve these issues.

Test Plan:
  - Viewed a notebook, no longer saw a "highlighting disabled" warning.
  - Forced a notebook to fail, got a useful inline error instead of a popup dialog error.
  - Forced a notebook to have a large number of differences, got a rendering out of it.

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20843
2019-09-30 10:40:12 -07:00
epriestley
9944b669ad Make HTTP/403 say "Yikes!"
Summary: This text is significantly more clear and helpful for users.

Test Plan: Tried to do something I'm not suppposed to, hit the 403 page.

Differential Revision: https://secure.phabricator.com/D20847
2019-09-30 09:27:39 -07:00
epriestley
f004b76465 Add a "Subtype" tag to the task graph view in Maniphest
Summary: See PHI1466. When an install defines task subtypes, show them on the task graph.

Test Plan:
  - On desktop with subtypes defined, column is visible.
  - On desktop with subtypes not defined, column is hidden.
  - On mobile, column is hidden.

{F6896845}

Differential Revision: https://secure.phabricator.com/D20842
2019-09-27 10:58:55 -07:00
epriestley
7a0090f4d0 Fix an issue where the "viewer" is not passed to Bulk Edit controls properly
Summary:
See PHI1442. If you have a bulk-editable datasource field with a composite datasource, it can currently fatal on the bulk edit workflow because the viewer is not passed correctly.

The error looks something like this:

> Argument 1 passed to PhabricatorDatasourceEngine::setViewer() must be an instance of PhabricatorUser, null given, called in /Users/epriestley/dev/core/lib/phabricator/src/applications/typeahead/datasource/PhabricatorTypeaheadCompositeDatasource.php on line 231

Test Plan: Configured a Maniphest custom field with a composite datasource, then tried a bulk edit. Things worked cleanly instead of fataling.

Differential Revision: https://secure.phabricator.com/D20841
2019-09-26 12:03:49 -07:00
epriestley
2c06815edb When rendering Jupyter notebook diffs, split code inputs into individual blocks
Summary:
Ref T13425. Currently, code inputs and all outputs are grouped into a single block. This is fine for display notebooks but not great for diffing notebooks.

Instead, split source code input into individual lines with one line per block, and each output into its own block.

This allows you to leave actual line-by-line inlines on source, and comment on outputs individually.

Test Plan: {F6888583}

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20840
2019-09-25 21:05:18 -07:00
epriestley
884cd74cc4 In prose diffs, use hash-and-diff for coarse "level 0" diffing to scale better
Summary: Depends on D20838. Fixes T13414. Instead of doing coarse diffing with "PhutilEditDistanceMatrix", use hash-and-diff with "DocumentEngine".

Test Plan:
  - On a large document (~3K top level blocks), saw a more sensible diff, instead of the whole thing falling back to "everything changed" mode.
  - On a small document, still saw a sensible granular diff.

{F6888249}

Maniphest Tasks: T13414

Differential Revision: https://secure.phabricator.com/D20839
2019-09-25 16:50:49 -07:00
epriestley
9d884f144f Add "PhutilProseDiff" classes to "phabricator/"
Summary: Depends on D20836. Ref T13414. Ref T13425. Ref T13395. Move these to "phabricator/" before trying to improve the high-level diff engine in prose diffs.

Test Plan: Ran "arc liberate", looked at a prose diff (no behavioral change).

Maniphest Tasks: T13425, T13414, T13395

Differential Revision: https://secure.phabricator.com/D20838
2019-09-25 16:49:54 -07:00
epriestley
281598d65c Use a hash-and-diff strategy to produce a diff layout for block-based documents
Summary:
Depends on D20835. Ref T13425. Ref T13414. When a document has a list of content blocks, we may not be able to diff it directly, but we can hash each block and then diff the hashes (internally "diff" also does approximately the same thing).

We could do this ourselves with slightly fewer layers of indirection, but: diff already exists; we already use it; we already have a bunch of abstractions on top of it; and it's likely much faster on large inputs than the best we can do in PHP.

Test Plan: {F6888169}

Maniphest Tasks: T13425, T13414

Differential Revision: https://secure.phabricator.com/D20836
2019-09-25 16:40:53 -07:00
epriestley
932d829af3 Improve behavior of inline comment highlight reticle for block diffs
Summary:
Depends on D20834. Ref T13425. After the change from "th" to "td" for accessibility, the algorithm picks which cells it should highlight slightly improperly (it picks too many cells since it can no longer find the line numbers).

Ideally, it would probably highlight //only// the source content, but there isn't an easy way to do this right now. Settle for an incremental improvement for the moment.

Test Plan: Hovered over line numbers, saw a more accurate highlight area.

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20835
2019-09-25 16:39:18 -07:00
epriestley
a09b298d85 Correct DOM node metadata to let inline comments work against block-based diffs
Summary: Depends on D20833. Ref T13425. This look like it "just works"?

Test Plan: Left inline comments on a Juptyer notebook. Nothing seemed broken? Confusing.

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20834
2019-09-25 16:38:06 -07:00
epriestley
1c4450d39f Allow the Jupyter engine to elect to emit diffs, and emit Jupyter documents as blocks
Summary:
Depends on D20832. Ref T13425. Emit Jupyter notebooks as diffable blocks with block keys.

No diffing or proper inlines yet.

Test Plan: {F6888058}

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20833
2019-09-25 16:32:36 -07:00
epriestley
7ae711ed3e Add a "View as..." option to diff dropdowns for selecting between document engines
Summary:
Depends on D20831. Ref T13425. As an escape hatch to get out of future DocumentEngine rendering behavior, provide a "View As.." option.

Now I can break DocumentEngine real bad and no one can complain.

Test Plan: Used "View As" to swap document engines for image files.

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20832
2019-09-25 16:29:21 -07:00
epriestley
bb71ef6ad6 Render image diffs as abstract blocks diffs via DocumentEngine
Summary:
Depends on D20830. Ref T13425. Have the image engine elect into block rendering, then emit blocks.

This is rough (the blocks aren't actually diffed yet) but image diffs were already pretty rough so this is approximately a net improvement.

Test Plan: Viewed image diffs, saw nothing worse than before.

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20831
2019-09-25 16:25:06 -07:00
epriestley
a73f592d7d Allow DocumentEngine to elect into diff construction
Summary:
Ref T13425. Allow DocumentEngines to claim they can produce diffs. If both sides of a change can be diffed by the same document engine and it can produce diffs, have it diff them.

This has no impact on runtime behavior because no upstream engine elects into diff generation yet.

Test Plan: Loaded some revisions, nothing broke.

Maniphest Tasks: T13425

Differential Revision: https://secure.phabricator.com/D20830
2019-09-25 16:23:06 -07:00
epriestley
06edcf2709 Fix an issue where ancestors of permanent refs might not be published during import or if a branch is later made permanent
Summary:
Fixes T13284. See that task for substantial discussion. There are currently two cases where we'll skip over commits which we should publish:

  - if a branch is not permanent, then later made permanent; or
  - in some cases, the first time we examine branches in a repository.

In both cases, this error is one-shot and things work correctly going forward. The root cause is conflation between the states "this ref currently permanent" and "this ref was permanent the last time we updated refs".

Separate these pieces of state and cover all these cases. Also introduce a "--rebuild" flag to fix the state of bad commits.

Test Plan:
See T13284 for the three major cases:

  - initial import;
  - push changes to a nonpermanent branch, update, then make it permanent;
  - push chanegs to a nonpermanent branch, update, push more changes, then make it permanent.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13284

Differential Revision: https://secure.phabricator.com/D20829
2019-09-25 08:54:50 -07:00
epriestley
6af776f84a Allow installs to provide "Request a Username Change" instructions
Summary: Fixes T13420. Allow installs to provide username change instructions if there's someone you should contact to get this done.

Test Plan: {F6885027}

Maniphest Tasks: T13420

Differential Revision: https://secure.phabricator.com/D20828
2019-09-24 11:09:26 -07:00
epriestley
0c331458a8 When non-administrators click "Change Username", explain why they can't continue
Summary: Ref T13420. This workflow currently dead-ends for non-administrators. Instead, provide explanatory text.

Test Plan:
  - Clicked "Change Username" as an administrator, same workflow as always.
  - Clicked "Change Username" as a non-administrator, got explanatory text.

Maniphest Tasks: T13420

Differential Revision: https://secure.phabricator.com/D20827
2019-09-24 11:08:49 -07:00
epriestley
058b7ae945 Update "Change Username" instructions to be less foreboding
Summary:
Ref T13420. These warnings are currently more severe than they need to be; weaken them.

Among other cases, the upstream supports and encourages changing usernames when users change human names.

The login/password instructions are also out of date since sessions were decoupled from usernames about a year ago.

Test Plan: Hit dialog as an administrator

Maniphest Tasks: T13420

Differential Revision: https://secure.phabricator.com/D20826
2019-09-24 11:08:11 -07:00
epriestley
b1d4d5c00c Add an "{anchor #xyz}" rule to Remarkup
Summary: Ref T13410. Fixes T4280. Allows you to put a named anchor into a document explicitly.

Test Plan: Used `{anchor ...}` in Remarkup, used location bar to jump to anchors.

Maniphest Tasks: T13410, T4280

Differential Revision: https://secure.phabricator.com/D20825
2019-09-24 11:04:19 -07:00
epriestley
bff72ce3b5 Generate more friendly anchor names for header sections in Remarkup
Summary:
Depends on D20820. Ref T13410. We currently cut anchor names in the middle, don't support emoji in anchors, and generate relatively short anchors.

Generate slightly longer anchors, allow more unicode, and try not to cut things in the middle.

Test Plan: Created a document with a variety of different anchors and saw them generate more usable names.

Maniphest Tasks: T13410

Differential Revision: https://secure.phabricator.com/D20821
2019-09-24 11:03:03 -07:00
epriestley
be2a0f8733 Fix a cursor paging issue in Conduit call logs
Summary: Fixes T13423. The "Query" class for conduit call logs is missing a "withIDs()" method.

Test Plan: Paged through Conduit call logs.

Maniphest Tasks: T13423

Differential Revision: https://secure.phabricator.com/D20823
2019-09-24 08:36:49 -07:00
epriestley
09d86c2d20 Unprototype "Facts" to clear Maniphest/chart fatals
Summary: Ref T13279. Facts is still fairly rough, but not broken/policy-violating, so it can be unprototyped to fix the issue where Maniphest reports (which are now driven by Facts) don't work if prototypes are disabled.

Test Plan: Viewed Maniphest reports and Project reports with prototypes on/off and Fact installed/uninstalled.

Subscribers: yelirekim

Maniphest Tasks: T13279

Differential Revision: https://secure.phabricator.com/D20822
2019-09-23 13:01:18 -07:00
epriestley
16de9151c7 Give "Burndown" charts a more straightforward definition and move all the event stuff into "Activity" charts
Summary:
Depends on D20818. Ref T13279. The behavior of the "burndown" chart has wandered fairly far afield; make it look more like a burndown.

Move the other thing into an "Activity" chart.

Test Plan: {F6865207}

Maniphest Tasks: T13279

Differential Revision: https://secure.phabricator.com/D20819
2019-09-17 13:30:29 -07:00
epriestley
c06dd4818b Support explicit stacking configuration in stacked area charts
Summary:
Ref T13279. Allow engines to choose how areas in a stacked area chart stack on top of one another.

This could also be accomplished by using multiple stacked area datasets, but datasets would still need to know if they're stacking "up" or "down" so it's probably about the same at the end of the day.

Test Plan: {F6865165}

Subscribers: yelirekim

Maniphest Tasks: T13279

Differential Revision: https://secure.phabricator.com/D20818
2019-09-17 13:26:07 -07:00
epriestley
d4ed5d0428 Make various UX improvements to charts so they're closer to making visual sense
Summary: Ref T13279. Fix some tabular stuff, draw areas better, make the "compose()" API more consistent, unfatal the demo chart, unfatal the project burndown, make the project chart do something roughly physical.

Test Plan: Looked at charts, saw fewer obvious horrors.

Subscribers: yelirekim

Maniphest Tasks: T13279

Differential Revision: https://secure.phabricator.com/D20817
2019-09-17 09:43:21 -07:00
epriestley
080e132aa7 Track chart datapoints from their sources and provide a tabular view of chart data
Summary: Depends on D20815. Ref T13279. Give datapoints "refs", which allow us to figure out where particular datapoints came from even after the point is transformed by functions. For now, show the raw points in a table below the chart.

Test Plan: Viewed chart data, saw reasonable-looking numbers.

Subscribers: yelirekim

Maniphest Tasks: T13279

Differential Revision: https://secure.phabricator.com/D20816
2019-09-17 09:41:02 -07:00
epriestley
769e745a3f In charts, make "min" and "max" into pure functions and formally mark pure functions as pure
Summary:
Depends on D20814. Currently, "min()" and "max()" are still "min(f, n)". This is no longer consistent with the construction of functions a function-generators that are composed at top level.

Turn them into "min(n)" and "max(n)" (i.e., not higher-order functions).

Then, mark all the functions which are pure mathematical functions and not higher-order as "pure". These functions have no function parameters and do not reference external data. For now, this distinction has no immediate implications, but it will simplify the next change (which tracks where data came from when it originated from an external source -- these pure functions never have any source information, since they only apply pure mathematical transformations to data).

Test Plan: Loaded a burnup chart, nothing seemed obviously broken.

Subscribers: yelirekim

Differential Revision: https://secure.phabricator.com/D20815
2019-09-17 09:28:23 -07:00
epriestley
f529abf900 In stacked area charts, group nearby points so they don't overlap
Summary: Ref T13279. We currently draw a point on the chart for each datapoint, but this leads to many overlapping circles. Instead, aggregate the raw points into display points ("events") at the end.

Test Plan: Viewed a stacked area chart with many points, saw a more palatable number of drawn dots.

Subscribers: yelirekim

Maniphest Tasks: T13279

Differential Revision: https://secure.phabricator.com/D20814
2019-09-17 09:26:54 -07:00
epriestley
c5a4dea8cf Fix a typo in preamble X-Forwarded-For psuedocode
Summary: This psueudocode should use the result of computation at the end.

Test Plan: Read carefully.

Differential Revision: https://secure.phabricator.com/D20813
2019-09-16 10:35:08 -07:00
epriestley
3f66203362 Fix a straggling callsite to "renderApplicationPolicy()"
Summary: Ref T13411. This is a leftover from recent policy rendering changes.

Test Plan: Viewed feed with application policy stories, no more fatal.

Maniphest Tasks: T13411

Differential Revision: https://secure.phabricator.com/D20811
2019-09-12 16:26:57 -07:00
epriestley
41f0b8b0a3 Allow subtypes to specify "mutations", to control the behavior of the "Change Subtype" action
Summary:
Fixes T13415. Provide a way for subtypes to customize the behavior of "Change Subtype" actions that appear above comment areas.

Subtypes may disable this action by specifying `"mutations": []`, or provide a list of subtypes.

The bulk editor and API can still perform any change.

Test Plan:
  - Tried to define an invalid "mutations" list with a bad subtype, got a sensible error.
  - Specified a limited mutations list and an empty mutations list, verified that corresponding tasks got corresponding actions.
  - Used the bulk editor to perform a freeform mutation.
  - Verified that tasks of a subtype with no "mutations" still work the same way they used to (allow mutation into any subtype).

Maniphest Tasks: T13415

Differential Revision: https://secure.phabricator.com/D20810
2019-09-12 16:17:02 -07:00
epriestley
3e60128037 Support "Subtype" in Herald
Summary: See PHI1434. For objects that support subtypes and have subtypes configured, allow Herald rules to act on subtypes.

Test Plan:
  - Configured task and project subtypes, wrote Herald rules, saw "Subtypes" as an option, saw appropriate typeahead values and detail page rendering.
  - Unconfigured project subtypes, saw field vanish from UI for new rules.
  - Wrote a "subtype"-depenent rule that added a comment, interacted with tasks of that subtype and a different subtype. Saw Herald act only on tasks with the correct subtype.

Differential Revision: https://secure.phabricator.com/D20809
2019-09-12 14:34:06 -07:00
epriestley
d60d4e6a05 Don't present users with Herald fields/actions for uninstalled applications, unless the rule already uses them
Summary:
Fixes T7961. Currently, we present Herald users with actions like "Require legalpad signatures" and "Run build plans" even if Legalpad and Harbormaster are not installed.

Instead, allow fields and actions to be made "unavailable", which means that we won't present them as options when adding to new or existing rules.

If you edit a rule which already uses one of these fields or actions, it isn't affected.

Test Plan:
  - Created a rule with a legalpad action, uninstalled legalpad, edited the rule. Action remained untouched.
  - Created a new rule, wasn't offered the legalpad action.
  - Reinstalled the application, saw the action again.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T7961

Differential Revision: https://secure.phabricator.com/D20808
2019-09-12 14:33:28 -07:00
epriestley
4f845d8f8c When users encounter a policy exception for a non-view capability with a custom policy, inline the policy rules
Summary:
Fixes T13411. This looks like the last case where you hit a policy explanation and have permission to see the policy, but we don't currently show you the policy rules.

This implementation is slightly clumsy, but likely harmless.

Test Plan: {F6856421}

Maniphest Tasks: T13411

Differential Revision: https://secure.phabricator.com/D20807
2019-09-12 09:49:17 -07:00
epriestley
0c7ea8c942 When users fail a "CAN_SEE" check, give them an "opaque" policy explanation
Summary:
Ref T13411. Currently, if you hit a policy exception because you can't view an object, we disclose details about the view policy of the object, particularly which project's members can see the object for project policies.

Although there's a large amount of grey area here, this feels like a more substantial disclosure than we offer in other contexts. Instead, if you encounter a policy exception while testing "CAN_VIEW" or don't have "CAN_VIEW", present an "opaque" explanation which omits details that viewers who can't view the object shouldn't have access to. Today, this is the name of "Project" policies (and, implicitly, the rulesets of custom policies, which we now disclose in other similar contexts).

Test Plan:
  - Hit policy exceptions for "CAN_VIEW" on an object with a project view policy, saw an opaque explanation.
  - Hit policy exceptions for "CAN_EDIT" on an object with a project edit policy and a view policy I satisfied, saw a more detailed explanation.

Maniphest Tasks: T13411

Differential Revision: https://secure.phabricator.com/D20806
2019-09-12 09:42:02 -07:00
epriestley
9a36e6931c Inline custom policy rules inside policy capability explanation dialogs
Summary: Ref T13411. When users click a link to explain a capability (like the policy header on many objects, or the link next to specific capabilities in "Applications", "Diffusion", etc), inline the full ruleset for the custom policy into the dialog if the object has a custom policy.

Test Plan: {F6856365}

Maniphest Tasks: T13411

Differential Revision: https://secure.phabricator.com/D20805
2019-09-12 09:40:50 -07:00
epriestley
506f93b4a3 Give policy name rendering explicit "text name", "capability link", and "transaction link" pathways
Summary:
Ref T13411. This cleans up policy name rendering. We ultimately render into three contexts:

  - Plain text contexts, like `bin/policy show`.
  - Transaction contexts, where we're showing a policy change. In these cases, we link some policies (like project policies and custom policies) but the links go directly to the relevant object or a minimal explanation of the change. We don't link policies like "All Users".
  - Capability contexts, where we're describing a capability, like "Can Push" or cases in Applicaitons. In these cases, we link all policies to the full policy explanation flow.

Test Plan:
  - Used `bin/policy show` to examine the policy of an object with a project policy, no longer saw HTML.
  - Viewed the transaction logs of Applications (ModularTransactions) and Tasks (not ModularTransactions) with policy edits, including project and custom policies.
  - Clicked "Custom Policy" in both logs, got consistent dialogs.
  - Viewed application detail pages, saw all capabities linked to explanatory capability dialogs. The value of having this dialog is that the user can get a full explanation of special rules even if the policy is something mundane like "All Users".

Maniphest Tasks: T13411

Differential Revision: https://secure.phabricator.com/D20804
2019-09-12 09:39:05 -07:00
epriestley
c9b0d107f0 Remove unused "icon" parameter from policy name rendering
Summary: Ref T13411. This pathway has an unused "icon" parameter with no callsites. Throw it away to ease refactoring.

Test Plan: Grepped for callsites, found none using this parameter.

Maniphest Tasks: T13411

Differential Revision: https://secure.phabricator.com/D20803
2019-09-12 09:38:01 -07:00
epriestley
9c6969e810 Remove "Editable By" description fields in Passphrase, Phame, and Spaces
Summary:
Ref T13411. These three applications render an "Editable By: <policy>" field in their descriptions.

The pages that these appear on all have "Edit <thing>" actions which either tell you the policy or allow you to discover the policy, and this field is unusual (the vast majority of objects don't have it). I think it largely got copy/pasted or used as space-filler and doesn't offer much of value.

Remove it to simplify/standardize these pages and make changes to how this field works simpler to implement.

Test Plan: Viewed a Credential, Blog, and Space; no longer saw the "Editable By" field.

Maniphest Tasks: T13411

Differential Revision: https://secure.phabricator.com/D20802
2019-09-12 09:36:50 -07:00
epriestley
a35d7c3c21 Update rendering of policy edit transactions in Applications
Summary:
Ref T13411. Since circa D19829, transactions have rendered policy changes in a modern way, notably making "Custom Policy" clickable to show the policy rules.

Edit transactions in Applications still use a separate, older approach to render policies. This produces policy renderings which don't use modern quoting rules and don't link in a modern way.

Make Applications use the same rendering code that other transactions (like normal edit/view edits) use.

Test Plan: Edited policies in Applications, saw more useful transactions in the log. Clicked "Custom Policy" in the transaction log and got a useful explanation of the policy.

Maniphest Tasks: T13411

Differential Revision: https://secure.phabricator.com/D20801
2019-09-12 09:32:52 -07:00
epriestley
a0ade503e1 Remove "Moved Document from ..." notice in Phriction
Summary:
Ref T13410. See PHI1431. Currently, when you move a document in Phriction, the target shows a "This document was moved from ..." banner until it is edited.

This banner isn't particularly useful, and it's distracting and it isn't obvious how to dismiss it, and making a trivial edit to dismiss it is awkward.

This information is also already available in the transaction log.

Just remove this banner since it doesn't really serve any clear purpose.

Test Plan:
  - Moved a page in Phriction, then loaded the destination page. Before change: header banner. After change: nothing.
  - Viewed a normal (non-moved) page, saw normal behavior.
  - Reviewed transactions, saw "Moved from ..." in the timeline.

Maniphest Tasks: T13410

Differential Revision: https://secure.phabricator.com/D20800
2019-09-12 09:32:26 -07:00
epriestley
d2e1c4163a When a project has a custom icon, use that icon to label the project policy in the policy dropown
Summary:
Fixes T8808. Currently, all project use the default ("Briefcase") project icon when they appear in a policy dropdown.

Since project policies are separated out into a "Members of Projects" section of the dropdown anyway, there is no reason not to use the actual project icon, which is often more clear.

Test Plan: {F6849927}

Maniphest Tasks: T8808

Differential Revision: https://secure.phabricator.com/D20799
2019-09-09 13:38:12 -07:00
epriestley
1d1a60fdda Improve rendering of Herald rules in "Another Herald rule..." field
Summary:
Fixes T9136.

  - Fix a bug where the name is rendered improperly.
  - Put disabled rules at the bottom.
  - Always show the rule monogram so you can distingiush between rules with the same name.

Test Plan: {F6849915}

Maniphest Tasks: T9136

Differential Revision: https://secure.phabricator.com/D20798
2019-09-09 13:29:49 -07:00
epriestley
7593a265d5 When Herald changes object subscribers, always hide the feed story
Summary:
Fixes T8952. These feed stories are not interesting and tend to be generated as collateral damage when a non-story update is made to an old task and someone has a "subscribe me" Herald rule.

Also clean up some of the Herald field/condition indexing behavior slightly.

Test Plan: Wrote a "Subscribe X" herald rule, made a trivial update to a task. Before: low-value feed story; after: no feed story.

Maniphest Tasks: T8952

Differential Revision: https://secure.phabricator.com/D20797
2019-09-09 13:17:36 -07:00
epriestley
4547714463 Add a "Remove flag" action to Herald
Summary: Fixes T13409. This is a companion to the existing "Mark with flag" rule.

Test Plan: Used a "remove flag" rule on an object with no flag (not removed), the right type of flag (removed), and a different type of flag (not removed).

Maniphest Tasks: T13409

Differential Revision: https://secure.phabricator.com/D20796
2019-09-09 13:15:52 -07:00
epriestley
d965d9a669 Index Herald fields, not just actions, when identifying objects related to a particular Herald rule
Summary:
Fixes T13408. Currently, when a package (or other object) appears in a field (rather than an action), it is not indexed.

Instead: index fields too, not just actions.

Test Plan:
  - Wrote a rule like "[ Affected packages include ] ...".
  - Updated the search index.
  - Saw rule appear on "Affected By Herald Rules" on the package detail page.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13408

Differential Revision: https://secure.phabricator.com/D20795
2019-09-09 12:50:43 -07:00
epriestley
aaaea57591 Fix fatal during redirection safety check for searching for Phabricator base-uri with no trailing slash
Summary: Fixes T13412. If you search for "https://phabricator.example.com" with no trailing slash, we currently redirect you to "", which is fouled by a safety check in the redirection flow.

Test Plan:
  - Searched for "https://local.phacility.com"; before: fatal in redirection; after: clean redirect.
  - Searched for other self-URIs, got normal redirects.

Maniphest Tasks: T13412

Differential Revision: https://secure.phabricator.com/D20794
2019-09-09 12:45:24 -07:00
epriestley
278092974f Don't offer personal saved queries in global "Search Scope" settings dropdown
Summary: Fixes T13405. We currently offer non-global custom saved queries here, but this doesn't make sense as a global default setting.

Test Plan: Saved a global search query, edited global search settings, no longer saw the non-global query as an option.

Maniphest Tasks: T13405

Differential Revision: https://secure.phabricator.com/D20793
2019-09-09 12:21:25 -07:00
epriestley
63c7302af1 Fix global search scope fatal on 404 page (or other pages with no Application)
Summary: Ref T13405. Some pages don't have a contextual application.

Test Plan: Viewed 404 page, no more fatal.

Maniphest Tasks: T13405

Differential Revision: https://secure.phabricator.com/D20792
2019-09-09 12:18:26 -07:00
epriestley
535c8e6bdc Remove the "ONLY_FULL_GROUP_BY" SQL mode setup warning and change the setup key for "STRICT_ALL_TABLES"
Summary:
Ref T13404. Except for one known issue in Multimeter, Phabricator appears to function properly in this mode. It is broadly desirable that we run in this mode; it's good on its own, and enabled by default in at least some recent MySQL.

Additionally, "ONLY_FULL_GROUP_BY" and "STRICT_ALL_TABLES" shared a setup key, so ignoring one would ignore both. Change the key so that existing ignores on "ONLY_FULL_GROUP_BY" do not mask "STRICT_ALL_TABLES" warnings.

Test Plan: Grepped for `ONLY_FULL_GROUP_BY`.

Maniphest Tasks: T13404

Differential Revision: https://secure.phabricator.com/D20791
2019-09-09 12:17:51 -07:00
epriestley
f16365ed07 Weaken the guidance recommending that installs enable "STRICT_ALL_TABLES"
Summary: Ref T13404. Enabling "STRICT_ALL_TABLES" is good, but if you don't want to bother it doesn't matter too much. All upstream development has been on "STRICT_ALL_TABLES" for a long time.

Test Plan: {F6847839}

Maniphest Tasks: T13404

Differential Revision: https://secure.phabricator.com/D20790
2019-09-09 12:17:05 -07:00
epriestley
caccbb69d2 When users try to log out with no providers configured, warn them of the consequences
Summary: Fixes T13406. On the logout screen, test for no configured providers and warn users they may be getting into more trouble than they expect.

Test Plan:
  - Logged out of a normal install and a fresh (unconfigured) install.

{F6847659}

Maniphest Tasks: T13406

Differential Revision: https://secure.phabricator.com/D20789
2019-09-08 12:27:29 -07:00
Aviv Eyal
318e8ebdac Allow bin/config to create config file
Summary:
See D20779, https://discourse.phabricator-community.org/t/3089. `bin/config set` complains about
missing config file as if it's un-writable.

Test Plan: run `bin/config set` with missing, writable, unwritable conf.json and parent dir.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D20788
2019-09-08 00:16:19 +00:00
epriestley
7e2bec9280 Add a global setting for controlling the default main menu search scope
Summary: Fixes T13405. The default behavior of the global search bar isn't currently configurable, but can be made configurable fairly easily.

Test Plan: Changed setting as an administrator, saw setting reflected as a user with no previous preference. As a user with an existing preference, saw preference retained.

Maniphest Tasks: T13405

Differential Revision: https://secure.phabricator.com/D20787
2019-09-06 08:39:28 -07:00
epriestley
adc2002d28 Make it easier to parse "X-Forwarded-For" with one or more load balancers
Summary:
Fixes T13392. If you have 17 load balancers in sequence, Phabricator will receive requests with at least 17 "X-Forwarded-For" components in the header.

We want to select the 17th-from-last element, since prior elements are not trustworthy.

This currently isn't very easy/obvious, and you have to add a kind of sketchy piece of custom code to `preamble.php` to do any "X-Forwarded-For" parsing. Make handling this correctly easier.

Test Plan:
  - Ran unit tests.
  - Configured my local `preamble.php` to call `preamble_trust_x_forwarded_for_header(4)`, then made `/debug/` dump the header and the final value of `REMOTE_ADDR`.

```
$ curl http://local.phacility.com/debug/
<pre>

HTTP_X_FORWARDED_FOR =
   FINAL REMOTE_ADDR = 127.0.0.1
</pre>
```

```
$ curl -H 'X-Forwarded-For: 1.1.1.1, 2.2.2.2, 3.3.3.3, 4.4.4.4, 5.5.5.5, 6.6.6.6' http://local.phacility.com/debug/
<pre>

HTTP_X_FORWARDED_FOR = 1.1.1.1, 2.2.2.2, 3.3.3.3, 4.4.4.4, 5.5.5.5, 6.6.6.6
   FINAL REMOTE_ADDR = 3.3.3.3
</pre>
```

```
$ curl -H 'X-Forwarded-For: 5.5.5.5, 6.6.6.6' http://local.phacility.com/debug/
<pre>

HTTP_X_FORWARDED_FOR = 5.5.5.5, 6.6.6.6
   FINAL REMOTE_ADDR = 5.5.5.5
</pre>
```

Maniphest Tasks: T13392

Differential Revision: https://secure.phabricator.com/D20785
2019-09-05 04:30:13 -07:00
epriestley
764db4869c Make "bin/storage destroy" target individual hosts in database cluster mode
Summary:
Ref T13336. Currently, "bin/storage destroy" destroys every master. This is wonderfully destructive, but if replication fails it's useful to be able to destroy only a replica.

Operate on a single host, and require "--host" to target the operation in cluster mode, so `bin/storage destroy --host dbreplica001` is a useful operation.

Test Plan: Ran `bin/storage destroy` with various flags locally. Will destroy `secure002` and refresh replication.

Maniphest Tasks: T13336

Differential Revision: https://secure.phabricator.com/D20784
2019-09-04 10:11:08 -07:00
epriestley
f7290bbbf2 Update a straggling "getAuthorities()" call in Fund
Summary: Ref T13366. The "authorities" mechanism was replaced, but I missed this callsite. Update it to use the request cache mechanism.

Test Plan: As a user without permission to view some initiatives, viewed a list of initiatives.

Maniphest Tasks: T13366

Differential Revision: https://secure.phabricator.com/D20783
2019-09-04 07:15:20 -07:00
epriestley
22b075df97 Fix "ONLY_FULL_GROUP_BY" issue in SystemAction queries
Summary: Ref T13404. This query is invalid under "sql_mode=ONLY_FULL_GROUP_BY". Rewrite it to avoid interacting with `actorIdentity` at all; this is a little more robust in the presence of weird data and not really more complicated.

Test Plan:
  - Enabled "ONLY_FULL_GROUP_BY".
  - Hit system actions (e.g., login).
    - Before: error.
    - After: clean login.
  - Tried to login with a bad password many times in a row, got properly limited by the system action rate limiter.

Maniphest Tasks: T13404

Differential Revision: https://secure.phabricator.com/D20782
2019-09-03 16:50:33 -07:00
epriestley
e0d6994adb Use the "@" operator to silence connection retry messages if initializing the stack with database config optional
Summary:
Depends on D20780. Ref T13403. During initial setup, it's routine to run "bin/config" with a bad database config. We start the stack in "config optional" mode to anticipate this.

However, even in this mode, we may emit warnings if the connection fails in certain ways. These warnings aren't useful; suppress them with "@".

(Possibly this message should move from "phlog()" to "--trace" at some point, but it has a certain amount of context/history around it.)

Test Plan:
  - Configured MySQL to fail with a retryable error, e.g. good host but bad port.
  - Ran `bin/config set ...`.
  - Before: saw retry warnings on stderr.
  - After: no retry warnings on stderr.
  - (Turned off suppression code artificially and verified warnings still appear under normal startup.)

Maniphest Tasks: T13403

Differential Revision: https://secure.phabricator.com/D20781
2019-09-03 12:54:17 -07:00
epriestley
f8eec38c94 When "mysqli->real_connect()" fails without setting an error code, recover more gracefully
Summary: Depends on D20779. Ref T13403. Bad parameters may cause this call to fail without setting an error code; if it does, catch the issue and go down the normal connection error pathway.

Test Plan:
  - With "mysql.port" set to "quack", ran `bin/storage probe`.
  - Before: wild mess of warnings as the code continued below and failed when trying to interact with the connection.
  - After: clean connection failure with a useful error message.

Maniphest Tasks: T13403

Differential Revision: https://secure.phabricator.com/D20780
2019-09-03 12:51:20 -07:00
epriestley
d9badba147 Give "bin/config" a friendlier error message if "local.json" is not writable
Summary: Ref T13403. We currently emit a useful error message, but it's not tailored and has a stack trace. Since this is a relatively routine error and on the first-time-setup path, tailor it so it's a bit nicer.

Test Plan:
  - Ran `bin/config set ...` with an unwritable "local.json".
  - Ran `bin/config set ...` normally.

Maniphest Tasks: T13403

Differential Revision: https://secure.phabricator.com/D20779
2019-09-03 12:47:06 -07:00
epriestley
8ff3a133c4 Generalize repository proxy retry logic to writes
Summary:
Ref T13286. The current (very safe / conservative) rules for retrying git reads generalize to git writes, so we can use the same ruleset in both cases.

Normally, writes converge rapidly to only having good nodes at the head of the list, so this has less impact than the similar change to reads, but it generally improves consistency and allows us to assert that writes which can be served will be served.

Test Plan:
  - In a cluster with an up node and a down node, pushed changes.
  - Saw a push to the down node fail, retry, and succeed.
  - Did some pulls, saw appropriate retries and success.
  - Note that once one write goes through, the node which received the write always ends up at the head of the writable list, so nodes need to be explicitly thawed to reproduce the failure/retry behavior.

Maniphest Tasks: T13286

Differential Revision: https://secure.phabricator.com/D20778
2019-09-03 12:34:10 -07:00
epriestley
ff3d1769b4 Instead of retrying safe reads 3 times, retry each eligible service once
Summary: Ref T13286. When retrying a read request, keep retrying as long as we have canididate services. Since we consume a service with each attempt, there's no real reason to abort early, and trying every service allows reads to always succeed even if (for example) 8 nodes of a 16-node cluster are dead because of a severed network link between datacenters.

Test Plan: Ran `git pull` in a clustered repository with an up node and a down node; saw retry count dynamically adjust to available node count.

Maniphest Tasks: T13286

Differential Revision: https://secure.phabricator.com/D20777
2019-09-03 10:43:33 -07:00
epriestley
95fb237ab3 On Git cluster read failure, retry safe requests
Summary:
Depends on D20775. Ref T13286. When a Git read request fails against a cluster and there are other nodes we could safely try, try more nodes.

We DO NOT retry the request if:

  - the client read anything;
  - the client wrote anything;
  - or we've already retried several times.

Although //some// requests where bytes went over the wire in either direction may be safe to retry, they're rare in practice under Git, and we'd need to puzzle out what state we can safely emit.

Since most types of failure result in an outright connection failure and this catches all of them, it's likely to almost always be sufficient in practice.

Test Plan:
  - Started a cluster with one up node and one down node, pulled it.
  - Half the time, hit the up node and got a clean pull.
  - Half the time, hit the down node and got a connection failure followed by a retry and a clean pull.
  - Forced `$err = 1` so even successful attempts would retry.
  - On hitting the up node, got a "failure" and a decline to retry (bytes already written).
  - On hitting the down node, got a failure and a real retry.
  - (Note that, in both cases, "git pull" exits "0" after the valid wire transaction takes place, even though the remote exited non-zero. If the server gave Git everything it asked for, it doesn't seem to care if the server then exited with an error code.)

Maniphest Tasks: T13286

Differential Revision: https://secure.phabricator.com/D20776
2019-09-03 10:08:43 -07:00
epriestley
b6420e0f0a Allow repository service lookups to return an ordered list of service refs
Summary:
Ref T13286. To support request retries, allow the service lookup method to return an ordered list of structured service references.

Existing callsites continue to immediately discard all but the first reference and pull a URI out of it.

Test Plan: Ran `git pull` in a clustered repository with an "up" node and a "down" node, saw 50% serivce failures and 50% clean pulls.

Maniphest Tasks: T13286

Differential Revision: https://secure.phabricator.com/D20775
2019-09-03 10:05:40 -07:00
epriestley
9316cbf7fd Move web application classes into "phabricator/"
Summary: Ref T13395. Companion change to D20773.

Test Plan: See D20773.

Maniphest Tasks: T13395

Differential Revision: https://secure.phabricator.com/D20774
2019-09-02 07:58:59 -07:00
epriestley
b2b17485b9 Clean up two straggling UI issues in Phortune
Ref T13401. The checkout UI didn't get fully updated to the new View objects,
and account handles are still manually building a URI that goes to the wrong
place.
2019-08-31 09:36:23 -07:00
epriestley
533a5535b6 Remove the "grant authority" mechanism from users
Summary:
Ref T13393. See some previous discussion in T13366.

Caching is hard and all approaches here have downsides, but the request cache likely has fewer practical downsides for this kind of policy check than other approaches. In particular, the grant approach (at least, as previously used in Phortune) has a major downside that "Query" classes can no longer fully enforce policies.

Since Phortune no longer depends on grants and they've now been removed from instances, drop the mechanism completely.

Test Plan: Grepped for callsites, found none.

Maniphest Tasks: T13393

Differential Revision: https://secure.phabricator.com/D20754
2019-08-30 09:26:08 -07:00
epriestley
3c26e38487 Provide a simple read-only maintenance mode for repositories
Summary:
Ref T13393. While doing a shard migration in the Phacility cluster, we'd like to stop writes to the migrating repository. It's safe to continue serving reads.

Add a simple maintenance mode for making repositories completely read-only during maintenance.

Test Plan: Put a repository into read-only mode, tried to write via HTTP + SSH. Viewed web UI. Took it back out of maintenance mode.

Maniphest Tasks: T13393

Differential Revision: https://secure.phabricator.com/D20748
2019-08-29 15:23:10 -07:00
epriestley
c6642213d5 Straighten out replication/cache behavior in "bin/storage dump"
Summary:
Fixes T13336.

  - Prevent `--no-indexes` from being combined with `--for-replica`, since combining these options can only lead to heartbreak.
  - In `--for-replica` mode, dump caches too. See discussion in T13336. It is probably "safe" to not dump these today, but fragile and not correct.
  - Mark the "MarkupCache" table as having "Cache" persistence, not "Data" persistence (no need to back it up, since it can be fully regenerated from other datasources).

Test Plan: Ran `bin/storage dump` with various combinations of flags.

Maniphest Tasks: T13336

Differential Revision: https://secure.phabricator.com/D20743
2019-08-28 08:25:40 -07:00
epriestley
0943561dcb Fix incorrect construction of subtype map when validating "subtype" transactions against non-subtypable objects
Summary:
Fixes T13389. Currently, we try to "newSubtypeMap()" unconditionally, even if the underlying object does not support subtypes.

  - Only try to build a subtype map if subtype transactions are actually being applied.
  - When subtype transactions are applied to a non-subtypable object, fail more explicitly.

Test Plan: Clicked "Make Editable" in a fresh Calendar transaction form, got an editable form instead of a fatal from "newSubtypeMap()". (Calendar events are not currently subtypable.)

Maniphest Tasks: T13389

Differential Revision: https://secure.phabricator.com/D20741
2019-08-28 06:57:04 -07:00
epriestley
7198bd7db7 When "utf8mb4" is available, use it as the default client charset when invoking standalone "mysql" commands
Summary:
Fixes T13390. We have some old code which doesn't dynamically select between "utf8mb4" and "utf8". This can lead to dumping utf8mb4 data over a utf8 connection in `bin/storage dump`, which possibly corrupts some emoji/whales.

Instead, prefer "utf8mb4" if it's available.

Test Plan: Ran `bin/storage dump` and `bin/storage shell`, saw sub-commands select utf8mb4 as the client charset.

Maniphest Tasks: T13390

Differential Revision: https://secure.phabricator.com/D20742
2019-08-27 16:36:48 -07:00
epriestley
97a4a59cf2 Give the Phortune external portal an order view
Summary:
Depends on D20739. Ref T13366. Slightly modularize/update components of order views, and make orders viewable from either an account context (existing view) or an external context (new view).

The new view is generally simpler so this mostly just reorganizes existing code.

Test Plan: Viewed orders as an account owner and an external user.

Maniphest Tasks: T13366

Differential Revision: https://secure.phabricator.com/D20740
2019-08-26 07:49:17 -07:00
epriestley
a0a3879712 In Phortune, send order email to account external addresses
Summary: Depends on D20738. Ref T13366. Fixes T8389. Now that the infrastructure is in place, actually send email to external addresses.

Test Plan: Used `bin/phortune invoice` to generate invoices and saw associated external accounts receive mail in `bin/mail list-outbound`.

Maniphest Tasks: T13366, T8389

Differential Revision: https://secure.phabricator.com/D20739
2019-08-26 07:48:27 -07:00
epriestley
4e13551e85 Add credential rotation and statuses (disabled, unsubscribed) to Phortune external email
Summary: Depends on D20737. Ref T13367. Allow external addresses to have their access key rotated. Account managers can disable them, and anyone with the link can permanently unsubscribe them.

Test Plan: Enabled/disabled addresses; permanently unsubscribed addresses.

Maniphest Tasks: T13367

Differential Revision: https://secure.phabricator.com/D20738
2019-08-26 07:47:44 -07:00
epriestley
8f6a1ab015 Roughly support external/email user views of Phortune recipts and invoices
Summary: Ref T13366. This gives each account email address an "external portal" section so they can access invoices and receipts without an account.

Test Plan: Viewed portal as user with authority and in an incognito window.

Maniphest Tasks: T13366

Differential Revision: https://secure.phabricator.com/D20737
2019-08-26 07:39:08 -07:00